We have been contacted by a few people last week about a small peak in the number of legitimate emails blacklisted by SORBS, which I believe was caused by escalated listings (see the blog post of SORBS for details). Though we discourage using the escalated zone of SORBS because of the higher chance for false positives (we removed response 127.0.0.6 from the DNSBL definition shipped with ORF a long time ago), some folks still have it enabled.

In short, ISPs do not care much if a paying customers is spamming until the whole IP block gets blacklisted by a DNSBL and other (innocent) customers start to complain. I do not think it’s the ideal way of doing things (i.e. to force the ISPs to get rid of spammers by blacklisting innocent people), on the other hand, I totally understand SORBS and other DNSBLs: they simply have no other choice if the ISP ignores their reports of abuse.

What do you think?

The ORF Log Viewer Tool is excellent for checking what happened to an email: by building simple filters with a few clicks, you have all the information you need. However, loading extremely huge log files (we are talking about hundreds of megabytes) may take quite a lot of time, as ORF logs of this size simply contain too much data.

For these cases, we provide a log parser tool which can parse and dump the log data into SQL databases, so you can use SQL queries to retrieve the information you seek. Moreover, you can also use this parsed data (now in SQL) for web-based reports for end-users (see the demo on the link above).

But what if a user asks you to show him all blacklisted emails sent to his address from the past week ASAP, and you simply don’t have the time or mood to setup an SQL database for this? Meet the Microsoft Log Parser.

The Microsoft Log parser is a “powerful, versatile tool that provides universal query access to text-based data”. By using it, you can

• simply search in multiple ORF log files at once
• export the query results to a CSV file, which can be then viewed in Microsoft Excel or OpenOffice.org’s Calc application

How can we do this?

1. First, download the Microsoft Log Parser from this site.
2. Install it.
3. Start it from the Start menu (it will open a command line window).

You should read the documentation about the query syntax. Basically, we need to set the input location and type, the output file (where we dump the query results), and the SQL query parameters. Here is a sample query with a single query parameter (recipient address is test@mydomain.com).

logparser -i:w3c “SELECT x-event-datetime AS Date-Time,x-event-class AS Event,x-event-severity AS Severity,x-event-msg AS Description,x-filtering-point AS Filter,x-ip AS IP-address,x-msg-id AS MsgID,x-msg-subject AS Subject,x-sender AS Sender,x-recipients AS Recipient INTO test_report.csv FROM ‘C:\Program Files\ORF Enterprise Edition\*.log’ WHERE recipient LIKE ‘%%test@mydomain.com%%’”

This command will create a file called test_report.csv in the Log Parser directory which includes all log entries for this recipient from the ORF logs (located in Program Files\ORF Enterprise Edition). Note that if you do not have the logging of the message ID in ORF Enabled (Configuration \ Global \ Log and events \ ORF Text log – Configure button \ Settings tab) you should remove the “x-msg-id AS MsgID” part.

As you can see, the query is flexible, so the possibilities are endless: feel free to experiment and share your ideas in the comments.

Quite often, our emails bounce back from our customers because of failed SPF test results. One might suspect our SPF record is broken, but actually the problem is caused by misconfigured ORF installations on the recipient’s end in all cases (our SPF record is fine):

Let’s assume the recipient has a primary MX with ORF installed, and the SPF test enabled. Also, there is a secondary MX hosted by the recipient’s ISP, but the secondary MX is not added to the Intermediate Host List of ORF. The Intermediate Host List (as the name implies) is a list of intermediate hosts through which emails are relayed to the primary MX (and ORF) from external sources. The sender server sends the email to the secondary MX, which relays it to the primary MX, finally, the recipient receives it. ORF should be aware of such hosts, otherwise it will think the secondary MX sent the email directly. This may cause spam emails allowed in, as the DNS Blacklist test will be uneffective (as ORF checks the secondary MX IP against them instead of the actual sender IP). Moreover, if you have the SPF test enabled, it will check the secondary MX IP against the SPF policy of the sender. Of course, the secondary MX is not authorized to send any emails in the name of the sender domain, so the (otherwise legitimate) email will be blacklisted.

To avoid this problem, you should add all external relaying hosts (such as secondary MXs) to your Intermediate Host List, so ORF can “look behind” your secondary MX at the On Arrival filtering point by checking the email headers. For more information, please read the Intermediate Host List and Header Analysis topics of the ORF Help.

As you may have heard, ORF won the VBSpam Award for being super effective (caught 99.13% of spam emails during the test with zero false positives).

A lot of you guys asked about the configuration used during the test, so here it goes. We used a very basic, quite conservative configuration with the following test enabled:

• Reverse DNS (Normal sender domain validation, PTR check was turned off – default setting)
• HELO Blacklist (”Is malformed” and “is the same as the recipient domain” options on – default setting)
• SPF (blacklist on SoftFail disabled – default setting)
• DNS Blacklist (the recommended lists were enabled: SpamCop, Spamhaus ZEN, SORBS, NJABL)
• URL Blacklist (the recommended lists were enabled: Spamhaus DBL*, SURBL: Combined, uribl.com)

That’s it. Like I said, it was pretty basic :) You can lower the chance for false positives even more by using the Auto Sender Whitelist test, but we could not use that during the VB testing, as no outgoing emails were involved.

If you do not see the above mentioned DNS and URL blacklists in your configuration, please read our best practices guide for detailed instructions on updating your blacklist definition sets. You will also find some useful tips & tricks on fine-tuning ORF and increasing the catch rate even further, so it is a must read for every ORF user. If any questions arise, feel free to contact us: we are happy to assist.

*The URL Blacklist definition of Spamhaus DBL is not included in any versions of ORF as of writing this (4.4 and older), but you can download it from our site and import it easily

We often receive emails with questions like “Why ORF blocked this email?” and “Why this was allowed through?” and “Did ORF blocked this one or something else?”. Of course, it is easy to check using the Log Viewer tool shipped with ORF, but it often turns out ORF users are not aware of the existence of this tool. So I decided to show them what they miss :)

What is it good for?
ORF logs everything (what it does to emails, errors, warnings, etc.) to its text log files. These files are stored in the ORF directory by default (Program Files \ ORF Enterprise Edition) with .log extension. The Log Viewer can be used to review these text logs, so you can find out what it did to incoming emails and why. Let’s start the Log Viewer and see what information can be retrieved from these logs and how:

Loading the log files
You can load the log files from their default location from the last 24 hours by pressing F5. The time period and the directory from which the log files are loaded can be changed anytime by selecting File | Settings… in the main menu. You can also configure the Log Viewer to load the specified log entries on startup immediately, or drag and drop any .log file to the Log Viewer window to load it.

Once the logs are loaded, the data is represented in a table view: each row represents an event or action (not an email!), and by clicking the columns you can re-order the log records by date, source IP, sender and recipient addresses and so on.

Interpreting the log records
There could be several entries for a single email. For example if you have both Before and On Arrival tests enabled, ORF will log what happened to each recipient at Before Arrival, what happened at On Arrival, and if there were any errors or problems during the testing of the email, etc. These are all logged in separate records. The message column indicates detailed information about each record, it is pretty straightforward.

Searching and filtering the log records
Any entries can be quickly located by using the Search option (Ctrl+F). If want to review multiple entries based on certain criterias, there is an excellent Filter builder in the Log Viewer (Shift + Ctrl + F). Just like the manual lists of the Administration Tool, it supports wildcards and regular expressions. For example you can setup a filter to list all blacklisted emails sent from sender@senderdomain.com to any recipient in mydomain.com where the subject does not contain the word “meeting”:

You can also filter for log record types like warnings and errors. Moreover, you can save your filters for future usage.

Next time, we will check out the ORF Reporting Tool, so stay tuned.

We received sporadic reports of ORF blacklisting emails from the users’ own domain, because it does not find any MX or A/CNAME record for it. Of course, these records clearly exists when checked from the outside using nslookup, which makes the customer believe it is something wrong with ORF.

Actually, the problem is DNS-related and typically occurs when somebody uses a local DNS server in ORF for DNS resolution (which is recommended), but the very same local DNS acts as the authoritative DNS for his own domain. A common factor leading to this if the internal AD domain is the same as the public domain (e.g. domain.com, instead of domain.local or domain.internal or something like that).

To solve this, you should either switch to external DNS servers in ORF, or consider setting up another DNS server (e.g. on the local host) that forwards to the root DNS servers. The latter is the recommended method.

Microsoft introduced the Edge and Hub roles in Exchange 2007. The idea was to separate the perimeter (gateway) functions in order to perform filtering before the email reaches the central server. This includes recipient validation, i.e. to reject all emails sent to non-existent recipients. This is performed by an Exchange transport agent called the “Recipient Filter Agent”.

The Active Directory-based Recipient Validation of ORF is not available on Edge servers, because the Edge server do not have direct access to the AD, so ORF cannot query the valid recipients. This would not be a problem, (since Edge will reject emails sent to non-existent recipients anyway), but the Directory Harvest Attack (DHA) Protection test of ORF relies on the recipient validation of ORF, so that test will not be available on Edge (unless you use TXT or SQL-based recipient validation).

Moreover, as the Honeypot test relies on spam emails sent to non-existent email addresses (which you published to lure spammers), that won’t work either, because Edge will reject spam before ORF could record the delivery attempt to the Honeypot database.

Luckily, we can work the latter problem around by configuring the Trasport Agent of ORF to run before the Recipient Filter Agent:

1) Start the Exchange Management Shell
2) Enter the following command:

Get-TransportAgent | Format-List

3) ORF has two agents, the “Vamsoft ORF Routing Agent” and the “Vamsoft ORF Receive Agent”. You should set their priority of the latter higher than the priority of the “Recipient Filter Agent”, so it would run first.

To change the priority (e.g. to 7), run the following command:

Set-TransportAgent -Identity “Vamsoft ORF Receive Agent” -Priority 7

4) Finally, restart the MSExchangeTransport Service to apply the changes:

Restart-Service MSExchangeTransport

This way, the Honeypot test is performed before the email is rejected by the recipient validation of Edge.

We have received many reports that opening the installer executable of the trial or registered version of ORF fails with the following error message:

“Windows cannot access the specified device, path, or file. You may not have the appropriate permission to access the item.”

This error is shown even if you downloaded and try to run the installer as an Administrator, and you have full control over the file and the folder it resides in.

This is caused by a default Windows security feature (or rather: annoyance) to prevent EXE files from running before you Unblock them (huh?)

To fix this, do the following:

1. Right-click the file in Windows Explorer, select Properties
2. On the General tab, click the Unblock button

Now you can proceed with the installation.

The current trial version of ORF (4.3) supports Exchange 2010 after installing a patch, but you may receive an error if you try to install the registered build on top of this patched trial version. That is because the installer of 4.3 registered does not incorporate the Exchange 2010 patch by default.

To solve this problem, you should simply ignore the installation error during the conversion process, install the patch again for the registered build, then issue the following command in the ORF installation directory:

orfainst -install

This will be addressed by the upcoming 4.4 version, which includes the patch by default.

Lately, we have received many inquiries regarding the filtering of display names in emails. The display name in your email client (e.g. Outlook) is actually the From: field of the MIME email header. The MIME headers can be retrieved by selecting View | Options in Outlook (”Internet Headers”). To check the MIME headers in other clients, please visit this page). Example:

[...]
From: “VIAGRA \(c\) Best Supplier” (email@address)
[...]

You can filter this MIME From: field using the Keyword Blacklist of ORF:

1. Download the filter expression by right-clicking this link and selecting “Save link as…” (XML file)
1. Start the ORF Administration Tool
2. Expand Configuration / Tests / Tests in the left navigation tree and make sure the Keyword Blacklist test is enabled
3. Select Configuration | Import | Keyword blacklist… from the main menu, or navigate to Configuration / Filtering – On Arrival / Keyword Blacklist, right-click in the expressions box and select “Import list…”
4. Select the XML file you downloaded and click Open
5. If you already have some expressions in the list, you will be prompted “Do you want to overwrite…?”. Click “No” (otherwise your current expressions will be wiped out)
6. Press Ctrl + S to save and apply the configuration changes (pre-4.3 users should press Ctrl + U)

And that’s it: the expression above will block any emails, which have “Viagra” in their MIME From: email header line.

However, I should point out that we suggest relying on automated tests of ORF (like DNS and URL blacklists) as much as possible instead of adding keyword filtering expressions every time you receive a new type of spam (and instead of adding the sender to the Sender or IP Blacklists), so you should probably read our best practices guide regarding the recommended configuration if you have received such spam we mentioned above (”viagra” in the display name).

Our own ORF instance at Vamsoft (which is configured according to the guide) caught all of these using automated tests ;)

UPDATE: some of you guys reported that the regex doesn’t work: that’s because the expression above is altered by our blog engine, Wordpress (it replaces the double quote characters with left double quotation marks). To work this around, download this XML file from the link and import it to your Keyword Blacklist.