Author Archives: Krisztian Fekete

About Krisztian Fekete

Tech support engineer and Jack of all trades at Vamsoft. Knows regular expressions and isn’t afraid to use them. Takes active part in the nightlife of Budapest. Occasionally stars as an extra in commercials and movies, just for the fun of it. Find me on LinkedIn

Tales From Tech Support – Part 17: Searching in Huge Log Files

The ORF Log Viewer Tool is excellent for checking what happened to an email: by building simple filters with a few clicks, you have all the information you need. However, loading extremely huge log files (we are talking about hundreds of megabytes) may take quite a lot of time, as ORF logs of this size simply contain too much data.

For these cases, we provide a log parser tool which can parse and dump the log data into SQL databases, so you can use SQL queries to retrieve the information you seek. Moreover, you can also use this parsed data (now in SQL) for web-based reports for end-users (see the demo on the link above).

But what if a user asks you to show him all blacklisted emails sent to his address from the past week ASAP, and you simply don’t have the time or mood to setup an SQL database for this? Meet the Microsoft Log Parser.

The Microsoft Log parser is a “powerful, versatile tool that provides universal query access to text-based data”. By using it, you can

  • simply search in multiple ORF log files at once
  • export the query results to a CSV file, which can be then viewed in Microsoft Excel or OpenOffice.org’s Calc application

How can we do this?

1. First, download the Microsoft Log Parser from this site.
2. Install it.
3. Start it from the Start menu (it will open a command line window).

You should read the documentation about the query syntax. Basically, we need to set the input location and type, the output file (where we dump the query results), and the SQL query parameters. Here is a sample query with a single query parameter (recipient address is test@mydomain.com).

logparser -i:w3c “SELECT x-event-datetime AS Date-Time,x-event-class AS Event,x-event-severity AS Severity,x-event-msg AS Description,x-filtering-point AS Filter,x-ip AS IP-address,x-msg-id AS MsgID,x-msg-subject AS Subject,x-sender AS Sender,x-recipients AS Recipient INTO test_report.csv FROM ‘C:\Program Files\ORF Enterprise Edition\*.log’ WHERE recipient LIKE ‘%%test@mydomain.com%%'”

This command will create a file called test_report.csv in the Log Parser directory which includes all log entries for this recipient from the ORF logs (located in Program Files\ORF Enterprise Edition). Note that if you do not have the logging of the message ID in ORF Enabled (Configuration \ Global \ Log and events \ ORF Text log – Configure button \ Settings tab) you should remove the “x-msg-id AS MsgID” part.

As you can see, the query is flexible, so the possibilities are endless: feel free to experiment and share your ideas in the comments.

Tales From Tech Support – Part 16: SPF test and Intermediate Hosts

Quite often, our emails bounce back from our customers because of failed SPF test results. One might suspect our SPF record is broken, but actually the problem is caused by misconfigured ORF installations on the recipient’s end in all cases (our SPF record is fine):

Let’s assume the recipient has a primary MX with ORF installed, and the SPF test enabled. Also, there is a secondary MX hosted by the recipient’s ISP, but the secondary MX is not added to the Intermediate Host List of ORF. The Intermediate Host List (as the name implies) is a list of intermediate hosts through which emails are relayed to the primary MX (and ORF) from external sources. The sender server sends the email to the secondary MX, which relays it to the primary MX, finally, the recipient receives it. ORF should be aware of such hosts, otherwise it will think the secondary MX sent the email directly. This may cause spam emails allowed in, as the DNS Blacklist test will be uneffective (as ORF checks the secondary MX IP against them instead of the actual sender IP). Moreover, if you have the SPF test enabled, it will check the secondary MX IP against the SPF policy of the sender. Of course, the secondary MX is not authorized to send any emails in the name of the sender domain, so the (otherwise legitimate) email will be blacklisted.

SPF Warning

To avoid this problem, you should add all external relaying hosts (such as secondary MXs) to your Intermediate Host List, so ORF can “look behind” your secondary MX at the On Arrival filtering point by checking the email headers. For more information, please read the Intermediate Host List and Header Analysis topics of the ORF Help.

Tales From Tech Support – Part 15: Configuration Used for the VBSpam Test

As you may have heard, ORF won the VBSpam Award for being super effective (caught 99.13% of spam emails during the test with zero false positives).

VBSpam Test Results, May 2010

A lot of you guys asked about the configuration used during the test, so here it goes. We used a very basic, quite conservative configuration with the following test enabled:

  • Reverse DNS (Normal sender domain validation, PTR check was turned off – default setting)
  • HELO Blacklist (“Is malformed” and “is the same as the recipient domain” options on – default setting)
  • SPF (blacklist on SoftFail disabled – default setting)
  • DNS Blacklist (the recommended lists were enabled: SpamCop, Spamhaus ZEN, SORBS, NJABL)
  • URL Blacklist (the recommended lists were enabled: Spamhaus DBL*, SURBL: Combined, uribl.com)

That’s it. Like I said, it was pretty basic :) You can lower the chance for false positives even more by using the Auto Sender Whitelist test, but we could not use that during the VB testing, as no outgoing emails were involved.

If you do not see the above mentioned DNS and URL blacklists in your configuration, please read our best practices guide for detailed instructions on updating your blacklist definition sets. You will also find some useful tips & tricks on fine-tuning ORF and increasing the catch rate even further, so it is a must read for every ORF user. If any questions arise, feel free to contact us: we are happy to assist.

*The URL Blacklist definition of Spamhaus DBL is not included in any versions of ORF as of writing this (4.4 and older), but you can download it from our site and import it easily

ORF Feature Highlights – Reporting

Welcome to our new series, which intends to demonstrate various features of ORF. The first part (and video) will show you the reporting capabilities of ORF, which were introduced in version 3.0. The reporting feature is enabled by default, so you can start creating reports immediately.

Report Creation
Let’s start the ORF Reporting Tool. To create a new report, click on Create new report on the opening screen (or press Ctrl + N). ORF will ask you to set the time period which from you would like to create a report. You can either set a start and end date using a simple calendar, or select a preset. Finally, click Create. Yes, it’s that simple :)

Report Settings

Reviewing Reports
Reports show you how many emails ORF filtered, how many was allowed through, what are the most spammed recipient addresses, which are the most effective tests and filtering expressions in your setup, etc. By clicking on Navigation & Info in the upper left corner, a navigation pane will pop in, which can be used to skip to various sections of the report. If you have trouble interpreting any details, just click on the question mark icon for detailed explanation, which will be displayed in the left pane.

Sample Report

Printing and Saving Reports
The Reporting Tool supports printing either the entire report or a single section of it, and you can always save the report in RPT format for later review. RPT is the native format of the Reporting Tool: saving the report to other formats are not supported as of writing this, but you can work this around by using 3rd party tools (like the free PrimoPDF printer to “print” the report to a PDF file).

Notes & Troubleshooting
In order to create reports, you need the so-called PowerLogs to be enabled in the Administration Tool. These PowerLogs were designed specifically for the reporting feature: they store a lot more information about the email traffic than the regular text log files. PowerLogging is enabled by default, but if you have disabled it in the past, you should enable it: start the Administration Tool and expand Configuration / Global / Log and events in the left navigation tree. Click the Configuration button under PowerLogs and make sure “powerlogging” is enabled. When PowerLog is enabled, the ORF service will generate PowerLog files to the specified path with .opg extension. ORF pre-processes these PowerLogs in idle time creating files with identical file names, but with .ppr extension. The reports can be generated from the latter files. Once the pre-processing is complete, the .opg files are needed no longer, so you can configure the ORF service to delete them to save disk space. Also, please note that if you just enabled PowerLogs, you will need to wait at least a day until the first .ppr file is created in order to generate reports.

PowerLog Settings

Summary
Reports can not only be used to amaze your boss and colleagues how effective ORF is, but they also provide useful information about your current configuration, which can be used to improve the filtering efficiency further. For example, the “Top spammed recipients” list is an excellent starting point for setting up your Honeypot address list.

Tales From Tech Support -Part 14: Using the ORF Log Viewer

We often receive emails with questions like “Why ORF blocked this email?” and “Why this was allowed through?” and “Did ORF blocked this one or something else?”. Of course, it is easy to check using the Log Viewer tool shipped with ORF, but it often turns out ORF users are not aware of the existence of this tool. So I decided to show them what they miss :)

What is it good for?
ORF logs everything (what it does to emails, errors, warnings, etc.) to its text log files. These files are stored in the ORF directory by default (Program Files \ ORF Enterprise Edition) with .log extension. The Log Viewer can be used to review these text logs, so you can find out what it did to incoming emails and why. Let’s start the Log Viewer and see what information can be retrieved from these logs and how:

Loading the log files
You can load the log files from their default location from the last 24 hours by pressing F5. The time period and the directory from which the log files are loaded can be changed anytime by selecting File | Settings… in the main menu. You can also configure the Log Viewer to load the specified log entries on startup immediately, or drag and drop any .log file to the Log Viewer window to load it.

Log Viewer settings

Once the logs are loaded, the data is represented in a table view: each row represents an event or action (not an email!), and by clicking the columns you can re-order the log records by date, source IP, sender and recipient addresses and so on.

Interpreting the log records
There could be several entries for a single email. For example if you have both Before and On Arrival tests enabled, ORF will log what happened to each recipient at Before Arrival, what happened at On Arrival, and if there were any errors or problems during the testing of the email, etc. These are all logged in separate records. The message column indicates detailed information about each record, it is pretty straightforward.

Searching and filtering the log records
Any entries can be quickly located by using the Search option (Ctrl+F). If want to review multiple entries based on certain criterias, there is an excellent Filter builder in the Log Viewer (Shift + Ctrl + F). Just like the manual lists of the Administration Tool, it supports wildcards and regular expressions. For example you can setup a filter to list all blacklisted emails sent from sender@senderdomain.com to any recipient in mydomain.com where the subject does not contain the word “meeting”:

filter

You can also filter for log record types like warnings and errors. Moreover, you can save your filters for future usage.

Next time, we will check out the ORF Reporting Tool, so stay tuned.