ORF PowerLog preprocessing works with large amounts of unsorted data and does this on “idle” priority (that means, PowerLog preprocessing will run only if your CPUs has nothing better to do). It is not entirely surprising that it is slow, but if your server deals with 1.000.000 emails a day, your daily PowerLog files are 500Mb large and preprocessing of a single PowerLog file takes some 19 hours, I guess you do not find solace in this. Neither did we, so optimizations were made and now ORF processes the 500Mb and 1.000.000 emails PowerLog .OPG file about 75 times faster, in about 15 minutes instead of 19 hours (measured on our test system). The improved preprocessing will be available in the next version.
Technical announcement
You might have experienced a 30 minutes downtime of our web site yesterday, the story follows.
While doing some software maintenance on our colocation server using RDP I took a look at the raid array just to realise that one of the disks was failed. And it was failed for two weeks already. It is a 3ware raid card, and of course it has a feature that alerts you via email if anything goes wrong – I just forgot to set it up when installed the server.
So I grabbed an identical hard disk – we did have some spare -, scheduled a visit at the colocation site at 22:00 CET, went in, replaced the hard drive, went home. The time I spend there was about 30 minutes, which could has been shorter if I can go to the server but it is not possible. I can only go to the visitor room and the technical staff brings the server there – no matter if it has hot-swappable disks.
The current config is RAID5, but I am worried about the possible data loss which can occur if 1. two disks fail at the same time, or 2. one other disk fails during array rebuild. The first option is quite possible, given that usually the disks that are built in a server are manufactured together, with almost identical serial numbers. So in the future we will opt for RAID 6, which seems to be a good choice. The cost of a gigabyte is ever decreasing, and there are 500G or even 750G drives already – one can easily put a number of terabytes into a server, which seemed unreachable even a few years ago.
After restarting the server, the actual rebuild took about 6 hours to complete. I had to fight some other problems as well – one was that the NNTP server did not start, required some indexing. This is a known problem – at least for us-, that thing usually fails about once a month.
Why does not ORF do “real Reverse DNS”?
Well, that is an easy question. The answer is 3.2%.
The “Real reverse DNS test”—that is, checking whether the sending server has a DNS name—is in the Top 10 requested features since years. So far we resisted adding it to ORF, due to the high false positive rate and once or twice a year I ran a test on our legitimate email sample body to see whether the false positive figures changed. My last test was performed on ORF tech support emails from the last 45 days and the false positive rate was an unacceptably high 3.2%.
Sure, we can treat this type of test a policy enforcement tool instead of a spam filter, but come on, who want to lose 3.2% or even 0,5% of the legitimate emails? Yes, we could add a big flashing warning sign telling “You will shoot yourself in the foot”, but I think it is better not to allow the user to shot herself in the foot at the first place.
What do you say?
More flexible network definitions
If you ever felt awkward for not knowing how to calculate subnet masks without using a subnet mask calculator, consider that neither we could have survived without NetCalc. There are so many things in life to waste your time with, all better than learning subnet calculation, right?

When we decided to use the dotted subnet mask notation for defining network ranges in ORF (you know, like 192.168.251.1/255.255.255.0), we thought that it is only us who have troubles converting the 192.168.251.1-192.168.251.255 range in mind into a subnet masked format. As it slowly turned out, we were wrong. Good news: finally we fix this.
The next ORF release is going support various network range notations, so you can basically enter IP ranges in whatever format they come. The supported formats are:
- 192.168.251.0-192.168.251.255 (text range)
- 192.168.251.0/24 (CIDR notation)
- 192.168.251.0/255.255.255.0 (dotted IPv4 subnet mask)
- 192.168.251.* (wildcarded)
The next thing to come is the Log Viewer update – brand new, improved filtering, free-text search and updates required by the log format changes.
Combined Actions and the ORF Log
I just finished implementing an ORF improvement that we call Combined Actions. This allows combining specific On Arrival actions, for instance, tagging the subject plus the header plus redirecting the email.
Unrelated as it may seem, this improvement bought some changes to the ORF Text log format. The latest 3.0 log format has event classes for each type of action (14 event classes, from TagMailHeader to IgnoreMailRecipient) and so combined actions would have required to introduce new event classes for every possible combination (or, just to allow multiple event classes for a single event), causing a boom in the number of possible event classes.
This, among with other reasons, made us to re-think how logs should like and to come up with a few changes that we believe make logs easier to read and more useful.
First, we reduced the number of event classes to five: System, Pass, Whitelist, Blacklist and Intermediate. I think the first four are quite straightforward, but the Intermediate event class requires a little explanation: this is an event class that reports an intermediate action performed on the email. Currently, these are:
- Recipient removal at On Arrival. This may occur if some, but not all of the email recipients are blacklisted, e.g. by the Active Directory recipient validation.
- Recipient whitelisting at On Arrival. Again, this may occur if some, but not all of the email recipients are whitelisted.
- Attachment replacement (Attachment Filtering).
To identify the exact action performed, a new Action column has been introduced (can describe multiple actions, e.g. TagSubject+Redirect).
Another change was to eliminate the optional “Server” column, as it made little sense to log the the server name with every log message when the server name cannot be changed without rebooting the server. The local server name was moved to the ORF log header and it is no longer optional.
The next version’s log will have one more thing that I guess you will like: an optional email subject column. The subject is logged in UTF-8 encoding (ASCII letters kept human-readable), in URL Encoded format.