…And… Action!

Refactoring done, PowerLog support is now integrated. We will release the first development version of ORF 3.0 to the Feature Test Program members next Monday. It is an important milestone, even though there is awful lot left to do.

*

Now that the Reporting feature is getting its final shape, we see that many report types requested by our clients will not get into 3.0, due to the compromises that we had to make between report availability and customizability. A good example is per-user reporting—that is, telling how tests worked for john@example.org and jane@example.org. Each report breakdown (per server, per virtual server, per recipient domain, per recipient) multiplies the amount of data to be processed, which affects report availability negatively.

Sure, it is impossible to meet everyone’s expectations, but can we do anything to allow clients to generate custom reports?

The first and most obvious answer is SQL logging. Once you have the data in a SQL database, you can do just about anything with it. Also, many administrators speak SQL to some extent. Can anything be wrong with this?

Well, yes. First, ORF PowerLog entries are quite complex structures and SQL logging would need some some 30+ tables (imagine your SELECTs). Second, you would probably have too much data to be queried. Third, (technical) supporting such feature would be crazy.

Ok, drop SQL logging and let’s see the next idea. What about providing the raw data to the interested clients and let them decide what to do with it? After all, this is what ORF will do internally. ORF will read the log entry, add 1 to the “Number of Tested Emails” counter, add 1 to the “Number of IP Whitelist Tests” counter, etc. and step to the next log entry. If you could do the same, you would have total freedom with data processing. Great, isn’t it? Can anything be wrong with this?

Well, yes. As you may already have figured out, this would require some programming skills and the number of administrators who can write scripts is much lower than those who master SQL. Sure, we can turn this into a mega-project (as opposed to publishing the log reader as a script-accessible COM API) and offer some smart GUI to reduce the code load, but in the end you would have to code a little for the custom reports.

Anyhow, if you are interested in this way of generating custom reports, post a comment. I am just thinking aloud, but let me know what you think.

Debug Prints in Release Code

This is a message to all the developers out there: Please don’t ship release code with debug prints!

We rely on DebugView a lot for debugging ORF internal releases and every software which pollutes the system debug output makes it harder to do. Not just for us, but for everyone else who uses this excellent debugging tool. Sure, we can exclude these processes from the trace, but time and time again a new one starts doing things like Office 2003’s type32.exe

[2932] Check exception: ..\dpgbase\regvalue.h (375), DPG::RegValue,class std::allocator > >::operator`class std::basic_string,class std::allocator >’, addr = 0x6126CA2E, last error = 123
[2932] Check exception: ..\dpgbase\regvalue.h (375), DPG::RegValue,class std::allocator > >::operator`class std::basic_string,class std::allocator >’, addr = 0x6126CA2E, last error = 123
[2932] Check exception: ..\dpgbase\regvalue.h (375), DPG::RegValue,class std::allocator > >::operator`class std::basic_string,class std::allocator >’, addr = 0x6126CA2E, last error = 123
[2932] Check exception: ..\dpgbase\regvalue.h (375), DPG::RegValue,class std::allocator > >::operator`class std::basic_string,class std::allocator >’, addr = 0x6126CA2E, last error = 123

Other notable programs are RssReader (ok, that is actually a debug release), Lookout for Outlook and Winamp, which dumps the entire playlist when it turns over.

Come on guys, it is only matter of conditional compiling.

Refactor As You Go

Extreme Programming (XP) has a nice principle called “Refactor as you go”. XP says that we should favor the simplest solution for a problem. I like the idea for a number of reasons, but most importantly for its cost-effective nature. Simpler design is faster to implement, results in less code (less bugs) and source code that is easier to read. “Designing simple” means designing for today’s problems; solutions designed for tomorrow often take ages to implement and output large amount of code to be tested.

This is not without price, however. Sometimes the design is proven too static for new requirements (see our last case) and often little attention is paid to modular design, reusability or consistency. This is where refactoring (improving code without changing its functionality) and “Refactor as you go” comes into the picture. The principle says that we should refactor to better code in small steps, as opposed to making big changes at once, because big refactorings are hard to do, open the door for new bugs and break at least one more XP discipline (“Small Releases”).

When programming methodologies and real life meet, you can feel the electricity in the air. Simple design serves short-term business goals very well (“get that new version shipped ASAP!”), but time-consuming and seemingly useless refactoring conflicts with them (“hey, why does it take two weeks to add a new report?”). It takes some self-discipline to take the Red Pill of refactoring, instead of the Blue Pill of the Commando Pattern. After all, it so much easier to get the job done quick, leaving tons of “TODO: Refactor” comments behind and it is so tempting to say that users care little about the software internals as long as it works. But then, points when you when you have to do a bigger refactoring always come.

Most of this week was spent with refactoring. Yes, it is a pretty big one, which means that there is seemingly no progress. ORF PowerLogs are still not fully integrated into ORF. The refactoring will take at least one more week, but even in mid-term, these changes will pay the price back, because the source code will be much easier to maintain. Not only PowerLog integration will be faster, but adding new tests to ORF or changing existing ones will take less time.

Stay tuned.

ORF Reporting: Design, Part 3: ORF PowerLogs

PowerLogs Raw CSV ScreenshotI gave you a brief introduction to ORF 3.0 PowerLogs in my previous article, now I would like to tell you more about the details.

PowerLogs aim to provide much more detailed information than the current ORF text logs do. Some of this new data is needed for reporting and others are merely for the user, such as the subject of the email. Let’s see what’s new:

  • Email subject. Yes, the logs will tell you the subject of the email :) This will help identifying false positives and emails in general. We have to admit that logging the Message-ID did not really work out, but we could not log the subject previously due to the ANSI charset limitation.
    Of course, this will only work at the On Arrival filtering point, as there is no such thing as “email” at the Before Arrival filtering point.
  • Complete test log of every email. This is required by Reporting—in order to generate test effectiveness reports for individual tests, we must be able to tell how many emails the IP Blacklist or the Spamcop DNSBL checked and caught.
  • List references. Got an email blacklisted by the Keyword Filtering, but you have no idea which one caused the blacklisting? No problem, the log will tell you which keyword filter it was. I mean, not only the keyword filter comment (if any), but the keyword filter itself. Similarly, it will tell you the related IP Whitelist, Sender Blacklist, DNSBL or SURBL expression. This also means that you will be able to generate reports about individual DNSBLs effectiveness (maybe not in 3.0, but in a later version—the data is there).

Some more unsorted facts:

  • Logs and list references (see above) will be stored in separate files (n:1 relation, n log files and 1 reference file) to save disk space.
  • Both files will be in regular CSV format, with Unicode UTF-8 encoding. No more “Unicode comment cannot be logged” complains :)
  • Timestamps will be in Coordinated Universal Time (UTC), to avoid Daylight Saving Time (Summer Time) problems.
  • Log file name format will be fixed and will always contain the date when the log was generated. ORF test logs offer flexibility in this regard, but unfortunately that flexibility adds lots of extra complexity to log processing . The date will be generated in UTC, which might be a bit strange for those whose time zone offset is more than 1-2 hours.

Obviously, the more data logged means larger log files. Lot larger. Due to this, real-time reporting with full log processing would have very poor performance. Our fine-tuned CSV reader has about 8Mb/s processing speed, which is nowhere near to the performance needed to generate yearly reports reasonably fast (it would take hours on a high-load server). Also, those who run servers with high load will have gigabytes of logs in a year, which they will barely keep, not even for reporting.

To reduce the time needed, ORF will generate preprocessed report files daily (or more often). As ORF users have to be able to generate reports for a specific time range, e.g. for Jan 1, 2006-July 1, 2006, we cannot just make an incremental report. Instead, full reports with a given resolution are needed and when the user requests a report for Q3 2006, generate a summary of these preprocessed reports between the above dates. Of course, preprocessed reports will also take disk space, so resolution might not be 1 second or 1 minute. As the range of reports and their exact contents are yet to be specified, the question of resolution is still open, but it may be 1 hour or 24 hours.

ORF Reporting: Design, Part 2

In my previous post, I told you about performance concerns of the ORF 3.0’s new Reporting feature. Alas, performance is not the only thing that slightly reminds me a design nightmare.

One significant problem with the current log format is that is does not contain enough data for reporting or the data contained is not in the required format.

  • Logging of specific ORF events can be disabled. Not only they can be, some are disabled by default, for instance, logging Before Arrival whitelist/accept events is disabled in a fresh ORF installation. This makes it impossible to compile reports about whitelists at the Before Arrival filtering point, because the reporting engine has no clue what happens Before Arrival, except if the email (actually, the recipient) was blacklisted, which is logged.

    These defaults and the customizability aimed to simplify reading the logs and saving disk space. They certainly do, but they break reporting as well.
  • Another scenario when data is missing is when we need individual test reports. Currently, ORF does not log what test was performed, expect if a hit (i.e. whitelisting/blacklisting) or error was generated by that test. We could make a pretty good guessing from the “Tests: “ log entries, but this would require sequential processing (see later) of the logs and not even in that case we would be informed why a hit did not occur (e.g. there could have been a list exception, like with the AD Exception List
  • Individual list items cannot be identified from the logs. If you ever want to see how many emails your new Viagra keyword filter caught, the current log format offers little help. The comment of the keyword filter is logged, but that we rarely seen comments assigned to the keyword filters. It is not really user friendly this way, so we need to tell what exactly the keyword filter was. The current log format offers no way to fully identify the filter, though.
  • Reports would need time-sequential log processing. While this sounds pretty easy, it has some challenges, because event date range cannot be determined from the file name for two reasons. First, the format can be configured by the user. Second, due to time zone changes, a log file can be re-opened, so it is perfectly expected that a log will have an event from 23:45 and the next event will be the same day 23:12.

    So we need to load all log files and sort all events into ascending order. Remember the performance concerns? Imagine sorting 365.000.000 event dates in real-time! In addition, some reports like Greylisting need the original order (time zone changes could break the original order).

For all the above reasons, you will not be able to generate reports from your current logs, unfortunately. We are working on a new log format (work name “ORF PowerLog” :), specifically built for reporting, but we plan this format to take over the role of the current ORF logs (not in 3.0 – the Log Viewer will not support PowerLogs, due to time constraints).

The benefits of the new logs are numerous, for instance, it will finally break the no-Unicode law and subject logging will be supported, but it will also provide full support to identify which keyword filter caused a hit.

The drawback is the size, of course. The more information need to be logged, the larger the log will be. We will do everything to make the new log format to be as compact as possible, but at least 200% growth can be expected.