Well, this had to come. When we compiled the feature set for ORF 3.0, my estimation for Reporting was 35-49 days—making it the biggest single feature we ever added to ORF. This estimation assumed that reports will be generated from the current logs, but later several problems were identified with this concept. We faced with a decision: either we will generate reports that A) are useful, or just generate B) some eye-candy summary. We decided to go with the option A), which required a new logging medium and lots of related work, doubling the resources needed. This is reflected in development: we are about to reach the 49 days limit and Reporting is about halfway to be completed.

It became obvious that Reporting will predate other planned features of ORF 3.0. We have updated the ORF Development Roadmap according to this.

We apologize for this feature creep, I know many of you expected these features much. We will do our best to get them into ORF as soon as possible.

Refactoring done, PowerLog support is now integrated. We will release the first development version of ORF 3.0 to the Feature Test Program members next Monday. It is an important milestone, even though there is awful lot left to do.

Now that the Reporting feature is getting its final shape, we see that many report types requested by our clients will not get into 3.0, due to the compromises that we had to make between report availability and customizability. A good example is per-user reporting—that is, telling how tests worked for john@example.org and jane@example.org. Each report breakdown (per server, per virtual server, per recipient domain, per recipient) multiplies the amount of data to be processed, which affects report availability negatively.

Sure, it is impossible to meet everyone’s expectations, but can we do anything to allow clients to generate custom reports?

The first and most obvious answer is SQL logging. Once you have the data in a SQL database, you can do just about anything with it. Also, many administrators speak SQL to some extent. Can anything be wrong with this?

Well, yes. First, ORF PowerLog entries are quite complex structures and SQL logging would need some some 30+ tables (imagine your SELECTs). Second, you would probably have too much data to be queried. Third, (technical) supporting such feature would be crazy.

Ok, drop SQL logging and let’s see the next idea. What about providing the raw data to the interested clients and let them decide what to do with it? After all, this is what ORF will do internally. ORF will read the log entry, add 1 to the “Number of Tested Emails” counter, add 1 to the “Number of IP Whitelist Tests” counter, etc. and step to the next log entry. If you could do the same, you would have total freedom with data processing. Great, isn’t it? Can anything be wrong with this?

Well, yes. As you may already have figured out, this would require some programming skills and the number of administrators who can write scripts is much lower than those who master SQL. Sure, we can turn this into a mega-project (as opposed to publishing the log reader as a script-accessible COM API) and offer some smart GUI to reduce the code load, but in the end you would have to code a little for the custom reports.

Anyhow, if you are interested in this way of generating custom reports, post a comment. I am just thinking aloud, but let me know what you think.