The role of the software architect is wonderful. As an architect, you face with a number of challenging design issues. You have to find the balance between time, resources, cost and requirements. You have to make compromises, even if you hate them. And at the end, you will blamed by the users :)

Now that I am designing the Reporting features of ORF 3.0, I find the role of the architect rather painful. If you have the strange passion to watch others making poor compromises, read on.

Due to the attention that we pay to performance, ORF is capable of filtering millions of email every day. It is not just marketing: we are aware of a few installations where ORF filters 1-3 million emails per day per server. I do not know what you say, but it is impressing for me. And it causes design headaches as well.

The new Reporting features in ORF 3.0 will be log-based, that is, reports will be derived from log data. As a single filtered email results in 1 to 2 or even more log entries, there would be about 365.000.000 to 730.000.000 log entries in a yearly report for a large client. In our logs (Verbose mode), the average log entry length is 190 bytes, so the size of the logs to be processed for a year will be around 69,350,000,000 – 138,700,000,000 bytes–that is, 64.6Gb – 129,2Gb.

That is awful lot of data to be processed. The Short log mode reduces this a little, but not that much. The log parser of ORF is optimized for speed, so its has pretty high throughput: about 22.07Mb/sec on my developer workstation. This means that parsing 64.6Gb log will take at about 2997 seconds or 50 minutes; in the less lucky case about 2 hours. And it is just parsing, transforming data for the reports will add significant delay. Alas, I do not really see how this could be made faster. Lots of information takes lots of time to be processed. Sorry guys, you will have to wait for your reports.

However, 90% of our clients fall in the 0-10,000 emails per day category, so most will have logs smaller by factor of 100 and 3.65 million entries will parse in 30 seconds. Or, if the client has faster disks, maybe even less – yes, the disk I/O can be a bottleneck.

This was speed. In my next post, I will talk about the significant concerns.

The feature set of ORF 3.0 is finally getting a solid shape and we are going to publish the development roadmap for 3.0 and maybe the version(s) after.

As the first ORF Software Maintenance Agreements are going to expire on December 1, 2005, it is pressing to let our clients know that renewing the agreement is a good investment: they are going to get improved software for their $99. And definitely we have to give 3.0 a deadline and tell about the expected features.

Now let us forget about how much I am concerned about keeping the deadline (man, it is tight!) and about feature selection, but I am also concerned about our competitors: is disclosing our plans to the competing companies a good idea?

Not that we have some mysterious super-secret technology on stock that is going to change spam filtering for once for all (but we do have a few good ideas :), it is just that our competitors are usually larger than us. I mean, larger by factors. They can easily do in weeks what we do in months and so they can entice our clients by giving them the features earlier that we want to give.

Regardless concerns, I think it is pretty much clear that our clients are first, we must tell them what they are going to get for their money. Competitors are our concern, not theirs. I feel quite naked, though :)

Come on Borland, you can do it better. From the Delphi 2006 page:

• The ultimate productivity force-multiplier for Windows® development
• Ensure visual consistency with customizable VCL designer guidelines
• Easily deploy persisted applications with executable state diagrams
• Seamlessly reverse-engineer, migrate, and re-use existing code