Refactoring done, PowerLog support is now integrated. We will release the first development version of ORF 3.0 to the Feature Test Program members next Monday. It is an important milestone, even though there is awful lot left to do.
Now that the Reporting feature is getting its final shape, we see that many report types requested by our clients will not get into 3.0, due to the compromises that we had to make between report availability and customizability. A good example is per-user reporting—that is, telling how tests worked for john@example.org and jane@example.org. Each report breakdown (per server, per virtual server, per recipient domain, per recipient) multiplies the amount of data to be processed, which affects report availability negatively.
Sure, it is impossible to meet everyone’s expectations, but can we do anything to allow clients to generate custom reports?
The first and most obvious answer is SQL logging. Once you have the data in a SQL database, you can do just about anything with it. Also, many administrators speak SQL to some extent. Can anything be wrong with this?
Well, yes. First, ORF PowerLog entries are quite complex structures and SQL logging would need some some 30+ tables (imagine your SELECTs). Second, you would probably have too much data to be queried. Third, (technical) supporting such feature would be crazy.
Ok, drop SQL logging and let’s see the next idea. What about providing the raw data to the interested clients and let them decide what to do with it? After all, this is what ORF will do internally. ORF will read the log entry, add 1 to the “Number of Tested Emails” counter, add 1 to the “Number of IP Whitelist Tests” counter, etc. and step to the next log entry. If you could do the same, you would have total freedom with data processing. Great, isn’t it? Can anything be wrong with this?
Well, yes. As you may already have figured out, this would require some programming skills and the number of administrators who can write scripts is much lower than those who master SQL. Sure, we can turn this into a mega-project (as opposed to publishing the log reader as a script-accessible COM API) and offer some smart GUI to reduce the code load, but in the end you would have to code a little for the custom reports.
Anyhow, if you are interested in this way of generating custom reports, post a comment. I am just thinking aloud, but let me know what you think.
Are you saying that the log will not be available in a CSV format?
I was kinda hoping that I could rewrite the tool I wrote previously. This sets an event watch on the logging directory and every time ORF writes to a file or creates a new one, it reads from the last book-mark.
Although it does not have an SQL database, it loads everything into a typed ADO.NET Dataset, in effect a memory resident SQL database. This is what we use in applications these days, we bind controls directly to the tables. When finished we send everything back and ADO.NET follows the referential constraints to update the back-end in the correct order.
The tool I wrote already has a very complex schema. I see no issues with what you are describing, in fact I spend a lot of my time rewriting badly written SQL to make it work like a rocket.
I was looking at a badly writen report yesterday that had a SQL statement that went on for about 30 pages with aggegates nested with aggregates etc. In terms of readability (i.e. indenting and aligning the levels) it was a dog’s breakfast. More importantly it was not doing what the spec said it should.
It had to deal with a database over 20 GB in size, i.e. very much bigger than my ORF logs to date.
The good thing is though that YOU won’t have to support it, I’ll be doing that for myself, heck if it bugs you I could even agree to make it available to third parties.
I would only provide a subset to keep support costs low. Just a simple scrolling grid showing the log live with an SQL back-end so that it does not have to re-load from text each time and leaves behind SQL data.
I might also provide a few example SQL Queries for user written reports and respond to questions on the NG.
In big Dot Net projects we use some open source software from Microsoft called the Enterprise Library. Basically a set of interlocking patterns and practices with application blocks (source code).
These include: SQL Access, Configuration management, Logging+Instrumentation, Encryption, and Exception policies.
Logging starts by writing out a message somewhere in your system. There are several classifier fields but the bulk of the message goes into a single field usually as formatted text.
The configurable distribution system then reformats these and sends them to a set of configured sinks. Examples include NT Event log, a Text Log, an Email log, an SQL Table, the WMI.
Apache developed a similar system called log4net. They call sinks appenders. There are some additional ones, in particular Syslogd: some of your clients use Syslogd to collect logs.
Log4net is part of a wider project. There are versions Cxx and Perl etc. The c++ can be called from Borland C++ 5.5 and my guess from within Delphi.
Configuration is via an XML file. Unlike .NET there is no configuration tool, you need to update it with notepad to set up the “appenders” and “filters” etc.
I would have thought that this is the way you would want to go. It would make ORF configurable to report to a text file as before for use in your reporting tool, as well as to any SQL database server and Syslogd.
The log table is very simple and flat, however it could still be normalised by an independent process into other tables for detailed analysis of the micrologs.
No, logs will be available in CSV format, but it does not mean that you can process them without knowing what the data means. It is like knowing that a picture is in JPEG format: it does not tell if it is a photo of a dog or an oil paint of a flower.
I know about the log4x projects, they are powerful indeed, but even if there would be a stable Delphi port or if we would create a DLL wrapper above e.g. log4net, I do not really see how they could help. Anyway, ORF works internally much like log4x, but PowerLogs require different data to be logged, that’s why it took much longer to add them to ORF.
What I outlined above would provide easier programmatic access to the logs than manual CSV processing, because you’d access log entries version-independently via a simple COM interface, as opposed to “hacking”.
log4x allows you to define your SQL table any way you want, i.e. add columns for all the fields in the current logs plus another for the microlog. Plus it adds appender dependent formatting definable in the config, and appender dependent filtering.
I don’t think too many of the ORF users would be into hacking into the table the way I do, but quite a few would appreciate being able to log to Syslogd or SQL or even live via TCP to a watcher. There are so many appenders to choose from.
If you already have an internal logger, then all you need to do is interface to the C++ at one point. Plus you get the source code to fix if it is wrong.
As for COM interfaces, I could write many very useful ORF add-ins if you could agree on an interface. For example over the weekend I was playing with the .NET based DKID open source kit from CERN. It is not much use in its current form because it does not register with the SMTP COM Interface for event notification. I could easily wrap it in a COM interface though, if ORF had a standard interface for authentication events and signing events. I could even write an authenticator based on other strategies. Stiil this is a long way off.
I think that a log4x COM+ events appender would have its uses. If I could catch when greylisting happens on one server and call a COM interface on the other to register in the same in DB over there that would be nice.
Sure, but IMHO the bottleneck here is not the storage media, but turning raw data into information that makes sense.
The proposed COM interface would work much like a data source (in the ODBC/OLEDB sense). In fact, an OLEDB provider was written for ORF logs a few years ago, but it never went official. Alas, due to the structure of the data stored, an OLEDB provider would make little sense here. What I have been thinking of is something like the below, in JScript-like pseudocode.
var parser = new ActiveXObject(“ORF.PowerLogParser”);
parser.collectLogsBetweenDates(yymmddhhnnss, yymmddhhnnss);
while(event = parser.readNextEvent()) {
if(event.IsMailEvent) {
for(var x = 0; x
Mmm, my post got truncated but I guess you get the idea.
I do, but it has no event call-back so I would still need to hook directory writes, and the only thing that is really different is that I don’t have to do a first level parse or file roll-overs. Some of your clients might find it helpful though. I am sure someone will release a simple ASP page that uses it so that end users can look for missing mail.
Storage in a DB has its advantages, slightly easier to query compared to PowerGrep and you can purge old data more selectively.
If you are running multiple servers you can channel them back to an SQL cluster. Many MS products like ISA2004 have SQL Server as an option, allows you to manage logging space and reporting in a single place. You should see some of the reports I have written based on ISA2004 logs.
Still I guess you are out of time on this one and need to move on to the other items on the roadmap.
A couple of questions on the PowerLogParser
1. It is a DLL not the ORF Service?
2. You use it in your own log viewer?
On another issue, have you ever considered publishing COM+ events from the ORF Service?
Back in 2002 we developed some very intense systems that used COM+ eventing, there were quite a few issues I helped MS sort out.
Compared to having to set up ConnectionPoints and manage subscribers, COM+ event publishing is trivial.
1. Create a DLL or simply a type library that defines the call you want to make when you rire an event.
2. Register the DLL in the Component Services MMC (can be easily packaged to distribute in the MSI)
3. Open the Interface and call the method to fire the event.
COM+ takes care of everything else, i.e. distributing the call to all the subscribers, allowing subscribers to register even when you are not publishing etc.
Some rules about the event methods, they must not return anything and all parameters must be passed by value.
So if you could define a simple method to fire when log a message, I could hook it and send to a configurable logging component.
Ideally the event handler should be configured in Component Services as a queued component i.e. your call returns immediately and the message gets buffered by MSMQ.
Using COM+ events it is still possible for the subscriber to send back a reply. To do this the subscriber must be a non-queued component. It makes an inward call into the publisher on another interface. This could be a way of communicating with the CERN DKIM component.
Indeed, this would not be real-time, but what you are proposing is actually a log plugin support as opposed to a programatic access to logs. Scripting support for custom tests / logging came up several times in the past, but we seen little interest in these compared to other features — features with high P/E enjoy priority to low P/E features.
As for PowerLogParser; it is not something we plan, but something we are thinking about. I am not sure yet how this could fit into our development plans.
What I have been thinking of is an in-proc COM (OLE) server, which can be accessed from scripts (VBScript, JScript, Perl, Python, etc) and any COM-supporting languages (from VB to C#).
Currently we do not plan turning ORF into a COM+ server.