Preprocessing Speed Improvement

ORF PowerLog preprocessing works with large amounts of unsorted data and does this on “idle” priority (that means, PowerLog preprocessing will run only if your CPUs has nothing better to do). It is not entirely surprising that it is slow, but if your server deals with 1.000.000 emails a day, your daily PowerLog files are 500Mb large and preprocessing of a single PowerLog file takes some 19 hours, I guess you do not find solace in this. Neither did we, so optimizations were made and now ORF processes the 500Mb and 1.000.000 emails PowerLog .OPG file about 75 times faster, in about 15 minutes instead of 19 hours (measured on our test system). The improved preprocessing will be available in the next version.

3 thoughts on “Preprocessing Speed Improvement

  1. Kurt (EC)

    As Peter will know (unless he’s suffering from Alzheimer, he will, as he’s been sending this link :))

    Just one word for it:
    G R E A T!

  2. Justin

    How soon can we expect the next release? My dual-CPU hyper-threaded servers (3) are not able to keep up with inbound load and finish processing the log files on most days now. There is some idle CPU, so I suspect the improvements will allow us to get back to completing processing again. (we are losing about 2 or 3 days each week due to too much log data to process.)


  3. Peter Post author

    Justin, the new version is expected for mid- or late spring 2007.

    If you have troubles preprocessing your logs, we can do the preprocessing for you. Contact us at to discuss the available options.

    By the way, the reason you are getting idle time is that preprocessing runs on one thread only, so on a dual-processor machine, the highest processor usage will be 50% only.

Leave a Reply

Your email address will not be published. Required fields are marked *

AlphaOmega Captcha Classica  –  Enter Security Code