Skype Support Experiences

A few weeks ago, we decided to test whether providing support via Skype would be a more popular (and suitable) way to contact us regarding ORF problems than our regular support lines (phone, email and our newsgroups). So I started to look for a good headset for this purpose, but surprisingly, neither of the manufacturers keep in mind that somebody will wear this thing for 8 hours straight… I read a hundred reviews but all headsets have this common problem: even the most expensive models are very uncomfortable to wear. OK, their sound is loud, clear and cool, but I do not really care if my skull is in a clamp for hours. I prefer large earpieces which are large enough to cover my (also large) ears on my large head. Maybe I am not part of the target audience, but even large headed people should use Skype sometimes :) (note to self: market gap)

Finally, we decided to buy a Sennheiser PC 135 USB headset. It has all the problems I mentioned above, but it will do.

The next step was creating a Skype account for the support calls and indicate our status on our website. It was easy to setup by following the steps described in the Skype FAQ. The final result was a link, which should indicate our status, but it does not: it shows me online even if I am offline. Great!

But at least our users could give us a call or chat with us about ORF problems, and we thought it would be a good thing none the less. So we waited for the first call and… in a couple of hours later, we solved the first problem via Skype, and some others later. Most of the callers preferred chat instead of talking, which is good, considering my terrible Hungarian accent. A couple of days later I started to see some weird things: Skype notified me about unanswered calls (from the US) from 4 days before. Why didn’t it notified me on the day after the calls took place? I have no idea… We solved these problems via email by the time Skype notified me, so it does not really matter, but still: it does not work the way it should.

As our company is located in Budapest (Hungary), at least the Skype calls are cheaper than calling us from the other side of the globe, but it cannot provide a solution for the timezone differences: if an ORF-user calls us when nobody is in the office, it doesn’t matter how he calls us… Additionally, we did not receive more calls via Skype then via phone, so it seems that the price of the call is not really a deciding factor.

Peter in the background

I think we will keep this channel open for our users, as even the low number of calls worth it if the result is a satisfied customer. (Oh, it sounds so cheap, but that’s the way it is :))

ORF 4.0 Database Changes

ORF has two major issues with the Greylisting and the Auto Sender Whitelist databases:

  • Sharing: databases cannot be shared between ORF instances
  • Reliability: the database engine is not rock solid on some systems

Sharing is a simple case. Database sharing would be needed on networks with more inbound email servers: it is no fun when an email gets temporarily rejected two times by Greylisting, just because the left hand doesn’t know what the right hand is doing.

Reliability is a bigger issue. Only few systems are affected by engine reliability issues (namely, database corruption) and we found no common points between them, also the issue never reproduced in our lab. Under unrealistically high direct engine load, we could get some database operations to fail, but we could never achieve anything as serious as a database corruption. Still, the fact that we could drive the database into failures indicates that something is wrong with the thread safety of the database engine.

In ORF 4.0, we will address both of these issues.

To fix the reliability problems, ORF 4.0 will limit all database operations to a single thread only. The engine is proven robust with single threaded use and although this limitation reduces the throughput, the database performance will be OK for most smaller systems. Typically, ORF operations on complete well within 1 second (even on a large database), so 60 * 60 * 24 = 86.400 operations can be performed a day, enough for approx. 50.000 emails/day. It is just rough estimation, but I think the actual figures will be better and for the question of scaling, the answer will be the same as for the other major issue.

Database sharing will be implemented using external databases, with support for the following servers: Microsoft SQL Server (MSSQL) 2005 Express Edition, MSSQL 2000 and MSSQL 2005 (initially). The benefits of this model are numerous:

  • allows sharing the databases between ORF instances,
  • provides high performance,
  • can scale well (SQL Server Express -> SQL Server),
  • allows programmatic access to Auto Sender Whitelist data.

The price of choosing SQL servers is that the databases have to be created manually. We will provide downloadable guides with setup instructions and the necessary SQL statements, but running database servers requires some field-specific knowledge (e.g. adding users, configuring network access, etc.) that we cannot fully eliminate by guides.

ORF No Longer Hangs on Windows Vista

This has turned out to be a lot more interesting issue than I expected. I was thinking about contacting MS PSS and so I wrote an as-simple-as-possible test application to reproduce the issue. Surprisingly, the test app worked flawlessly on Vista, so started investigating the differences.

First, my test application was using the HTML Help API (hhctrl.ocx) directly and was statically linked (which means that Windows would not allow the test app to start if hhctrl.ocx is not available on the system path). ORF, on the other hand, uses a third-party library that loads hhctrl.ocx dynamically and uses some registry magic to determine the location of hhctrl.ocx, regardless the system path.

My initial thought was that the ORF Log Viewer and the test app was using different versions of hhctrl.ocx and the Log Viewer found the broken one. To test my theory, I started both applications and checked the loaded DLLs by Process Explorer just to find that Log Viewer doesn’t even have hhctrl.ocx loaded. Wow, that makes sense! Calls to hhctrl.ocx will hardly work if it’s not loaded. Actually, our calls to HtmlHelp() ended up in the middle of invalid memory.

I checked the library code again, now digging deeper in the loader code and found that it tries to determine the location of hhctrl.ocx by checking its COM registration under HKCR\CLSID\{adb880a6-d8ff-11cf-9377-00aa003b7a11}\InprocServer32. Under pre-Vista operating systems, the default value for this key used to be simple String Value (REG_SZ), containing e.g. “C:\WINDOWS\system32\hhctrl.ocx”. On Vista, however, this is an Expandable String Value, pointing to e.g. “%SystemRoot%\System32\hhctrl.ocx”. The library fails to check for this and to expand the the value into a real file system path, so it also fails to load hhctrl.ocx. Even worse, the library does not offer proper error handling, so the caller application is not aware of the failed load and calls the trusted HtmlHelp() call, resulting in some accidental endless loop.

End of story, library fixed.

If you can’t wait for ORF 4.0 to run the Log Viewer and the Reporting Tool on Vista, you can apply a manual fix as described below:

  • Run regedt32.exe as Administrator (%SystemRoot%\System32\regedt32.exe)
  • Open the HKEY_CLASSES_ROOT\CLSID\{adb880a6-d8ff-11cf-9377-00aa003b7a11}\InprocServer32 key
  • Change the %SystemRoot% variable in the value to C:\Windows (or what is your local system root path)

Expect a little fun with permissions. Ah, and don’t forget to change it back to %SystemRoot% when ORF 4.0 ships—I am not aware of any problems with hardcoding the system path, but who knows.

ORF Hangs on Windows Vista

No big deal because there is no point installing ORF on Windows Vista: the IIS SMTP component has been removed from IIS 7 of Vista (rant below). Yet even if you do not install ORF on Vista, you may try running the ORF Log Viewer on your brand new OS, but what you get is a program that eats 100% of the processor time and has to be shut down from Task Manager.

I’ve done some debugging and found that the ORF Log Viewer (and the ORF Reporting Tool) hangs on the HTML Help initialization:

HtmlHelp(0, nil, HH_INITIALIZE, dwCookie);
(see MSDN)

It is a Microsoft API that should just work, but it does not. Any ideas?

And now, the promised ranting (I love ranting! :). Why IIS SMTP had to be removed from IIS 7 on desktop Vistas? According to a Microsoft blog post , the whole point of shipping IIS 7 with Vista before Longhorn Server is released is to make life of developers easier. So why they actually make our life harder? As the blogger points out, developers typically use desktop Windows versions for development and testing. We do not run server operating systems on our desktop computers. Now if we decide to upgrade to Vista for developer desktop, we will have to install Windows 2000/2003 Server or XP into a virtualization solution, testing code in the virtual machines. That is big PITA, so Microsoft, I am not happy for your decision.

Test Exception Design

Ever since the Active Directory-based recipient validation was implemented in ORF, we receive requests to make it exempt from whitelisting, i.e. to reject the invalid recipient even when e.g. the sender is whitelisted.

The request is perfectly reasonable, yet it does not play very well with ORF.

Here is the problem: ORF is primarily a spam filter, but some of its features have email security role more than anti-spam. These tests are the AD Test, Recipient Blacklist (for recipient validation), Attachment Filtering and External Agents (agents filtering for viruses). Regardless role, these are blacklisting tests, so whitelists take precedence over them. This results in hardly reasonable scenarios like letting a virus-infected email enter to your system, just because the sender is (auto) whitelisted.

Obviously, we have to deal with this problem somehow. We went through to quite a few possible solutions, but unfortunately none of these were without tradeoffs.

First, we cannot just re-prioritize these tests to run without whitelist control. Remember that ORF monitors all incoming SMTP connections, including SMTP connections that are used to relay outbound emails. If no whitelisting mechanism is employed, ORF filters outbound email as well. That may sound acceptable, or maybe even great for Attachment Filtering and External Agents, but I am quite confident that you do not want the recipients of your outbound emails validated as local addresses. A few would pass.

Ok, so we still need to exclude outbound emails from these tests. Good news: ORF already does this. Bad news: it does it a weird (but otherwise practical) way.

The best way to tell whether the email is inbound or outbound is to check the recipient domain and if it is a domain that your server is responsible for, the email is inbound, otherwise outbound. Cool, but you might have noticed that ORF does not keep track of the local domains.

Instead, ORF uses two whitelists which almost accidentally eliminate the need for outgoing email detection. The first is the AUTH Whitelist that whitelists emails arriving via authenticated SMTP connections, thus it covers authenticated outbound relaying. The second whitelist is the IP Whitelist, which

  • Whitelists email from the local host
  • Whitelists emails coming from Intermediate Hosts, which always includes the private intranet address ranges.

These two IP Whitelist rules cover the typical outbound relaying cases, and when not, it is enough to add relaying server’s IP address to the Intermediate Host List.

Excellent, but there is a glitch: the Intermediate Host List is not for detecting outbound emails. Typically, you add your secondary MXs and front-end servers to the IHL, allowing ORF to “look behind” these in the delivery path and check the actual source IP (DNS Blacklists, SPF Test, etc). In other words, the IHL-based IP Whitelist rule tells you whether the email is coming from a trusted source and this covers your trusted outbound relay hosts, too.

So a design decision had to be made: either (A) we require the user to list the local domains for proper
inbound/outbound email detection or (B) run the AUTH Whitelist + IP Whitelist always, even when email security tests are exempt from whitelist scope.

After long consideration, we decided to use option (B). Here are the advanteges and disadvantages:

Advantages:

  • No administrative overhead
  • Exceptions for email security tests can be enabled by default. This is important because many ORF users actually think that ORF already works this way, so they will not seek a configuration option to enable exceptions.

Disadvantages:

  • If you have a front-end server, you will not have recipient validation at the Before Arrival filtering point. It is not that you had that before, but technically you could have it now.
  • Your front-ends, secondary MXs and hosts listed on the IP Whitelist or anyone who successfully authenticates with your SMTP server will be able to bypass these email security tests.

We believe this is still better than requiring to list all domains and maintaining that list (sure we can detect the domain list, but eventually you would have to at least confirm that list and maintain it later).