Exchange Management Changes

One of the big changes in Exchange 2007 is less visible for the administrators: it was (almost) completely .NETified, i.e. rewritten in managed code. This change affects the management interface, too: previous IIS/Exchange versions could be managed using COM and ADSI (metabase), but Exchange 2007 offers a Windows PowerShell-based management interface instead.

In fact, PowerShell is the only way to manage Exchange 2007: the graphical MMC administration tools of Exchange also use PowerShell commands. For instance, when you add a new Accepted Domain using the Exchange Management Console, the console MMC actually runs the New-AcceptedDomain PowerShell command. Even refreshing the list of accepted domains is done using PowerShell, by submitting a Get-AcceptedDomain command and processing the command results.

Due to this dependence of Exchange on PowerShell, it is not surprising that ORF has to use PowerShell to manage the ORF 4.1 Exchange transport agents. For instance, to start using a new transport agent, you have to first install the agent using the Install-TransportAgent command (actually, cmdlet in the PowerShell terminology), then use Enable-TransportAgent to enable the newly installed agent and then Restart-Service MSExchangeTransport to apply the changes by restarting the MSExchangeTransport service.

And how does this affect ORF?

Well, among other things, it would look quite odd if you would have to start the Exchange Management Shell right after installing ORF and type five rather cryptic-looking commands to install the two transport agents used by ORF. To avoid this, we have to automatize the installation somehow. Another thing is that we need to query the status of the transport agents and display them in the ORF Admin Tool.

So, two ORF components are necessarily affected: the ORF Setup and the ORF Administration Tool.

We played with the idea of creating separate ORF versions for Exchange 2007 and IIS/Exchange 2000/Exchange 2003, but eventually we realized that it would be just pain in the ass for the ORF users, so any changes in the above components will be triggered by the ORF Connector Mode, i.e. ORF components will detect if Exchange 2007 is installed and act accordingly.

In IIS Connector Mode, the ORF Setup will install the ORF SMTP Module just like previous versions did. In Exchange 2007 Connector Mode, Setup will install the ORF Transport Agents (and ask the user whether restarting the MSExchangeTransport service is a good idea at the moment). Similarly, when updating, ORF will ask to stop the IISAdmin service in IIS Mode and to stop the MSExchangeTransport service in Exchange 2007 Mode. (Actually, it’s not that simple. See below).

The ORF Administration Tool will feature a reworked Status Information page for the Exchange 2007 Connector Mode. The new page will display the ORF Service status, the status of the ORF Transport Agents and a summarized status which tells whether ORF is ready to filter emails (based on the agent + service status).

To help the Setup and the Admin Tool, we will introduce a new ORF component called the ORF Agent Installer (orfainst.exe), a command-line tool written entirely in .NET (C#) that allows installing/uninstalling the ORF Transport Agents and also querying their status. A separate component was needed, because the ORF Administration Tool is written in a native language (Borland Delphi) and thus it cannot use directly use PowerShell, however, it can run the Agent Installer and process the output.

Once you will get ORF 4.1, you will notice that the ORF Administration Tool does not immediately show you the status of the ORF Transport Agents, it will take up to 10 seconds to get them. This is mostly because of PowerShell: it takes ages to load the base PowerShell environment, another year is taken by loading the Exchange PowerShell snap-in and then the Get-TransportAgent cmdlet runs reasonably fast.

Back to the installer for a short moment: one interesting question raised by the common Exchange 2007 and IIS/Exchange 2000/Exchange 2003 installer is what happens when someone runs ORF in IIS Mode and then upgrades to Exchange 2007, without uninstalling ORF? Obviously, ORF will stop working, because the ORF SMTP Module installed for IIS no longer connects ORF to the incoming emails. Our current plan to deal with this situation two ways:

• The ORF Administration Tool will detect the Connector Mode changes and will tell the user what to do to connect ORF to Exchange 2007
• The ORF uninstaller will take the above case into consideration and will stop services and uninstall components accordingly.

Some details of the above may change as we are still working on the necessary changes in ORF.

Transport Agents versus SMTP Event Sinks

Exchange Server 2000 and 2003 both used the IIS SMTP Service for email transport. ORF hooked specific events occuring the IIS SMTP Service using a programming interface called SMTP Event Sinks and this provided filtering both for IIS servers and Exchange 2000/2003.

Exchange 2007, however, no longer relies on IIS SMTP. Instead, it uses its own MSExchangeTransport service for email transport. Also, a new programming interface was introduced, called Transport Agents.

SMTP Event Sinks and Transport Agents are quite different beasts for programmers, because:

• SMTP event sinks are COM servers. ORF’s SMTP event sink is called the ORF SMTP Module, which is a lightweight component that connects IIS SMTP to the ORF Service. Event sinks can be written in any native programming language that supports creating COM servers—for instance, originally, the ORF SMTP Module was written in Borland Delphi and was just recently ported to C++ (for 64-bit compatibility).
• Transport Agents are .NET managed objects. This means that they have to be written in a .NET managed language, such as C# or Visual Basic.NET, but not in C++ or Borland Delphi.

…from this, it is quite obvious that ORF requires a new Exchange Transport Agent component, written in a managed language, but the list of changes does not stop here, because the Transport Agent API is different from the SMTP Event Sink API…

• Some events have no 1:1 mapping. The ORF SMTP Module hooks 3 events of the IIS SMTP. The first event is when the SMTP client issues the RCPT TO recipient specification command on an incoming SMTP connection. This is the Before Arrival filtering point. The second event hooked is when the email transfer was just finished on an inbound SMTP connection (On Arrival filtering point). The third, optional event is what ORF hooks when it has an outbound binding to an SMTP virtual server. This event is triggered when a MAIL FROM sender specification command is submitted by the local server on an outgoing connection. ORF uses this event to collect the outbound email sender / recipient addresses and to store this information in the Auto Sender Whitelist.

  Using Exchange 2007 Transport Agents, you can hook the first two events (using an agent subtype called SMTP Receive Agents), but not the third one, because there is no way to hook events occuring on outbound connections. For some reason, Microsoft has decided that this feature is not important to have in Exchange 2007.

  Consequently, ORF will have to hook a different type of event to monitor the outbound emails. This event type of called a Routing Event and it is invoked for both inbound and outbound emails. In this type of event, it is the responsibility of the caller to decide whether the email is outbound, which is not an obvious task.

• Some properties that ORF previously relied on do not exist. The ORF SMTP Module queries the SMTP authentication status and the authenticated user’s name from IIS to allow whitelisting of authenticated SMTP sessions. In Exchange 2007, the user name is not available and the authenticated status can be Anonymous, Organization and Partner. It is not entirely clear at the moment what these mean (yes, I did not do my homework :), but we suppose that anything other than Anonymous should be considered authenticated. It also looks possible to hook the authentication event itself to get the authenticated user’s name (where it makes sense), but most likely we will not do that—it would be too little gain for the price paid.

  There is another issue with the Pickup folder path that I will talk about in another article.

• And then there are new properties with no matching pair in SMTP event sinks, such as AntispamBypass, which “indicates whether the session is a connection with a partner configured for anti-spam bypass”. We are not sure yet how to categorize this, but this check will be probably part of the AUTH Whitelist.

  A very nice new property is called LastExternalIPAddress, which would have been terribly useful in the previous Exchange versions. It is basically what ORF determines on IIS using the Intermediate Host List. So far it seems we will not rely on this property, however, because it adds no extra value compared to the Intermediate Host List, which we will continue to use due to its other roles.

• Transport Agent changes are not as instant as SMTP Event Sink changes. IIS could install and uninstall SMTP Event Sinks on the fly, however Exchange 2007 requires changing the MSExchangeTransport service on virtually any transport agent change.

Of course, this above is only a limited list of the most visible changes only, but you will see more in the next article of the series when I will talk about the management changes.

From the ORF viewpoint, the most significant change that Exchange 2007 introduced is that it no longer relies on the Internet Information Services (IIS) SMTP Service for email delivery. Instead, it uses its own transport layer provided by the MSExchangeTransport service. As ORF integrates with IIS SMTP using a technology called SMTP Event Sinks, it cannot provide filtering for Exchange 2007 the same way it did previously.

Exchange 2007 uses a new integration technology called Transport Agents and ORF need to be changed significantly to be compatible with the new interface.

In this series series of articles I will talk about how the changes introduced by Exchange 2007 affect ORF and how do we deal with them.

Sorry, we still have no reliable ETA on ORF 4.1 with Exchange 2007 support as there are too many uncertain points in the development process. We are sorting out technical issues one by one, but with some of these we need assistance from the outside and the lack of official, guaranteed Exchange 2007 development support channels does not particularly help the development. Also, the numerous problems with ORF 4.0 allocated most of our development resources, not allowing us to focus on Exchange 2007 exclusively, but that is changing and I believe we are on the right track now. We are getting closer to version 4.1 and hopefully we can go beta soon.

The “Interface Not Supported” error occured first with ORF version 4.0.4, on a very limited number of systems, but it took us weeks to find and fix the problem. This post explains the technical background of the issues.

Although not mentioned in the ORF Change Log, ORF version 4.0.4 changed the way how the ORF SMTP Module and the ORF Service communicates. Previously, we were using named memory mapped files to pass email content around, but due to the nature of memory mapped files, this was not entirely reliable. We decided to change the transfer protocol to use IStream, so the ORF 4.0.4 SMTP Module actually passes two streams to the ORF Service, one for reading the email content and another for writing. As the COM marshalling of IStream is supported by Windows in every version and it is also quite performance-wise, we were happy that we found the perfect alternative to memory mapped files.

That is, until we started getting reports about decreased filtering performance and lots “Interface not supported” errors in the ORF log. It was discovered quickly that the issue occurred when the ORF Service failed to query IStream reference from the data got from SMTP Module, i.e. the stream did not arrive for reason. Of course, the issue did not reproduce on our test systems, so we had to compile special ORF versions and write test tools to gather more data. With the help of the customers who experienced the issue (thank you!), one of our test tools finally gave us results that helped to explain the issue: a CoMarshalInterface() call had failed with “Interface not registered” error. A quick check at the customer server revealed that the IStream interface registration was simply missing from the registry, the HKEY_CLASSES_ROOT\Interface\{0000000C-0000-0000-C000-000000000046} key did not exist and this effectively broke the IStream marshalling of Windows. The fix was very easy: we just had to re-register the broken Windows interfaces using the „regsvr32 ole32.dll” command.

Apparently, we are not the only one who ran into this issue. Most sources blame a specific version of Nero for removing these keys, but all of the customers who experienced the issue confirmed they never had Nero installed on their systems, so watch your step: there may be some software out there that does the same „magic”.