Exchange 2013 CU9 Health Probe woes

You probably know the Managed Availability feature of Exchange 2013, which sends periodic health probe emails to check server health. You probably never wondered, though, why these emails are omitted from the ORF logs. And you are right of course, it just works, so why bother? That is, until it doesn’t and health probes start showing up in the logs to add a headache-inducing noise every couple of minutes.

Health probes showing up in the logs (pre-CU9, detection disabled)

Health probes showing up in the logs (pre-CU9, detection disabled)

To save you from the pain, ORF detects the list of reserved Heath Mailboxes during installation and automatically ignores any email sent to a health mailbox address. This prevents these emails from getting into the logs and your sanity is preserved.

Unfortunately, this only lasts until you upgrade to the 9th Cumulative Update of Exchange 2013 (CU9), which breaks this feature and then you get this:

Health probes in the logs after CU9, without recipient information

Health probes in the logs after CU9, without recipient information

This is due to a new Transport Agent introduced by CU9 called the System Probe Drop Smtp Agent — undocumented, but apparently responsible for dropping probe emails — which does not “drop the email” in a traditional sense, but removes all email recipients instead. Alas, this method of stopping an email from getting delivered has an adverse effect, namely the email still gets passed down the pipeline, but now with the recipient information destroyed. As ORF’s health probe detection is based on the recipient address, it also renders this feature useless.

When can I expect a fix for this?

We are still evaluating the best way to deal with this. We plan to address the issue in the next regular release. As this potentially affects other third-party Transport Agent vendors as well, we are also reaching out to Microsoft to discuss this matter.

Update July 31, 2015: The upcoming beta version of ORF 5.4 will arrive with a fix for this problem.

Is there a workaround?

You can set up a filtered view in the ORF Log Viewer for the Sender field (e.g. a regex for (HealthMailbox.*@yourdomain|[email protected]\.com)$ with rule inversion).

Can’t I just change the agent priorities?

Unfortunately, no. The System Probe Drop Smtp Agent hooks the OnEndOfHeaders event, which always occurs before the OnEndOfData event hooked by ORF.

I don’t have CU9 and still get health probes in the log

You probably added a new Mailbox server after ORF was installed and you need to update the list of Health Mailboxes. Read more on this in our Knowledge Base article.

Snowshoe spam recommendations

If you have experienced a sudden rise in spam reaching end-users recently and you are located in the USA, you might be affected by snowshoe spam. This article tells you how to configure your ORF to deal with this type of spam.

How do I tell if I am affected?

Open a recent log file in the ORF Log Viewer and sort the events by the Related IP column. Scroll through the sorted list of events and look for large, continuous network blocks (e.g. a /24) with spammy email subjects. If the sender email addresses for these look like [email protected], you are affected. Otherwise, the increase in spam may be due to a technical issue and you should contact our Customer Service to investigate.

Typical example of a snowshoe campaign in the ORF Log Viewer.

Typical example of a snowshoe campaign in the ORF Log Viewer.

How do I stop this campaign?

As of writing this, the Greylisting Test of ORF can stop the campaign easily (when configured a certain way).

  • Open the Blacklists / Greylisting page in the Administration Tool.
  • Make sure the test is enabled (see “On” next to the page title).
  • Disable the “Accept delivery retries from the same /24 subnet” option
  • Disable the “Skip Greylisting if the sender explicitly passes the SPF Test” option
The recommended ORF configuration for the Greylisting Test against snowshoe spam

The recommended ORF configuration for the Greylisting Test against snowshoe spam

Greylisting is an aggressive measure that delays emails from unknown senders for 5-15 minutes. We are working on a less aggressive technology against this type of spam. Meanwhile, make sure you also have the Auto Sender Whitelist enabled and assigned to the Before Arrival or both filtering points.

Why do I have to configure ORF this way?

The campaign is run remarkably well compared to the average spam. Instead of using a network of compromised computers (botnets), the spammer abuses reputable hosting companies with a clean IP reputation for very short periods of time (2-4 hours) before it moving on the next provider. The usual heuristics, such as reverse DNS and SPF checks also check out.

We suspect that custom spamware is used for email delivery which Greylisting fends of well. However, as emails arrive from a continuous range of IPs and SPF is properly set up for the spamming domains, Greylisting defaults needs to be adjusted for this campaign.

SPF Syntax Validator

SPF Validation Service Updates

Our SPF Syntax Validator and SPF Policy Tester tools have just received a major overhaul and I am happy to report that both services are now fully compliant with the latest RFC7208 version of the SPF standard. Not only that, both services have received full IPv6 support and feature a much improved syntax validator that catches more errors and raises more warnings to help you troubleshoot SPF problems in no time.

SPF Policy Tester in action

Just like the previous versions, the most recent incarnation of our SPF services is powered by Vamsoft’s own SPF client library, the very same one used in our ORF email security product — which means the changes are coming to ORF, too. Stay tuned for the next version!

Gory technical details in the changelog below:

  • All SPF services updated to RFC7208 from the previous SPF-C RFC draft version.
  • Complete IPv6 support.
  • Live evaluation logging.
  • Various validation improvements:
    • Domain-spec type arguments are now validated during syntax check time, as long as they are macro-free.
    • Warning during syntax validation if more than 10 DNS-based SPF terms are used in the policy (RFC 7208, Section 4.6.4 defines a hard limit of 10 such terms during evaluation).
    • Extensive logging of error scenarios for the “exp” modifier, including syntax errors, DNS errors.
    • Improved IPv4 validation for the “ip4” mechanism: the octal notation of IPv4 addresses is no longer accepted and a hint is offered if the IPv4 address is valid, but does not conform the IPv4 dotted-quad decimal notation.
    • Warning when a “redirect” modifier is present somewhere in the string along with an “all” mechanism. In this case, the “redirect” modifier will never be used, because “redirect” is only used when all mechanisms fail to match and “all” always matches.
    • A warning is now raised if duplicate mechanisms are found in the policy string.
    • Components of variable-length macro-expands are now verified for positional issues (digits, followed by “r”, followed by delimiters).
    • Warning during syntax validation if a macro-free domain-spec argument is detected to end with a trailing dot (ambiguous).
    • Maximum domain length for extended domain expressions is now enforced.
    • Warning raised if the SPF policy length exceeds 450 characters (see RFC 7208, Section 3.4.).
    • The expanded “exp” modifier string is now validated for character set and length as per SMTP rules in RFC 5321.
  • Various changes as required by the RFC update:
    • Unknown mechanisms now trigger an error. Previously, such mechanism only triggered a warning that reaching the unknown mechanism will stop the processing with an error. Fun trivia: there is no such thing as an unknown mechanism as per the RFC, but an unknown/invalid SPF term may look like an SPF mechanism.
    • Unescaped “%” in macro-expands are now treated as a syntax error (original RFC required treating them as a literal, RFC 7208 requires it to be treated as syntax error).
    • A warning is now raised on the use of the “ptr” mechanism. This mechanism should no longer be published, as explained in RFC 7208, Section 5.5.
    • The macro letter ‘r’ is now only allowed in the “exp” modifier (RFC 7208 limitation).
    • A warning is now raised on the use of the “p” macro letter. This macro letter should no longer be used.
    • RFC change: Trailing dots are now accepted in domain-spec arguments.
    • Bugfixes
      • The cidr-length 0 is now allowed by the SPF parser (it wasn’t allowed previously). A cidr-length of 0 is explicitly allowed by the RFC ABNF.
      • Cosmetic bugfixes

A New Showshoe Spam External Agent

UPDATE: The External Agent discussed in this article no longer works against the snowshoe spam campaign discussed below. Visit our latest article regarding this type of spam.

In the past few days, we have received reports of a snowshoe spam campaign hitting many of our US-based clients heavily. The campaign is characterized by a sudden influx of spam in high volumes from a continuous range of IP addresses. Outbreaks last for 2-4 hours before the spammer moves to another provider. Due to the nature of this campaign, the usual heuristics provided by ORF do very little to stop these emails.

We are now releasing a quick relief External Agent to address this campaign specifically. The agent relies on DNS-based heuristics to safely identify and blacklist these emails.

A typical example of the snowshoe campaign in the ORF logs.

A typical example of the snowshoe campaign in the ORF logs.

Is it for me?

You only need this agent if you are affected. If you are located in the US and suddenly started to see spam coming through ORF in numbers, you can verify if you are affected by opening a recent log file in the ORF Log Viewer and sorting the events by the Related IP column. When you scroll through the sorted list of events, look for large, continuous network blocks (e.g. a /24) with spammy email subjects. If the sender email addresses look like [email protected], you are affected and you should install this agent.

How do I install this?

Download the agent from

http://vamsoft.com/downloads/agents/agent-snowshow-c15q1-01/agent.zip (95kB ZIP)

and extract the archive contents to any local directory on your server.

If you have used Windows Explorer to extract the archive contents, the ARSoft.Tools.Net.dll file in the arsoft-resolver folder may be blocked by Windows as being potentially harmful for its Internet origin. To verify and unblock the DLL right-click on the file, select Properties and click Unblock on the General tab, if required.

Import the External Agent definition agent-definition.xml. Once this is done, customize the definition to your system:

  • On the Run tab, set the Agent Executable to your powershell.exe binary. This is usually found under \Windows\System32\WindowsPowerShell\1.0\powershell.exe.
  • On the same Run tab, in the Command-Line Parameters box, replace the path in the -File parameter with the path of your script, e.g. “C:\ORF\snowshoe\agent-showshoe-c15q1-01.ps1”.
  • In the same edit box, replace “8.8.8.8” with your DNS server IP address. Note that you must specify the DNS server by IP address, names will not work.

Once you’re done with editing, be sure to save your configuration.

What else should I know?

  • The agent is written in PowerShell and was tested with PowerShell 3.0 and .NET Framework 4.0. The minimum system requirement should be PowerShell 2.0 and .NET Framework 2.0.
  • As this is a quick relief agent, it was not subjected to the stringent standards of testing we usually apply to our code. Use with caution.