This has turned out to be a lot more interesting issue than I expected. I was thinking about contacting MS PSS and so I wrote an as-simple-as-possible test application to reproduce the issue. Surprisingly, the test app worked flawlessly on Vista, so started investigating the differences.

First, my test application was using the HTML Help API (hhctrl.ocx) directly and was statically linked (which means that Windows would not allow the test app to start if hhctrl.ocx is not available on the system path). ORF, on the other hand, uses a third-party library that loads hhctrl.ocx dynamically and uses some registry magic to determine the location of hhctrl.ocx, regardless the system path.

My initial thought was that the ORF Log Viewer and the test app was using different versions of hhctrl.ocx and the Log Viewer found the broken one. To test my theory, I started both applications and checked the loaded DLLs by Process Explorer just to find that Log Viewer doesn’t even have hhctrl.ocx loaded. Wow, that makes sense! Calls to hhctrl.ocx will hardly work if it’s not loaded. Actually, our calls to HtmlHelp() ended up in the middle of invalid memory.

I checked the library code again, now digging deeper in the loader code and found that it tries to determine the location of hhctrl.ocx by checking its COM registration under HKCR\CLSID\{adb880a6-d8ff-11cf-9377-00aa003b7a11}\InprocServer32. Under pre-Vista operating systems, the default value for this key used to be simple String Value (REG_SZ), containing e.g. “C:\WINDOWS\system32\hhctrl.ocx”. On Vista, however, this is an Expandable String Value, pointing to e.g. “%SystemRoot%\System32\hhctrl.ocx”. The library fails to check for this and to expand the the value into a real file system path, so it also fails to load hhctrl.ocx. Even worse, the library does not offer proper error handling, so the caller application is not aware of the failed load and calls the trusted HtmlHelp() call, resulting in some accidental endless loop.

End of story, library fixed.

If you can’t wait for ORF 4.0 to run the Log Viewer and the Reporting Tool on Vista, you can apply a manual fix as described below:

• Run regedt32.exe as Administrator (%SystemRoot%\System32\regedt32.exe)
• Open the HKEY_CLASSES_ROOT\CLSID\{adb880a6-d8ff-11cf-9377-00aa003b7a11}\InprocServer32 key
• Change the %SystemRoot% variable in the value to C:\Windows (or what is your local system root path)

Expect a little fun with permissions. Ah, and don’t forget to change it back to %SystemRoot% when ORF 4.0 ships—I am not aware of any problems with hardcoding the system path, but who knows.