I am having a problem with the FIPS_mode_set function on Windows 7 and 8. I am using OpenSSL FIPS 2.0.5 with OpenSSL 1.0.1e, and I'm seeing this problem with a 32-bit application. The problem I'm seeing is exactly the same as described here (http://comments.gmane.org/gmane.comp.encryption.openssl.devel/18309) but the solutions in that thread do not work for me. I have tried asking this question on the openssl-users mailing list with no responses, so I'm hoping someone here has an idea.
My executable loads a DLL (A) which loads a second DLL (B), and that second DLL is linked with the OpenSSL libraries, so the libeay32.dll and ssleay32.dll are loaded automatically when B loads. However, libeay32.dll does not always load at the right address, and so the fingerprint check within FIPS_mode_set fails. I tried changing my OpenSSL build procedure (as described in the thread linked above) to include /FIXED and /DYNAMICBASE:NO but that did not solve the problem. Now, attempting to load DLL B fails with ERROR_INVALID_ADDRESS. Note that the failure is intermittent - most of the time the FIPS_mode_set call fails, but sometimes it succeeds and everything works perfectly. Then I shut down my application and start it up again right away and it fails. I was seeing this problem with my 64-bit application as well for a while before the problem vanished, and I haven't seen it since. Now it only happens on the 32-bit application. I don't know if I unknowingly fixed it in 64-bit or if it's still lurking there, ready to re-appear once we ship. I'd appreciate any ideas anyone has. Graeme Perrow
