(Responding to the list so that the solution gets archived with the rest of the thread.)
Thanks for your response and the pointer to VMMap, a very useful tool which helped me to figure out the problem. My application allocates a very large chunk of memory, which (usually) included the address that the libeay32.dll wanted to use. If I load the DLL before allocating that chunk of memory, the problem vanishes since the address is available. Graeme -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Tuesday, September 10, 2013 1:18 PM To: [email protected] Subject: Re: FIPS and rebasing Hi, > My executable loads a DLL (A) which loads a second DLL (B), and that > second DLL is linked with the OpenSSL libraries, so the libeay32.dll > and ssleay32.dll are loaded automatically when B loads. However, > libeay32.dll does not always load at the right address, and so the > fingerprint check within FIPS_mode_set fails. I tried changing my > OpenSSL build procedure (as described in the thread linked above) to > include /FIXED and /DYNAMICBASE:NO but that did not solve the problem. > Now, attempting to load DLL B fails with ERROR_INVALID_ADDRESS. I guess you're problem ist that the executable loads lots of DLLs which are assigned to more or less random addresses, then it finally loads DLL A, which in turn might load even more DLL, which get assigned to random addresses. Finally DLL A loads DLL B, which again might load further DLLs before trying to load ssleay32.dl and libeay32.dll . And when the system finally tries to load the OpenSSL DLLs, there preferred address happens to be already taken by some other DLL and you get ERROR_INVALID_ADDRESS. One approach might be to grab VMMap from http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx and look at what DLLs get loaded where during runtime and try to figure out a base adress that "usually" is not taken and recompile the FIPS stuff with that base. That's certainly going to be interesting but most likely somewhat hopeless, at least if most of the DLLs are loaded at random addresses (so turning ASLR off as much as possible could help. Just resetting that flag in the DLLs used by means of a suitable tool, e.g. cygwin's(?) "peflags" will do). Anyway, IMHO a better approach would be to make sure that your application just loads the OpenSSL libraries at the very beginning (a LoadLibrary call at the beginning of the main routine might already by enough, linking the main executable with the OpenSSL libraries would be even better), so they can try to grab their needed addresses before all the other DLLs can fill up that critical slot. > I was seeing this problem with my 64-bit > application as well for a while before the problem vanished, and I > haven't seen it since. Now it only happens on the 32-bit application. > I don't know if I unknowingly fixed it in 64-bit or if it's still > lurking there, ready to re-appear once we ship. Well, with 64-bit there are obviously many more addresses at which DLLs can be loaded, so getting a collision like above should happen much less frequently, but it still seems like it's just a matter of luck. HTH, Stefan ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List [email protected] Automated List Manager [email protected] ______________________________________________________________________ OpenSSL Project http://www.openssl.org Development Mailing List [email protected] Automated List Manager [email protected]
