Hey Paweł, In message <capv+sjgwxnbkjpd4ykyotmeaeib-wcp5detbezm2xnsv9id...@mail.gmail.com> on Thu, 10 Dec 2015 10:20:02 +0100, Paweł Witas <pw178...@gmail.com> said:
pw178860> I'm working on implementing PKCS#11 encrypted communication on Windows pw178860> platform. pw178860> This crash occurs on Windows Vista and above when engine_pkcs11.dll is pw178860> compiled by mingw toolchain and OpenSSL is compiled by Visual Studio pw178860> 2012. pw178860> It does not occur on Windows XP or when both engine_pkcs11.dll and pw178860> OpenSSL are compiled by mingw toolchain. pw178860> pw178860> The cause of this crash are different and incompatible implementations pw178860> of memory allocators in engine_pkcs11.dll (from Windows kernel's pw178860> msvcrt.dll) and OpenSSL (from VS212 msvcr110.dll). pw178860> The private key is allocated by engine_pkcs11.dll on its private heap pw178860> via callback from OpenSSL, but freed by the OpenSSL library itself. Here's a detail, IMPLEMENT_DYNAMIC_BIND_FN, passes down the memory allocation callbacks to the OpenSSL that the engine uses (be it statically linked or not). This means that memory allocated with OPENSSL_malloc and friends in engine_pkcs11.dll *should* be allocated in OpenSSL library heap rather than engine_pkcs11.dll private heap. However, if anything returned through EVP_PKEY* returns (I suspect the issues come up around keys) is allocated using something other than OPENSSL allocation calls (such as malloc(), calloc()...), then yes, you do get in trouble. I just spent a bit of time eyeing through engine_pkcs11 and libp11 and couldn't immediately find a culprit of that sort, but ... pw178860> This is troublesome, because I can compile OpenSSL by mingw for my pw178860> clients and put it at the beginning of the PATH,, but I can't replace pw178860> OpenSSL statically linked with third party products, i.e. Symantec pw178860> Antivirus LicenseMan.dll, which causes antivirus crash, because it pw178860> loads my openssl.cnf with engine_pkcs11.dll configured and tries to pw178860> use it (why?). I have no idea what LicenseMan.dll is or why it uses OpenSSL. However, if it does, it may very well call OPENSSL_config or CONF_modules_load_file, which will load whatever configuration file that's given through the OPENSSL_CONF environment variable. It might be that they need to use a different default section (the default is "openssl_conf"); that might be something to report back to Symantec. pw178860> I solved this problem by replacing references to environment variable pw178860> "OPENSSL_CONF" by "OPENSSL_KONF" in my compilation of OpenSSL, but the pw178860> real solution would be allowing the engine_pkcs11.dll library to pw178860> deallocate its own keys by the deallocating callback from OpenSSL. pw178860> It will require modifications to both OpenSSL and engine_pkcs11.dll pw178860> library. I agree with those thoughts. That will be for a future major OpenSSL version, I cannot say which one for the moment. -- Richard Levitte levi...@openssl.org OpenSSL Project http://www.openssl.org/~levitte/ _______________________________________________ openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
