Hello I'm working on implementing PKCS#11 encrypted communication on Windows platform. This crash occurs on Windows Vista and above when engine_pkcs11.dll is compiled by mingw toolchain and OpenSSL is compiled by Visual Studio 2012. It does not occur on Windows XP or when both engine_pkcs11.dll and OpenSSL are compiled by mingw toolchain.
The cause of this crash are different and incompatible implementations of memory allocators in engine_pkcs11.dll (from Windows kernel's msvcrt.dll) and OpenSSL (from VS212 msvcr110.dll). The private key is allocated by engine_pkcs11.dll on its private heap via callback from OpenSSL, but freed by the OpenSSL library itself. This is troublesome, because I can compile OpenSSL by mingw for my clients and put it at the beginning of the PATH,, but I can't replace OpenSSL statically linked with third party products, i.e. Symantec Antivirus LicenseMan.dll, which causes antivirus crash, because it loads my openssl.cnf with engine_pkcs11.dll configured and tries to use it (why?). I solved this problem by replacing references to environment variable "OPENSSL_CONF" by "OPENSSL_KONF" in my compilation of OpenSSL, but the real solution would be allowing the engine_pkcs11.dll library to deallocate its own keys by the deallocating callback from OpenSSL. It will require modifications to both OpenSSL and engine_pkcs11.dll library. Best regards Pawel
_______________________________________________ openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
