JVM crashing with caCertificatePath in server.xml
Hi, just ran into this today. The JVM is crashing when caCertificatePath is added to server.xml. I tried the latest Zulu JRE 8 and 11 but still had the crash. ENVIRONMENT Tomcat: 9.0.89 (64-bit Windows zip) OS: Windows Server 2019 JVM: openjdk version "1.8.0_322" OpenJDK Runtime Environment (Zulu 8.60.0.21-CA-win64) (build 1.8.0_322-b06) OpenJDK 64-Bit Server VM (Zulu 8.60.0.21-CA-win64) (build 25.322-b06, mixed mode) CRASH INFO When caCertificatePath is present in server.xml and points to a valid directory (empty or with PEM files) the JVM crashes during Tomcat startup. This is the JVM console output: 14-May-2024 17:34:58.443 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio2-1.2.3.4-443"] # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x0001800ccd10, pid=1244, tid=0x0ab0 # # JRE version: OpenJDK Runtime Environment (Zulu 8.60.0.21-CA-win64) (8.0_322-b06) (build 1.8.0_322-b06) # Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode windows-amd64 compressed oops) # Problematic frame: # C [tcnative-1.dll+0xccd10] # # Core dump written. Default location: D:\Program Files\apache-tomcat\bin\hs_err_pid1244.mdmp # # An error report file with more information is saved as: # D:\Program Files\apache-tomcat\bin\hs_err_pid1244.log # # If you would like to submit a bug report, please visit: # http://www.azul.com/support/ # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # CONFIG INFO Here’s the server.xml that causes the JVM crash. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM crashing with caCertificatePath in server.xml
Sure thing - ADDITIONAL ENVIRONMENT INFO: libtcnative: tcnative-1.dll is included in the Tomcat 9.0.89 64-bit Windows zip download, not sure about the version... OpenSSL version: 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024) (with FIPS 140-2) Regarding expecting a directory of certificate hash files, I wasn’t aware of this, assumed it would pick up CA cert PEM files in a directory. I would however not expect this or an empty directory to crash the JVM however… -Andy On May 14, 2024, at 2:53 PM, Michael Osipov wrote: Please provide the log file, the OpenSSL version used and the libtcnative version used. Please note that caCertificatePath expects a directory with certificate hash files. Plain certs won't work. M
Re: JVM crashing with caCertificatePath in server.xml
ADDITIONAL ENVIRONMENT INFO UPDATE: libtcnative: org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.3.0] using APR version [1.7.4]. CRASH LOG See enclosed: hs_err_pid4464.log c_rehash.pl I didn’t have perl, tried strawberry perl, it didn’t seem to create symlinks on Windows so I do it with a powershell using "openssl x509 -subject_hash -fingerprint -noout -in " making symlinks in the same directory for each CA cert PEM e.g. a655d288.0 (link) -> cert.pem (file). This didn’t seem to make a difference though, JVM still crashed. -Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM crashing with caCertificatePath in server.xml
Ah wasn’t sure if attachments worked, log content information below. Yea the docs just say directory for trusted CA PEM certificates. TOMCAT DOCS https://tomcat.apache.org/tomcat-9.0-doc/config/http.html: caCertificatePath (OpenSSL only) Name of the directory that contains the certificates for the trusted certificate authorities. The format is PEM-encoded. CATALINA LOG FINE LEVEL CONTENT 15-May-2024 01:37:45.569 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/9.0.89 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: May 3 2024 20:22:11 UTC 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.89.0 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Windows Server 2019 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 10.0 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: D:\Program Files\Java\jre 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_322-b06 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Azul Systems, Inc. 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: D:\Program Files\apache-tomcat 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: D:\Program Files\apache-tomcat 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=D:\Program Files\apache-tomcat\conf\logging.properties 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs= 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=D:\Program Files\apache-tomcat 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=D:\Program Files\apache-tomcat 15-May-2024 01:37:45.584 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=D:\Program Files\apache-tomcat\temp 15-May-2024 01:37:45.600 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.3.0] using APR version [1.7.4]. 15-May-2024 01:37:45.600 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true]. 15-May-2024 01:37:45.600 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 15-May-2024 01:37:45.647 FINE [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL Current FIPS mode: [1] 15-May-2024 01:37:45.647 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL Using OpenSSL with the FIPS provider as the default provider 15-May-2024 01:37:45.647 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 3.0.13 30 Jan 2024] 15-May-2024 01:37:45.756 FINE [main] org.apache.tomcat.util.modeler.Registry.getMBeanServer Created MBeanServer 15-May-2024 01:37:46.069 FINE [main] org.apache.catalina.util.LifecycleBase.setStateInternal Setting state for [org.apache.catalina.deploy.NamingResourcesImpl@5e955596] to [INITIALIZING] 15-May-2024 01:37:46.084 FINE [main] org.apache.catalina.util.LifecycleBase.setStateInternal Setting state for [org.apache.catalina.deploy.NamingResourcesImpl@5e955596] to [INITIALIZED] 15-May-2024 01:37:46.116 FINE [main] org.apache.catalina.util.LifecycleBase.setStateInternal Setting state for [StandardService[Catalina]] to [INITIALIZING] 15-May-2024 01:37:46.116 FINE [main] org.apache.catalina.util.LifecycleBase.setStateInternal Setting state for [StandardEngine[Catalin
Re: JVM crashing with caCertificatePath in server.xml
Ok great! Thank you for taking the time and making the effort to look into this Michael, much appreciated! -Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM crashing with caCertificatePath in server.xml
Hi Micheal, you had asked to try these - http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/. I replaced my files with these but Tomcat failed to start at this point with this message - 22-May-2024 00:02:30.808 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio2-10.232.115.117-443"] OPENSSL_Uplink(7FFEEBF10C88,08): no OPENSSL_Applink -Andy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM crashing with caCertificatePath in server.xml
Michael, good news, it’s working now. Issue was on my end, was using a custom OpenSSL installer that was built with FIPS and it had also put the two openssl lib DLLs in Window System32, after fixing that Tomcat started without JVM crash with caCertificatePath set in server.xml. Thanks! -Andy
Re: JVM crashing with caCertificatePath in server.xml
Hi Michael, After re-reading my previous message, I realized it might have been ambiguous regarding whether I observed caCertificatePath working with or without your first posted file set from http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/. To clarify, it was indeed your first posted file set that made it work. The issue I initially encountered on my end was due to some unnecessary copies of the original OpenSSL binaries elsewhere in the system path. These copies were likely causing different results as they were being loaded without my awareness. After removing them I observed Tomcat startup with caCertificatePath in server.xml without JVM crash using the original binaries you provided. I hope this clears up any ambiguity from my previous message. Thanks! -Andy
