Tomcat 8 not responding - socket data not being read from TCP socket queue
Hi, We encountered an issue where Tomcat suddenly and randomly stops responding to HTTP requests. Tomcat startup logs: INFO: Server version: Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: Server built: unknown Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: Server number: 8.5.x Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: OS Name: Linux Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: OS Version:3.10.0-693.17.1.el7.x86_64 Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: Architecture: amd64 Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: Java Home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre Mar 15, 2019 7:59:35 AM org.apache.catalina.startup.VersionLoggerListener log INFO: JVM Version: 1.8.0_161-b14 Tomcat is configured to use APRConnector Mar 15, 2019 7:59:35 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Loaded APR based Apache Tomcat Native library [1.2.14] using APR version [1.4.8]. Mar 15, 2019 7:59:35 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Mar 15, 2019 7:59:35 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] Mar 15, 2019 7:59:35 AM org.apache.catalina.core.AprLifecycleListener initializeSSL INFO: OpenSSL successfully initialized [OpenSSL 1.0.2k-fips 26 Jan 2017] Mar 15, 2019 7:59:35 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-apr-8081"] Mar 15, 2019 7:59:35 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["https-openssl-apr-0:0:0:0:0:0:0:0-8443"] After some troubleshooting, we found that when this happens, Java worker threads are idle and waiting for tasks, while sockets are in ESTABLISHED and CLOSE_WAIT state. All the TCP sockets at the OS level indicate they have bytes in the TCP buffer that is not been read by the application. so looks like something went wrong, and the APR connector library is no more reading from the socket, or some issue between java socket Acceptor call and APR library. Has anyone encountered any problem like this or have any suggestion on what should be checked that may help to isolate the problem. Below the lsof output with TCP Q data and thread dump for threads for port 8443. Thanks, -- Indrajeet java 25399tomcatuser 55u IPv6 485303682 0t0TCP *:8443 (LISTEN QR=0 QS=0) java 25399tomcatuser 60u IPv6 542452576 0t0TCP 10.77.xx.xx:8443->10.xx.xx.212:62472 (CLOSE_WAIT QR=880 QS=0) java 25399tomcatuser 63u IPv6 542319858 0t0TCP 10.77.xx.xx:8443->10.xx.xx.110:56315 (CLOSE_WAIT QR=704 QS=0) java 25399tomcatuser 64u IPv6 542365742 0t0TCP 10.77.xx.xx:8443->10.xx.xx.227:65114 (CLOSE_WAIT QR=704 QS=0) java 25399tomcatuser 73u IPv6 542271640 0t0TCP 10.77.xx.xx:8443->10.xx.xx.234:53983 (CLOSE_WAIT QR=848 QS=0) java 25399tomcatuser 78u IPv6 542313271 0t0TCP 10.77.xx.xx:8443->10.xx.xx.212:61847 (CLOSE_WAIT QR=832 QS=0) java 25399tomcatuser 80u IPv6 542312628 0t0TCP 10.77.xx.xx:8443->10.xx.xx.110:56277 (CLOSE_WAIT QR=704 QS=0) java 25399tomcatuser 87u IPv6 542322416 0t0TCP 10.77.xx.xx:8443->10.xx.xx.110:56350 (CLOSE_WAIT QR=704 QS=0) java 25399tomcatuser 92u IPv6 542301688 0t0TCP 10.77.xx.xx:8443->10.xx.xx.110:56251 (CLOSE_WAIT QR=784 QS=0) java 25399tomcatuser 94u IPv6 542344204 0t0TCP 10.77.xx.xx:8443->10.xx.xx.212:61954 (CLOSE_WAIT QR=816 QS=0) java 25399tomcatuser 95u IPv6 542301753 0t0TCP 10.77.xx.xx:8443->10.xx.xx.227:64780 (CLOSE_WAIT QR=784 QS=0) java 25399tomcatuser 98u IPv6 542313111 0t0TCP 10.77.xx.xx:8443->10.xx.xx.110:56292 (CLOSE_WAIT QR=704 QS=0) java 25399tomcatuser 109u IPv6 542312567 0t0TCP 10.77.xx.xx:8443->10.xx.xx.130:51038 (ESTABLISHED QR=447 QS=0) java 25399tomcatuser 111u IPv6 542301754 0t0TCP 10.77.xx.xx:8443->10.xx.xx.129:54443 (CLOSE_WAIT QR=800 QS=0) java 25399tomcatuser 112u IPv6 542312764 0t0TCP 10.77.xx.xx:8443->10.xx.xx.212:61834 (CLOSE_WAIT QR=816 QS=0) java 25399tomcatuser 123u
Re: Tomcat 8.5.39 failed to initialize connector
On 28/03/2019 17:18, Ethan Jensen wrote: > On Thu, Mar 28, 2019 at 11:11 AM Mark Thomas wrote: >> Can you post the header of your private key file? It should look >> something like: >> >> -BEGIN RSA PRIVATE KEY- >> Proc-Type: 4,ENCRYPTED >> DEK-Info: AES-256-CBC,D02DE734A8C2DBA625FC4180E7AECC78 >> >> Thanks, >> >> Mark >> >> > Here you are: > > Bag Attributes > localKeyID: 14 A3 77 23 14 44 3E 99 FD 7D A4 BE C3 4C 10 D0 DD 5A DA 0B > friendlyName: mydomain.com > Key Attributes: > -BEGIN ENCRYPTED PRIVATE KEY- Bingo. That is a PKCS#8 format file that OpenSSL understands but JSSE does not. The fix I had in mind does work. Now I understand why the problem occurred and can confirm that the fix works I'll apply it for the next release. A a workaround you can convert that private key to PKCS#1 format. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re: Resource Request - MySQL Data Pool
Chris, Thanks. Lots to go through... On 3/26/2019 9:00 AM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Richard, On 3/25/19 14:15, Richard Huntrods wrote: It's time to update my application to use "real" (i.e. current best practices) data connection pooling. :) My application is Java Servlets, no beans, no JSP. Database is MySQL. System etc. details: Ubuntu live server 18.04.2, built March 6, 2019. MySQL - latest installed via 'apt-get install mysql-server' after system build. So... MariaDB, then? Or does Ubuntu still stock MySQL binaries? Seems to be MySQL. See next... OpenJVM - 11? - again, latest version installed via 'apt-get install default-jdk' at same time. Should be pretty easy to determine this: $ java -version I typed 'java -version' and this was the output: openjdk version "10.0.2" 2018-07-17 OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4) OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.4, mixed mode) I also typed 'mysql -version' and got this (still not sure if Ubuntu uses MariaDB or MySQL by default): mysql Ver 14.14 Distrib 5.7.25, for Linux (x86_64) using EditLine wrapper Tomcat 8.5.39 - just updated the same day it came out. Sounds good so far. This system has been running in production since the early 2001's. OS has changed over the years from Sun Solaris 8.x to Solaris 10.x and now to Ubuntu 18.04 (server). Java has been updated over the years as well, as has Tomcat and MySQL. Through all that the system works quite perfectly. Except... there are occasional hangs that implicate the 'home grown' data connection pool. I wrote this by hand (in Java) back in 2001 because there was nothing much available back then. Since it kept working, I didn't have the time/inclination to change over the years. You may find that your home-grown connection pool is actually okay, but it's being used incorrectly by client code (which is also your code). IF you have problems with the client code, the "real" connection-pool can help you tolerate them, but it won't magically fix them. I had problems some years ago with one particular version of the connector which had a memory leak (in the connector). I avoided that version and have had no particular problems. Some years ago I did a pretty exhaustive examination of my application using various metering tools to see if I was creating memory leaks with my database code. I found one (forgot to close the connection), fixed it and there weren't any more. I also encapsulate ALL my database access into a single "DBMS.java" class which is used by all the servlets to access data. The DBMS class calls the various pool creation and management classes as needed, so all my data "stuff" is in one place (or set of classes). That makes it simple but also makes it more complex as the "roll my own" pool is quite integrated into the DBMS code. I'll have to simplify and then do the testing as you suggest below. But the latest connector (mysql-connector-java-8.0.15.jar, a.k.a. "com.mysql.cj.jdbc.Driver" is giving me some hiccups. I thought rather than trying to debug my own connection pool, it was time to switch over to a proper "modern" supported connection pooling system. Which brings me to my question. Would the community please weigh in on the BEST tutorials / documents regarding creating a Tomcat/MySQL database connection pool for Servlets (not JSP or beans) with some good code examples and server.xml examples? I've already done some extensive internet searches, but when you are doing something for the first time it's hard to tell the difference between "really really good" and "blogger who has not really tried it". You will want Tomcat to create the connection pool for you. Anything else is a waste of time. Here's what happens: Here we are in total agreement. I *want* Tomcat to manage the pool as I suspect pool timeouts are the overall issue that I'm seeing. Basically, after several hours of inactivity (the application isn't used a lot these days), it just "loses" it's connection and then subsequent data accesses generate exceptions as the connection is no longer present. It does not happen when the system is being used and data accessed regularly, only if the system sits idle for several hours. So at least to me it's clearly an issue with the home-grown connection pool "losing" touch with the database but not in a way that is evident to my current code. I've resorted to "tricks" using cron and another servlet to regularly access the database to keep the pool open, but I figured a Tomcat managed pool would have more capability to handle such things. I could rewrite my own pool, but at this point I'd rather use Tomcat pools as I can just offload that portion of my code to a community resource, which I suspect is much better if only because of all the eyeballs on the Tomcat code. 1. During application startup, Tomcat creates a
Thread pools
Hi, Is any parameter to recycle Tomcat connector thread pools like Datasource pools? Or we can only define maxThreads value to connector element. Please advise. Thanks, Rajendra -- Thanks Rajendra
Re: Re: Resource Request - MySQL Data Pool
Luis, Thanks very much. I'll have a look. Cheers, -Richard On 3/26/2019 1:43 AM, Luis Rodríguez Fernández wrote: Hello Richard, In my experience the best is to "start simple". I would have a look at the apache tomcat doc [1], configure your pool with a minimal setup and test. Everything depends on your application workload, how your queries looks like, etc, so I am afraid that there are no "silver bullets" in this domain. Hope it helps, Luis [1] https://tomcat.apache.org/tomcat-8.5-doc/jndi-datasource-examples-howto.html El lun., 25 mar. 2019 a las 19:15, Richard Huntrods () escribió: It's time to update my application to use "real" (i.e. current best practices) data connection pooling. My application is Java Servlets, no beans, no JSP. Database is MySQL. System etc. details: Ubuntu live server 18.04.2, built March 6, 2019. MySQL - latest installed via 'apt-get install mysql-server' after system build. OpenJVM - 11? - again, latest version installed via 'apt-get install default-jdk' at same time. Tomcat 8.5.39 - just updated the same day it came out. This system has been running in production since the early 2001's. OS has changed over the years from Sun Solaris 8.x to Solaris 10.x and now to Ubuntu 18.04 (server). Java has been updated over the years as well, as has Tomcat and MySQL. Through all that the system works quite perfectly. Except... there are occasional hangs that implicate the 'home grown' data connection pool. I wrote this by hand (in Java) back in 2001 because there was nothing much available back then. Since it kept working, I didn't have the time/inclination to change over the years. But the latest connector (mysql-connector-java-8.0.15.jar, a.k.a. "com.mysql.cj.jdbc.Driver" is giving me some hiccups. I thought rather than trying to debug my own connection pool, it was time to switch over to a proper "modern" supported connection pooling system. Which brings me to my question. Would the community please weigh in on the BEST tutorials / documents regarding creating a Tomcat/MySQL database connection pool for Servlets (not JSP or beans) with some good code examples and server.xml examples? I've already done some extensive internet searches, but when you are doing something for the first time it's hard to tell the difference between "really really good" and "blogger who has not really tried it". Thanks very much in advance. -Richard --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.5.39 failed to initialize connector
On Thu, Mar 28, 2019 at 11:11 AM Mark Thomas wrote: > On 28/03/2019 16:50, Ethan Jensen wrote: > > On Fri, Mar 22, 2019 at 1:13 PM Ethan Jensen > > wrote: > > > >> On Fri, Mar 22, 2019 at 12:51 PM Mark Thomas wrote: > >> > >>> On 22/03/2019 17:18, Ethan Jensen wrote: > On Fri, Mar 22, 2019 at 11:07 AM Ethan Jensen < > sr.agent.r...@gmail.com> > wrote: > > > > > > > On Fri, Mar 22, 2019 at 10:56 AM Mark Thomas > wrote: > > > >> On 22/03/2019 16:40, Ethan Jensen wrote: > >>> OS: Windows Server 2012 R2 > >>> JDK: Oracle JDK 1.8.0_201 > >>> > >>> Attempting to migrate from Tomcat 8.5.38 -> 8.5.39 results in > >>> > >>> Failed to initialize connector [Connector[HTTP/1.1-443]] > >>> > >>> when using the exact same configuration. Tomcat's > >>> .../conf/server.xml > >> is > >>> unchanged. Did a configuration parameter change or get renamed? > The > >>> exception is fairly cryptic from my point of view. > >> > >> > >> > >>> Caused by: java.lang.IllegalArgumentException: ObjectIdentifier() > -- > >> data > >>> isn't an object ID (tag = 48) > >>> at > >>> org.apache.tomcat.util.net > >> .AprEndpoint.createSSLContext(AprEndpoint.java:404) > >>> at org.apache.tomcat.util.net > >> .AprEndpoint.bind(AprEndpoint.java:368) > >>> at > >>> org.apache.tomcat.util.net > >> .AbstractEndpoint.init(AbstractEndpoint.java:1105) > >>> at > >> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581) > >>> at > >>> > >> > >>> > org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68) > >>> at > >>> > >>> > org.apache.catalina.connector.Connector.initInternal(Connector.java:993) > >>> ... 13 more > >> > >> Looks like a certificate in a format JSSE can't handle. If you can > >> provide the steps (e.g. OpenSSL commands) to recreate a > >>> key/certificate > >> in that format we should be able to reproduce it and figure out a > fix. > >> > >> Mark > >> > >> > > Mark, > > > > These are the steps I used to create my certificate a couple of years > >>> ago > > (3 year validity). > > > > 1. Generate CSR: > > > > openssl req -out cert.csr -new -newkey rsa:2048 -nodes -keyout > cert.key > > > > 2. Create a certificate chain file, using the certificates from CA: > > > > cat CERT.crt > chain_certs.pem && > > echo "" >> chain_certs.pem && > > cat OV_NetworkSolutionsOVServerCA2.crt >> chain_certs.pem && > > echo "" >> chain_certs.pem && > > cat OV_USERTrustRSACertificationAuthority.crt >> chain_certs.pem && > > echo "" >> chain_certs.pem > > > > 3. Use openssl to package the certificate chain and private key into > a > > PKCS#12 container: > > > > openssl pkcs12 -export -out cert.p12 -inkey cert.key -in > >>> chain_certs.pem > > -name "cert_name" > > > > > > > Also, it should be noted that for the APR connector, I'm using the raw > individual certificate/chain/key files for the configuration > parameters. > The pkcs12 step I only use with the NIO fallback connector (currently > commented out in my server.xml) in the event the APR connector is > >>> broken. > >>> > >>> Thanks for the additional info. Those steps are effectively identical > to > >>> the ones we use to create the test certificates for Tomcat. > >>> > >>> It looks like the difference is the encryption you are using for the > >>> private key. What are you using? I've tried a few different ones here > >>> and while JSSE can't process the PEM file it throws a KeyStoreException > >>> which causes Tomcat to pass the cert directly to OpenSSL. > >>> > >>> I'd like to be able to reproduce this before I patch it although I do > >>> have a patch in mind for you to test based on the stack trace. > >>> > >>> Mark > >>> > >>> > >>> > >> I'm not quite clear what you mean here Can you elaborate?: > >> > >> "It looks like the difference is the encryption you are using for the > >> private key. What are you using?" > >> > >> I'm assuming whatever is the default (I generated the certificate on a > >> CentOS 7 host). Using the steps I outlined above, the only thing it > asked > >> me for was an Export Password to be tied to the private key. Perhaps > some > >> special characters in that password are tripping things up with the new > >> JSSE configuration? > >> > >> -- > >> Ethan > >> > >> > > > > Mark, > > > > Did you need any additional information from me regarding this config? > Or > > did you get everything you needed? > > Sorry, I missed replying to this. > > Can you post the header of your private key file? It should look > something like: > > -BEGIN RSA PRIVATE KEY- > Proc-Type: 4,ENCRYPTED > DEK-Info: AES-256-CBC,D02DE734A8C2DBA625FC4180E7AECC78 > > Thanks, > > Mark > > Here you are: Bag
Re: Tomcat 8.5.39 failed to initialize connector
On 28/03/2019 16:50, Ethan Jensen wrote: > On Fri, Mar 22, 2019 at 1:13 PM Ethan Jensen > wrote: > >> On Fri, Mar 22, 2019 at 12:51 PM Mark Thomas wrote: >> >>> On 22/03/2019 17:18, Ethan Jensen wrote: On Fri, Mar 22, 2019 at 11:07 AM Ethan Jensen wrote: > > > On Fri, Mar 22, 2019 at 10:56 AM Mark Thomas wrote: > >> On 22/03/2019 16:40, Ethan Jensen wrote: >>> OS: Windows Server 2012 R2 >>> JDK: Oracle JDK 1.8.0_201 >>> >>> Attempting to migrate from Tomcat 8.5.38 -> 8.5.39 results in >>> >>> Failed to initialize connector [Connector[HTTP/1.1-443]] >>> >>> when using the exact same configuration. Tomcat's >>> .../conf/server.xml >> is >>> unchanged. Did a configuration parameter change or get renamed? The >>> exception is fairly cryptic from my point of view. >> >> >> >>> Caused by: java.lang.IllegalArgumentException: ObjectIdentifier() -- >> data >>> isn't an object ID (tag = 48) >>> at >>> org.apache.tomcat.util.net >> .AprEndpoint.createSSLContext(AprEndpoint.java:404) >>> at org.apache.tomcat.util.net >> .AprEndpoint.bind(AprEndpoint.java:368) >>> at >>> org.apache.tomcat.util.net >> .AbstractEndpoint.init(AbstractEndpoint.java:1105) >>> at >> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581) >>> at >>> >> >>> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68) >>> at >>> >>> org.apache.catalina.connector.Connector.initInternal(Connector.java:993) >>> ... 13 more >> >> Looks like a certificate in a format JSSE can't handle. If you can >> provide the steps (e.g. OpenSSL commands) to recreate a >>> key/certificate >> in that format we should be able to reproduce it and figure out a fix. >> >> Mark >> >> > Mark, > > These are the steps I used to create my certificate a couple of years >>> ago > (3 year validity). > > 1. Generate CSR: > > openssl req -out cert.csr -new -newkey rsa:2048 -nodes -keyout cert.key > > 2. Create a certificate chain file, using the certificates from CA: > > cat CERT.crt > chain_certs.pem && > echo "" >> chain_certs.pem && > cat OV_NetworkSolutionsOVServerCA2.crt >> chain_certs.pem && > echo "" >> chain_certs.pem && > cat OV_USERTrustRSACertificationAuthority.crt >> chain_certs.pem && > echo "" >> chain_certs.pem > > 3. Use openssl to package the certificate chain and private key into a > PKCS#12 container: > > openssl pkcs12 -export -out cert.p12 -inkey cert.key -in >>> chain_certs.pem > -name "cert_name" > > > Also, it should be noted that for the APR connector, I'm using the raw individual certificate/chain/key files for the configuration parameters. The pkcs12 step I only use with the NIO fallback connector (currently commented out in my server.xml) in the event the APR connector is >>> broken. >>> >>> Thanks for the additional info. Those steps are effectively identical to >>> the ones we use to create the test certificates for Tomcat. >>> >>> It looks like the difference is the encryption you are using for the >>> private key. What are you using? I've tried a few different ones here >>> and while JSSE can't process the PEM file it throws a KeyStoreException >>> which causes Tomcat to pass the cert directly to OpenSSL. >>> >>> I'd like to be able to reproduce this before I patch it although I do >>> have a patch in mind for you to test based on the stack trace. >>> >>> Mark >>> >>> >>> >> I'm not quite clear what you mean here Can you elaborate?: >> >> "It looks like the difference is the encryption you are using for the >> private key. What are you using?" >> >> I'm assuming whatever is the default (I generated the certificate on a >> CentOS 7 host). Using the steps I outlined above, the only thing it asked >> me for was an Export Password to be tied to the private key. Perhaps some >> special characters in that password are tripping things up with the new >> JSSE configuration? >> >> -- >> Ethan >> >> > > Mark, > > Did you need any additional information from me regarding this config? Or > did you get everything you needed? Sorry, I missed replying to this. Can you post the header of your private key file? It should look something like: -BEGIN RSA PRIVATE KEY- Proc-Type: 4,ENCRYPTED DEK-Info: AES-256-CBC,D02DE734A8C2DBA625FC4180E7AECC78 Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.5.39 failed to initialize connector
On Fri, Mar 22, 2019 at 1:13 PM Ethan Jensen wrote: > On Fri, Mar 22, 2019 at 12:51 PM Mark Thomas wrote: > >> On 22/03/2019 17:18, Ethan Jensen wrote: >> > On Fri, Mar 22, 2019 at 11:07 AM Ethan Jensen >> > wrote: >> > >> >> >> >> >> >> On Fri, Mar 22, 2019 at 10:56 AM Mark Thomas wrote: >> >> >> >>> On 22/03/2019 16:40, Ethan Jensen wrote: >> OS: Windows Server 2012 R2 >> JDK: Oracle JDK 1.8.0_201 >> >> Attempting to migrate from Tomcat 8.5.38 -> 8.5.39 results in >> >> Failed to initialize connector [Connector[HTTP/1.1-443]] >> >> when using the exact same configuration. Tomcat's >> .../conf/server.xml >> >>> is >> unchanged. Did a configuration parameter change or get renamed? The >> exception is fairly cryptic from my point of view. >> >>> >> >>> >> >>> >> Caused by: java.lang.IllegalArgumentException: ObjectIdentifier() -- >> >>> data >> isn't an object ID (tag = 48) >> at >> org.apache.tomcat.util.net >> >>> .AprEndpoint.createSSLContext(AprEndpoint.java:404) >> at org.apache.tomcat.util.net >> >>> .AprEndpoint.bind(AprEndpoint.java:368) >> at >> org.apache.tomcat.util.net >> >>> .AbstractEndpoint.init(AbstractEndpoint.java:1105) >> at >> >>> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581) >> at >> >> >>> >> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68) >> at >> >> org.apache.catalina.connector.Connector.initInternal(Connector.java:993) >> ... 13 more >> >>> >> >>> Looks like a certificate in a format JSSE can't handle. If you can >> >>> provide the steps (e.g. OpenSSL commands) to recreate a >> key/certificate >> >>> in that format we should be able to reproduce it and figure out a fix. >> >>> >> >>> Mark >> >>> >> >>> >> >> Mark, >> >> >> >> These are the steps I used to create my certificate a couple of years >> ago >> >> (3 year validity). >> >> >> >> 1. Generate CSR: >> >> >> >> openssl req -out cert.csr -new -newkey rsa:2048 -nodes -keyout cert.key >> >> >> >> 2. Create a certificate chain file, using the certificates from CA: >> >> >> >> cat CERT.crt > chain_certs.pem && >> >> echo "" >> chain_certs.pem && >> >> cat OV_NetworkSolutionsOVServerCA2.crt >> chain_certs.pem && >> >> echo "" >> chain_certs.pem && >> >> cat OV_USERTrustRSACertificationAuthority.crt >> chain_certs.pem && >> >> echo "" >> chain_certs.pem >> >> >> >> 3. Use openssl to package the certificate chain and private key into a >> >> PKCS#12 container: >> >> >> >> openssl pkcs12 -export -out cert.p12 -inkey cert.key -in >> chain_certs.pem >> >> -name "cert_name" >> >> >> >> >> >> >> > Also, it should be noted that for the APR connector, I'm using the raw >> > individual certificate/chain/key files for the configuration parameters. >> > The pkcs12 step I only use with the NIO fallback connector (currently >> > commented out in my server.xml) in the event the APR connector is >> broken. >> >> Thanks for the additional info. Those steps are effectively identical to >> the ones we use to create the test certificates for Tomcat. >> >> It looks like the difference is the encryption you are using for the >> private key. What are you using? I've tried a few different ones here >> and while JSSE can't process the PEM file it throws a KeyStoreException >> which causes Tomcat to pass the cert directly to OpenSSL. >> >> I'd like to be able to reproduce this before I patch it although I do >> have a patch in mind for you to test based on the stack trace. >> >> Mark >> >> >> > I'm not quite clear what you mean here Can you elaborate?: > > "It looks like the difference is the encryption you are using for the > private key. What are you using?" > > I'm assuming whatever is the default (I generated the certificate on a > CentOS 7 host). Using the steps I outlined above, the only thing it asked > me for was an Export Password to be tied to the private key. Perhaps some > special characters in that password are tripping things up with the new > JSSE configuration? > > -- > Ethan > > Mark, Did you need any additional information from me regarding this config? Or did you get everything you needed? -- Ethan
RE: [EXTERNAL] Re: Could not find datasource: java:/comp/env/jdbc/TOPSDB when start Tomcat 9.0.13
Chris: I did what you suggested, that is to remove all drivers from the application's WEB-INF/lib directory and leave the one in Tomcat's lib/ directory. Now the error "FATAL connection.DatasourceConnectionProvider - Could not find datasource: java:/comp/env/jdbc/TOPSDB " and " java.lang.ClassCastException: org.apache.tomcat.dbcp.dbcp2.BasicDataSource cannot be cast to javax.sql.DataSource " is gone. I am going to do more cleaning and re-arrangement to the jars. Thanks Gary -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, March 26, 2019 10:43 AM To: users@tomcat.apache.org Subject: Re: [EXTERNAL] Re: Could not find datasource: java:/comp/env/jdbc/TOPSDB when start Tomcat 9.0.13 -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Gary, On 3/25/19 12:08, Hua, Gary - Saint Louis, MO - Contractor wrote: > Olaf: > > Thanks for the input.I removed jdbc2_0-stdext.jar and > tomcat-dbcp.jar from > /opt/TomCat/apache-tomcat-9.0.13/webapps/TOPS-WEB/WEB-INF/lib and > did some cleaning on /opt/TomCat/apache-tomcat-9.0.13/lib, now > that is my lib folder looks like: Wow, this must be a very old web application. You still have some cleaning-up to do. > atadmin@eagnmnmed1f45:/opt/TomCat/apache-tomcat-9.0.13/webapps/TOPS-WE B/WEB-INF/lib>ls > -l total 20648 -rwxrwxrwx 1 atadmin atadmin 433164 Dec 17 17:47 > antlr-2.7.5H3.jar -rwxrwxrwx 1 atadmin atadmin 281998 Dec 17 17:45 > cglib-2.1.jar Super old. [...] > -rwxrwxrwx 1 atadmin atadmin 16777 Dec 17 17:45 asm-attrs.jar > -rwxrwxrwx 1 atadmin atadmin 26360 Dec 17 17:45 asm.jar > -rwxrwxrwx 1 atadmin atadmin 188671 Dec 17 17:47 > commons-beanutils.jar -rwxrwxrwx 1 atadmin atadmin 165119 Dec 17 > 17:45 commons-collections.jar > -rwxrwxrwx 1 atadmin atadmin 168446 Dec 17 17:47 > commons-digester.jar -rwxrwxrwx 1 atadmin atadmin 26388 Dec 17 > 17:47 commons-logging.jar -rwxrwxrwx 1 atadmin atadmin 84462 Dec > 17 17:47 commons-validator.jar -rwxrwxrwx 1 atadmin atadmin > 153115 Dec 17 17:45 jdom.jar -rwxrwxrwx 1 atadmin atadmin8812 > Dec 17 17:45 jta.jar -rwxrwxrwx 1 atadmin atadmin 367444 Dec 17 > 17:45 log4j.jar I'm always suspicious of library JAR files that have no version number. You might want to take a look at what these are and re-name them appropriately. > -rwxrwxrwx 1 atadmin atadmin 1196109 Dec 17 17:47 classes12.jar classes12.jar is Oracle's JDBC driver written for Java 1.2. I'm fairly sure that was hand-coded by Hammurabi himself. If you are indeed using Oracle DB, you need to upgrade to a library version released during this century. > -rwxrwxrwx 1 atadmin atadmin 3698857 Mar 15 15:32 ojdbc7.jar This is a second Oracle JDBC driver. Do you need both of them? > -rwxrwxrwx 1 atadmin atadmin 4604132 Dec 17 17:45 > com.ibm.ws.webcontainer.jar This is a WebSphere library. Presumably, you have left WebSphere behind in favor of Tomcat? Or maybe you need some service that WS provides and you have taken it with you? > -rwxrwxrwx 1 atadmin atadmin 205318 Mar 19 11:20 > commons-dbcp2-2.6.0.jar -rwxrwxrwx 1 atadmin atadmin 70604 Dec 17 > 17:45 commons-fileupload-1.3.3.jar -rwxrwxrwx 1 atadmin atadmin > 214788 Dec 17 17:45 commons-io-2.6.jar -rwxrwxrwx 1 atadmin > atadmin 207723 Dec 17 17:47 commons-lang-2.1.jar -rwxrwxrwx 1 > atadmin atadmin 315805 Dec 17 17:47 commons-lang3-3.1.jar > -rwxrwxrwx 1 atadmin atadmin 210432 Dec 17 17:45 > displaytag-1.1.jar -rwxrwxrwx 1 atadmin atadmin 12590 Dec 17 > 17:45 displaytag-export-poi-1.1.jar -rwxrwxrwx 1 atadmin atadmin > 312509 Dec 17 17:45 dom4j-1.5.2.jar -rwxrwxrwx 1 atadmin atadmin > 47531 Dec 17 17:45ehcache-1.1.jar -rwxrwxrwx 1 atadmin atadmin > 1925498 Dec 17 17:45 hibernate3.jar -rwxrwxrwx 1 atadmin atadmin > 65425 Dec 17 17:45jakarta-oro.jar > -rwxrwxrwx 1 atadmin atadmin 1979523 Dec 17 17:41 javaee-api-8.0.jar I'm fairly sure this should be removed. Tomcat provides all of the APIs that you need. While this may be a compile-time dependency, everything should be provided at runtime by Tomcat. > -rwxrwxrwx 1 atadmin atadmin 414240 Dec 17 16:29 jstl-1.2.jar > -rwxrwxrwx 1 atadmin atadmin 105355 Dec 17 17:45 > old_lcms-webtools.jar -rwxrwxrwx 1 atadmin atadmin 795231 Dec 17 > 17:45 poi-2.5-final-20040302.jar -rwxrwxrwx 1 atadmin atadmin > 55210 Dec 17 17:45poi-contrib-2.5-final-20040302.jar -rwxrwxrwx 1 > atadmin atadmin 188942 Dec 17 17:45 > poi-scratchpad-2.5-final-20040302.jar -rwxrwxrwx 1 atadmin atadmin > 475943 Dec 17 17:45 proxool-0.8.3.jar -rwxrwxrwx 1 atadmin atadmin > 543706 Dec 17 17:47 struts.jar Aha, I see. You are running Struts 1 which requires ancient versions of certain libraries. > -rwxrwxrwx 1 atadmin atadmin 495271 Dec 17 17:47 > Struts-Layout.jar -rwxrwxrwx 1 atadmin atadmin 68046 Dec 17 17:47 > struts-menu-2.4.3.jar -rwxrwxrwx 1 atadmin atadmin
Expect: 100-continue not working with curl and HTTP/2
Hi folks,
right away, I don't know whether it is us (Tomcat) or curl. I'd lke to
narrow down the cause.
I am trying to enable HTTP/2 for some upload services via Tomcat
directly (HTTPd is currently out of scope here).
I am running off:
Server version:Apache Tomcat/8.5.38
> Server built: Feb 5 2019 11:42:42 UTC
> Server number: 8.5.38.0
> OS Name: FreeBSD
> OS Version:12.0-STABLE
> Architecture: amd64
> Java Home: /usr/local/openjdk8/jre
> JVM Version: 1.8.0_202-b08
> JVM Vendor:Oracle Corporation
>
> Loaded APR based Apache Tomcat Native library [1.2.21] using APR
version [1.6.5].
> capabilities: IPv6 [true], sendfile [true], accept filters [true],
random [true].
> APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]
> OpenSSL successfully initialized [OpenSSL 1.1.1a-freebsd 20 Nov 2018]
>
>maxHttpHeaderSize="24576" maxThreads="250"
> SSLEnabled="true" scheme="https" secure="true"
> defaultSSLHostConfigName="sitex-ldadw.ad001.siemens.net">
>
> protocols="TLSv1.2+TLSv1.3"
> honorCipherOrder="true"
ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!DSS">
> certificateFile="/etc/ssl/sitex-ldadw.ad001.siemens.net/cert/public.pem"
>
certificateKeyFile="/etc/ssl/sitex-ldadw.ad001.siemens.net/key/private.pem"
> certificateKeyPassword="..."
> type="RSA" />
>
>
>
> $ curl --version
> curl 7.64.0 (amd64-portbld-freebsd12.0) libcurl/7.64.0 OpenSSL/1.1.1b
zlib/1.2.11 nghttp2/1.37.0
> Release-Date: 2019-02-06
> Protocols: file http https rtsp smtp smtps
> Features: AsynchDNS Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB
SSL libz HTTP2 UnixSockets HTTPS-proxy
Now trying to upload a WAR file to the manager, with curl for example, fails
(while it works with HTTP/1.1 flawlessly for years):
> $ time curl --verbose -H 'Expect: 100-continue' --negotiate -u :
--upload-file target/lda-docgen-webapp-0.1-SNAPSHOT-backend-dev.war -o
manager-response.txt
'https://sitex-ldadw.ad001.siemens.net:8445/backend-dev/manager-1/text/deploy?path=/backend-dev=false=003'
> * Expire in 0 ms for 6 (transfer 0x800d65000)
> * Uses proxy env variable NO_PROXY == 'localhost .siemens.net
.siemens.com .siemens.de'
> * Expire in 1 ms for 1 (transfer 0x800d65000)
> % Total% Received % Xferd Average Speed TimeTime
Time Current
> Dload Upload Total Spent
Left Speed
> 0 00 00 0 0 0 --:--:-- --:--:--
--:--:-- 0* Expire in 0 ms for 1 (transfer 0x800d65000)
> * Expire in 1 ms for 1 (transfer 0x800d65000)
> ...
> * Trying 147.54.64.55...
> * TCP_NODELAY set
> * Expire in 200 ms for 4 (transfer 0x800d65000)
> * Connected to sitex-ldadw.ad001.siemens.net (147.54.64.55) port 8445
(#0)
> * ALPN, offering h2
> * ALPN, offering http/1.1
> * successfully set certificate verify locations:
> * CAfile: /usr/local/etc/ssl/cert.pem
> CApath: none
> } [5 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Client hello (1):
> } [512 bytes data]
> * TLSv1.3 (IN), TLS handshake, Server hello (2):
> { [122 bytes data]
> * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
> { [19 bytes data]
> * TLSv1.3 (IN), TLS handshake, Certificate (11):
> { [6195 bytes data]
> * TLSv1.3 (IN), TLS handshake, CERT verify (15):
> { [520 bytes data]
> * TLSv1.3 (IN), TLS handshake, Finished (20):
> { [52 bytes data]
> * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
> } [1 bytes data]
> * TLSv1.3 (OUT), TLS handshake, Finished (20):
> } [52 bytes data]
> * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
> * ALPN, server accepted to use h2
> * Server certificate:
> * subject: C=DE; O=Siemens; OU=LDA DW; CN=sitex-ldadw.ad001.siemens.net
> * start date: Mar 19 13:10:13 2019 GMT
> * expire date: Mar 19 13:10:13 2020 GMT
> * subjectAltName: host "sitex-ldadw.ad001.siemens.net" matched
cert's "sitex-ldadw.ad001.siemens.net"
> * issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens;
serialNumber=ZZB7; OU=Siemens Trust Center; CN=Siemens Issuing CA
Intranet Server 2017
> * SSL certificate verify ok.
> * Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after
upgrade: len=0
> } [5 bytes data]
> * Using Stream ID: 1 (easy handle 0x800d65000)
> } [5 bytes data]
> > PUT
/backend-dev/manager-1/text/deploy?path=/backend-dev=false=003
HTTP/2
> > Host: sitex-ldadw.ad001.siemens.net:8445
> > User-Agent: curl/7.64.0
> > Accept: */*
> > Expect: 100-continue
> > Content-Length: 6502195
> >
> { [5 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [281 bytes data]
> * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
> { [281 bytes data]
> * old SSL session ID is stale, removing
> } [5 bytes data]
> * Connection state changed (MAX_CONCURRENT_STREAMS == 200)!
> } [5 bytes data]
> <
