Re: Connection closed error and certificateVerification="required"
Mark, > Am 19.04.2018 um 20:58 schrieb Mark Thomas: > > On 19/04/18 16:50, Peter@Kreuser-Online wrote: > > > >> Do you mind to share more about the root cause? I’ve followed this mail >> communication from the start and am curious. > > Sure. > > Tomcat was configured to require CLIENT-CERT auth and the main client > was configured to use this. Occasionally, the main client would see > connection problems when using 8.5.x or later with NIO and the OpenSSL > TLS implementation. > > There was a second client the performed health checks on the server. > This client did not use a certificate. The requests it made always > failed but did so with a predictable error message that the health check > looked for. > > OpenSSL error states are stored per thread. > > Each Java thread is mapped 1-to-1 to an OS thread. > > The sequence of events that cause the problem was as follows: > > - Health check ran > - TLS connection failed because no client certificate was provided > - OpenSSL set an error state that - depending on the timing of the > socket closure - was not always cleaned up > - Standard request was received and was handled by the thread that > previously experienced the error > - Because the error had not been cleaned up, this new connection thought > the error was meant for it and closed the connection > > The fix was to ensure that, whenever the Tomcat code made a call to > OpenSSL that looked like this: > - Do something via the OpenSSL API > - Check the OpenSSL error state > > the code was changed so it looked like this: > - Clear the OpenSSL error state > - Do something via the OpenSSL API > - Check the OpenSSL error state > > I also added a TODO for the arguably more complete fix which is to check > the OpenSSL error state after every call to the OpenSSL API. > >> Let me tell you that your endurance on all the tricky issues here is >> admirable! >> >> Thank you for that! > > Thank you. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > Much appreciated. That was a mean one. Best regards. Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 19/04/18 16:50, Peter@Kreuser-Online wrote: > Do you mind to share more about the root cause? I’ve followed this mail > communication from the start and am curious. Sure. Tomcat was configured to require CLIENT-CERT auth and the main client was configured to use this. Occasionally, the main client would see connection problems when using 8.5.x or later with NIO and the OpenSSL TLS implementation. There was a second client the performed health checks on the server. This client did not use a certificate. The requests it made always failed but did so with a predictable error message that the health check looked for. OpenSSL error states are stored per thread. Each Java thread is mapped 1-to-1 to an OS thread. The sequence of events that cause the problem was as follows: - Health check ran - TLS connection failed because no client certificate was provided - OpenSSL set an error state that - depending on the timing of the socket closure - was not always cleaned up - Standard request was received and was handled by the thread that previously experienced the error - Because the error had not been cleaned up, this new connection thought the error was meant for it and closed the connection The fix was to ensure that, whenever the Tomcat code made a call to OpenSSL that looked like this: - Do something via the OpenSSL API - Check the OpenSSL error state the code was changed so it looked like this: - Clear the OpenSSL error state - Do something via the OpenSSL API - Check the OpenSSL error state I also added a TODO for the arguably more complete fix which is to check the OpenSSL error state after every call to the OpenSSL API. > Let me tell you that your endurance on all the tricky issues here is > admirable! > > Thank you for that! Thank you. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Mark, >> Am 18.04.2018 um 11:55 schrieb Mark Thomas: >> >> On 18/04/18 10:36, Richard Tearle wrote: >> On 17 April 2018 at 16:45, Richard Tearle >> wrote: >>> On 17 April 2018 at 14:54, Mark Thomas wrote: > On 17/04/18 11:36, Mark Thomas wrote: > On 17/04/18 10:14, Richard Tearle wrote: > Now all we need to to do is to figure out how to fix this. With the > understanding of what is (probably) going wrong, the problem can be > produced with a clean build and the certs we use for unit tests which > makes things a lot easier. I hope to make progress on this today. I believe this is fixed in trunk for both 9.0.x and 8.5.x. Are you able to build either of those from source and test? If not, I can provide a snapshot build for you to test with. Cheers, Mark >>> >>> I've built from source, re-enabled the health check, and so >>> far so good - it's been running an hour without an error. >>> >>> Once this run has finished, I'll run it overnight, and let you >>> know tomorrow morning. >>> >>> Many thanks for your help on this. >>> >>> -- >>> Richard >> >> Just a quick follow up - I've run our test for 8 hours, without the >> connection closed error. > > Excellent. That is really good news. > >> Again, many thanks. > > No problem. Happy to help. Thanks for your assistance with this issue. > Your test case and debug logs were invaluable. I couldn't have fixed > this without them. > > Mark > Do you mind to share more about the root cause? I’ve followed this mail communication from the start and am curious. Let me tell you that your endurance on all the tricky issues here is admirable! Thank you for that! Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 18/04/18 10:36, Richard Tearle wrote: > On 17 April 2018 at 16:45, Richard Tearle >wrote: >> On 17 April 2018 at 14:54, Mark Thomas wrote: >>> On 17/04/18 11:36, Mark Thomas wrote: On 17/04/18 10:14, Richard Tearle wrote: >>> >>> >>> Now all we need to to do is to figure out how to fix this. With the understanding of what is (probably) going wrong, the problem can be produced with a clean build and the certs we use for unit tests which makes things a lot easier. I hope to make progress on this today. >>> >>> I believe this is fixed in trunk for both 9.0.x and 8.5.x. >>> >>> Are you able to build either of those from source and test? If not, I >>> can provide a snapshot build for you to test with. >>> >>> Cheers, >>> >>> Mark >>> >> >> I've built from source, re-enabled the health check, and so >> far so good - it's been running an hour without an error. >> >> Once this run has finished, I'll run it overnight, and let you >> know tomorrow morning. >> >> Many thanks for your help on this. >> >> -- >> Richard > > Just a quick follow up - I've run our test for 8 hours, without the > connection closed error. Excellent. That is really good news. > Again, many thanks. No problem. Happy to help. Thanks for your assistance with this issue. Your test case and debug logs were invaluable. I couldn't have fixed this without them. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 17 April 2018 at 16:45, Richard Tearlewrote: > On 17 April 2018 at 14:54, Mark Thomas wrote: >> On 17/04/18 11:36, Mark Thomas wrote: >>> On 17/04/18 10:14, Richard Tearle wrote: >> >> >> >>> Now all we need to to do is to figure out how to fix this. With the >>> understanding of what is (probably) going wrong, the problem can be >>> produced with a clean build and the certs we use for unit tests which >>> makes things a lot easier. I hope to make progress on this today. >> >> I believe this is fixed in trunk for both 9.0.x and 8.5.x. >> >> Are you able to build either of those from source and test? If not, I >> can provide a snapshot build for you to test with. >> >> Cheers, >> >> Mark >> > > I've built from source, re-enabled the health check, and so > far so good - it's been running an hour without an error. > > Once this run has finished, I'll run it overnight, and let you > know tomorrow morning. > > Many thanks for your help on this. > > -- > Richard Just a quick follow up - I've run our test for 8 hours, without the connection closed error. Again, many thanks. -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 17 April 2018 at 14:54, Mark Thomaswrote: > On 17/04/18 11:36, Mark Thomas wrote: >> On 17/04/18 10:14, Richard Tearle wrote: > > > >> Now all we need to to do is to figure out how to fix this. With the >> understanding of what is (probably) going wrong, the problem can be >> produced with a clean build and the certs we use for unit tests which >> makes things a lot easier. I hope to make progress on this today. > > I believe this is fixed in trunk for both 9.0.x and 8.5.x. > > Are you able to build either of those from source and test? If not, I > can provide a snapshot build for you to test with. > > Cheers, > > Mark > I've built from source, re-enabled the health check, and so far so good - it's been running an hour without an error. Once this run has finished, I'll run it overnight, and let you know tomorrow morning. Many thanks for your help on this. -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 17/04/18 11:36, Mark Thomas wrote: > On 17/04/18 10:14, Richard Tearle wrote: >> I've also disabled the health check on ESB container, and my tests >> ran through for an hour, without a connection closed error. > > That is good news. That is a strong indicator that we are on the right > track. It also explains why I could not reproduce the problem with your > test case. And finally, it is another example of the debug logging added > to the I/O layer proving worth while. > > Now all we need to to do is to figure out how to fix this. With the > understanding of what is (probably) going wrong, the problem can be > produced with a clean build and the certs we use for unit tests which > makes things a lot easier. I hope to make progress on this today. I believe this is fixed in trunk for both 9.0.x and 8.5.x. Are you able to build either of those from source and test? If not, I can provide a snapshot build for you to test with. Cheers, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 17/04/18 10:14, Richard Tearle wrote: > On 16 April 2018 at 22:04, Mark Thomaswrote: >> I've started to look at them. I don't have any firm conclusions yet. I >> have noticed that the problem occurs after a connection is made to the >> service from localhost rather than the remote IP that is making the >> other requests. The localhost client does not present a certificate. >> >> My working theory (so chances are it is completely wrong) is that the >> missing certificate in the request from localhost puts the OpenSSL >> engine into an error state that is not correctly handled by Tomcat >> causing the subsequent request to fail. >> >> I've also noticed that the debug level log message consistently report 0 >> bytes being read which looks wrong but is probably a separate (minor) issue. The above message are correct but misleading. I'm planning to add additional debug logging which should clarify things. > Ah that rings a bell. > > Our containers have a simple health check, simply does > > curl --connect-timeout 5 --max-time 20 -k -s -S --stderr -\ > https://localhost:${TOMCATS_PORT}/ |\ > grep -q "NSS: client certificate not found" || exit 1 > > just to make sure the ESB is responding, with something we expect. > These are set to run at an interval of every 2m30s. The full parameters > in the docker-compose[1] file are: > > healthcheck: > test: ["CMD", "/usr/local/bin/healthcheck.sh"] > interval: 2m30s > timeout: 10s > retries: 3 > start_period: 20s > > I've also disabled the health check on ESB container, and my tests > ran through for an hour, without a connection closed error. That is good news. That is a strong indicator that we are on the right track. It also explains why I could not reproduce the problem with your test case. And finally, it is another example of the debug logging added to the I/O layer proving worth while. Now all we need to to do is to figure out how to fix this. With the understanding of what is (probably) going wrong, the problem can be produced with a clean build and the certs we use for unit tests which makes things a lot easier. I hope to make progress on this today. Mark > > [1] https://docs.docker.com/compose/compose-file/#healthcheck > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 16 April 2018 at 22:04, Mark Thomaswrote: > On 11/04/18 09:22, Richard Tearle wrote: > > > >> I've built tomcat from source using the link you provided, and rebuilt the >> containers with this tomcat, and can still reproduce the issue. I've uploaded >> the logs (30s before the connection closed error), to dropbox: >> >> https://www.dropbox.com/s/qe50jbd196krtyo/logs-10-04-17.zip?dl=0 > > Thanks for these. > > I've started to look at them. I don't have any firm conclusions yet. I > have noticed that the problem occurs after a connection is made to the > service from localhost rather than the remote IP that is making the > other requests. The localhost client does not present a certificate. > > My working theory (so chances are it is completely wrong) is that the > missing certificate in the request from localhost puts the OpenSSL > engine into an error state that is not correctly handled by Tomcat > causing the subsequent request to fail. > > I've also noticed that the debug level log message consistently report 0 > bytes being read which looks wrong but is probably a separate (minor) issue. > > Investigations continue. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > Ah that rings a bell. Our containers have a simple health check, simply does curl --connect-timeout 5 --max-time 20 -k -s -S --stderr -\ https://localhost:${TOMCATS_PORT}/ |\ grep -q "NSS: client certificate not found" || exit 1 just to make sure the ESB is responding, with something we expect. These are set to run at an interval of every 2m30s. The full parameters in the docker-compose[1] file are: healthcheck: test: ["CMD", "/usr/local/bin/healthcheck.sh"] interval: 2m30s timeout: 10s retries: 3 start_period: 20s I've also disabled the health check on ESB container, and my tests ran through for an hour, without a connection closed error. [1] https://docs.docker.com/compose/compose-file/#healthcheck -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 11/04/18 09:22, Richard Tearle wrote: > I've built tomcat from source using the link you provided, and rebuilt the > containers with this tomcat, and can still reproduce the issue. I've uploaded > the logs (30s before the connection closed error), to dropbox: > > https://www.dropbox.com/s/qe50jbd196krtyo/logs-10-04-17.zip?dl=0 Thanks for these. I've started to look at them. I don't have any firm conclusions yet. I have noticed that the problem occurs after a connection is made to the service from localhost rather than the remote IP that is making the other requests. The localhost client does not present a certificate. My working theory (so chances are it is completely wrong) is that the missing certificate in the request from localhost puts the OpenSSL engine into an error state that is not correctly handled by Tomcat causing the subsequent request to fail. I've also noticed that the debug level log message consistently report 0 bytes being read which looks wrong but is probably a separate (minor) issue. Investigations continue. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 5 April 2018 at 08:35, Richard Tearlewrote: > > On 4 April 2018 at 17:58, Mark Thomas wrote: > > On 26/03/18 08:25, Richard Tearle wrote: > > > > > > > > Thanks. I've got the test application and UI running but I haven't yet > > reproduced the problem. What parameters are you calling run-test.sh with? > > This usually get's a failure within 10 minutes of running: > > ./run-test.sh 28800 5000 5000 0 0 true > > (I've just tried it and it failed after 4m 23s - from the previous rounds > of testing, it failed at around the 4m 30s mark 7 times out of 10) > > > I'll continue to try and reproduce the issue but I think it makes sense > > to try and generate some debug data on your system as you can reproduce it. > > > > I'll get onto this, but it might not be until next week. > > Thanks > > Richard. I've built tomcat from source using the link you provided, and rebuilt the containers with this tomcat, and can still reproduce the issue. I've uploaded the logs (30s before the connection closed error), to dropbox: https://www.dropbox.com/s/qe50jbd196krtyo/logs-10-04-17.zip?dl=0 Regards -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 4 April 2018 at 17:58, Mark Thomaswrote: > On 26/03/18 08:25, Richard Tearle wrote: > > > > Thanks. I've got the test application and UI running but I haven't yet > reproduced the problem. What parameters are you calling run-test.sh with? This usually get's a failure within 10 minutes of running: ./run-test.sh 28800 5000 5000 0 0 true (I've just tried it and it failed after 4m 23s - from the previous rounds of testing, it failed at around the 4m 30s mark 7 times out of 10) > I'll continue to try and reproduce the issue but I think it makes sense > to try and generate some debug data on your system as you can reproduce it. > I'll get onto this, but it might not be until next week. Thanks Richard. -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 26/03/18 08:25, Richard Tearle wrote: > I've uploaded a ZIP with my test "UI" code (standalone java program), > and the "ESB" > code which goes into tomcat. > > https://www.dropbox.com/s/nhfx7va4uzkr728/Source.zip?dl=0 > > In the support folder within the ZIP are updated scripts to create the > certificates - which > now includes generating the client certificate as well. Also in there > are the server.xml > and other tomcat configuration files that are changed as part of our > installation process > - although these are the same as I'd included in the previous ZIP. > > Also included is a very simple shell script I use to call the UI. > Usually setting the ESB > delay to 5 seconds causes the connection closed error to occur in > around 5 minutes of > running the program. Thanks. I've got the test application and UI running but I haven't yet reproduced the problem. What parameters are you calling run-test.sh with? I'll continue to try and reproduce the issue but I think it makes sense to try and generate some debug data on your system as you can reproduce it. The first step is to move to testing on a Tomcat instance built from source. This will enable me to add additional debug logging as we go to try and narrow down the root cause. The build process is fairly simple. You'll need to check out https://svn.apache.org/repos/asf/tomcat/trunk/ and follow the instructions in BUILDING.txt. You won't need a release build. The default ant deploy should be fine. Assuming you can still repeat this on a build from trunk, please enable the following logging: - SSL debug logging on client and server via -Djavax.net.debug=ssl - Tomcat I/O layer debug logging by adding the following to conf/logging.properties: org.apache.tomcat.util.net.level = FINE If you can repeat the issue with the above debug logging enabled, if you can provide the logs for the 30s or so before the error occurs hopefully there will be some clues in there as to what is going wrong. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Hi On 24 March 2018 at 23:06, Mark Thomaswrote: > On 23/03/18 15:00, Richard Tearle wrote: >> On 22 March 2018 at 23:06, Mark Thomas wrote: >>> On 22/03/18 15:27, Richard Tearle wrote: On 22 March 2018 at 14:49, Mark Thomas wrote: >>> >>> >>> > > I've taken another look at the configuration options and > disableSessionTickets="true" is the only one that stands out as a > possibility. > > I think we have reached the point where we need a reproducible test case. > > Mark > I've uploaded a ZIP with my test "UI" code (standalone java program), and the "ESB" code which goes into tomcat. https://www.dropbox.com/s/nhfx7va4uzkr728/Source.zip?dl=0 In the support folder within the ZIP are updated scripts to create the certificates - which now includes generating the client certificate as well. Also in there are the server.xml and other tomcat configuration files that are changed as part of our installation process - although these are the same as I'd included in the previous ZIP. Also included is a very simple shell script I use to call the UI. Usually setting the ESB delay to 5 seconds causes the connection closed error to occur in around 5 minutes of running the program. Kinds Regards -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 23/03/18 15:00, Richard Tearle wrote: > On 22 March 2018 at 23:06, Mark Thomaswrote: >> On 22/03/18 15:27, Richard Tearle wrote: >>> On 22 March 2018 at 14:49, Mark Thomas wrote: >> >> >> >> OK. Time to think about this. NIO + JSSE works whereas NIO + OpenSSL >> doesn't with the same configuration apart from the presence of the >> native library. >> >> That points to something OpenSSL specific. >> >> Disabling client verification fixes the problem. >> >> So it looks to be something to do with how OpenSSL handles client >> verification. It feels like configuration at this point rather than a >> bug but it needs some more thought. >> >> There will probably be some configuration combinations to experiment >> with but if they fail, something we can use to reproduce this is going >> to be the next step. >> >> Mark >> > > That's fine and many thanks for your help - I can get quite a good turn around > on testing various configuration changes. Anything that looks > promising, I'll run > for 8 hours, and we can usually get an inkling after an hour. I've taken another look at the configuration options and disableSessionTickets="true" is the only one that stands out as a possibility. I think we have reached the point where we need a reproducible test case. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 22 March 2018 at 23:06, Mark Thomaswrote: > On 22/03/18 15:27, Richard Tearle wrote: >> On 22 March 2018 at 14:49, Mark Thomas wrote: > > > > OK. Time to think about this. NIO + JSSE works whereas NIO + OpenSSL > doesn't with the same configuration apart from the presence of the > native library. > > That points to something OpenSSL specific. > > Disabling client verification fixes the problem. > > So it looks to be something to do with how OpenSSL handles client > verification. It feels like configuration at this point rather than a > bug but it needs some more thought. > > There will probably be some configuration combinations to experiment > with but if they fail, something we can use to reproduce this is going > to be the next step. > > Mark > That's fine and many thanks for your help - I can get quite a good turn around on testing various configuration changes. Anything that looks promising, I'll run for 8 hours, and we can usually get an inkling after an hour. -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 22/03/18 15:27, Richard Tearle wrote: > On 22 March 2018 at 14:49, Mark Thomaswrote: >> What we have so far is: >> >> 8.0.x, http-nio- (this is always JSSE in 8.0.x), clientAuth="true" >> This works. > > Yes this works. > >> 8.5.x, http-nio-openssl-, certificateVerification="required" >> This fails intermittently > > Its https-openssl-nio- and yes this fails intermittently. > >> 8.5.x, http-nio-jsse-, certificateVerification="required" >> This works > > Its https-jsse-nio-, and yes this works > >> Is this correct? >> >> Thanks, >> >> Mark >> > > Also working is 8.5.x, https-openssl-nio-, certificateVerification="none" OK. Time to think about this. NIO + JSSE works whereas NIO + OpenSSL doesn't with the same configuration apart from the presence of the native library. That points to something OpenSSL specific. Disabling client verification fixes the problem. So it looks to be something to do with how OpenSSL handles client verification. It feels like configuration at this point rather than a bug but it needs some more thought. There will probably be some configuration combinations to experiment with but if they fail, something we can use to reproduce this is going to be the next step. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 22 March 2018 at 14:49, Mark Thomaswrote: > On 22/03/18 07:46, Richard Tearle wrote: >> On 21 March 2018 at 14:54, Mark Thomas wrote: [snip] > Excellent. > > There have been a few moving parts here so I'd like to get some > clarification on exactly where we are. I know from bitter personal > experience that it is all too easy to end up using a slightly different > TLS configuration to the one you think you are using so please could you > confirm the following. > > The connector name can be obtained from the logs. You'll see lines that > look like this: > > 22-Mar-2018 14:39:30.156 INFO [main] > org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler > ["http-nio-8080"] > 22-Mar-2018 14:39:30.161 INFO [main] > org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler > ["https-jsse-nio-8443"] > 22-Mar-2018 14:39:30.163 INFO [main] > org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler > ["ajp-nio-8009"] > > The part I am using below is the bit in the square brackes. The format > is ---. > > What we have so far is: > > 8.0.x, http-nio- (this is always JSSE in 8.0.x), clientAuth="true" > This works. Yes this works. > 8.5.x, http-nio-openssl-, certificateVerification="required" > This fails intermittently Its https-openssl-nio- and yes this fails intermittently. > 8.5.x, http-nio-jsse-, certificateVerification="required" > This works Its https-jsse-nio-, and yes this works > Is this correct? > > Thanks, > > Mark > Also working is 8.5.x, https-openssl-nio-, certificateVerification="none" Thanks -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 22/03/18 07:46, Richard Tearle wrote: > On 21 March 2018 at 14:54, Mark Thomaswrote: >> Please can you test your set-up with 8.5.x, the modified trust store and >> the same configuration as 8.0.x (NIO, JSSE). That should help us track >> down where the problem may lie. >> >> Thanks, >> >> Mark >> > > I created the PKCS12 as you showed above used my 8.0.x configuration and > ran my test application for 8 hours without a single connection closed error . Excellent. There have been a few moving parts here so I'd like to get some clarification on exactly where we are. I know from bitter personal experience that it is all too easy to end up using a slightly different TLS configuration to the one you think you are using so please could you confirm the following. The connector name can be obtained from the logs. You'll see lines that look like this: 22-Mar-2018 14:39:30.156 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 22-Mar-2018 14:39:30.161 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["https-jsse-nio-8443"] 22-Mar-2018 14:39:30.163 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] The part I am using below is the bit in the square brackes. The format is ---. What we have so far is: 8.0.x, http-nio- (this is always JSSE in 8.0.x), clientAuth="true" This works. 8.5.x, http-nio-openssl-, certificateVerification="required" This fails intermittently 8.5.x, http-nio-jsse-, certificateVerification="required" This works Is this correct? Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 21 March 2018 at 14:54, Mark Thomaswrote: > > > Progress. > > Tomcat 8.0.x is more relaxed about the content of PKCS12 trust stores > then 8.5.x because of a change[1] made so that the effectiveness of the > certificateVerificationDepth configuration attribute did not depend on > the presence of a certificate revocation list. > > The PKCS12 store the scripts you provided creates includes the private > key of the trusted certificate. This is ... unusual. 8.5.x skips this > cert as it does not expect a trusted cert to include the private key. > > I've tried various ways to get openssl to create a PKCS12 file without > the private key but with the certificate without success. In the end I > used keytool to do this and that worked. Something along these lines: > > keytool -storetype pkcs12 -importcert -file ca-cert.pem \ > -keystore ca-truststore.p12 > > With the modified trust store 8.5.x started with the same configuration > as 8.0.x. > > Please can you test your set-up with 8.5.x, the modified trust store and > the same configuration as 8.0.x (NIO, JSSE). That should help us track > down where the problem may lie. > > Thanks, > > Mark > I created the PKCS12 as you showed above used my 8.0.x configuration and ran my test application for 8 hours without a single connection closed error . -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 21/03/18 12:14, Mark Thomas wrote: > On 21/03/18 08:35, Richard Tearle wrote: >> On 20 March 2018 at 19:58, Mark Thomaswrote: >> >>> On 20/03/18 14:49, Richard Tearle wrote: >>> OK. Can you share you configuration and the steps you used to create the >>> self-signed certificate. I'd like to see if I can reproduce this. >>> >>> >>> Mark >>> >> >> I thought it might be easier to drop the configuration and certificate >> generating scripts into a ZIP on dropbox: >> >> https://www.dropbox.com/s/ib98y6ti2bem53v/TomcatCertsIssue.zip?dl=0 >> >> In the root of the ZIP contains two scripts, run the create-cert.sh, >> to generate them. >> >> Our Java installation has the Java Cryptography Extension (JCE) >> installed, and generally we run with the java security manager >> enabled, but I've tested running without it doesn't seem to affect the >> error we get. > > Perfect. I tweaked the scripts slightly but that was a lot easier than > starting from scratch. > > I can reproduce the problem. I need to do some more research to figure > out what is going wrong and if we can fix it. Progress. Tomcat 8.0.x is more relaxed about the content of PKCS12 trust stores then 8.5.x because of a change[1] made so that the effectiveness of the certificateVerificationDepth configuration attribute did not depend on the presence of a certificate revocation list. The PKCS12 store the scripts you provided creates includes the private key of the trusted certificate. This is ... unusual. 8.5.x skips this cert as it does not expect a trusted cert to include the private key. I've tried various ways to get openssl to create a PKCS12 file without the private key but with the certificate without success. In the end I used keytool to do this and that worked. Something along these lines: keytool -storetype pkcs12 -importcert -file ca-cert.pem \ -keystore ca-truststore.p12 With the modified trust store 8.5.x started with the same configuration as 8.0.x. Please can you test your set-up with 8.5.x, the modified trust store and the same configuration as 8.0.x (NIO, JSSE). That should help us track down where the problem may lie. Thanks, Mark [1] https://svn.apache.org/viewvc?view=revision=1757578 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 21/03/18 08:35, Richard Tearle wrote: > On 20 March 2018 at 19:58, Mark Thomaswrote: > >> On 20/03/18 14:49, Richard Tearle wrote: >> OK. Can you share you configuration and the steps you used to create the >> self-signed certificate. I'd like to see if I can reproduce this. >> >> >> Mark >> > > I thought it might be easier to drop the configuration and certificate > generating scripts into a ZIP on dropbox: > > https://www.dropbox.com/s/ib98y6ti2bem53v/TomcatCertsIssue.zip?dl=0 > > In the root of the ZIP contains two scripts, run the create-cert.sh, > to generate them. > > Our Java installation has the Java Cryptography Extension (JCE) > installed, and generally we run with the java security manager > enabled, but I've tested running without it doesn't seem to affect the > error we get. Perfect. I tweaked the scripts slightly but that was a lot easier than starting from scratch. I can reproduce the problem. I need to do some more research to figure out what is going wrong and if we can fix it. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 20 March 2018 at 19:58, Mark Thomaswrote: > On 20/03/18 14:49, Richard Tearle wrote: > OK. Can you share you configuration and the steps you used to create the > self-signed certificate. I'd like to see if I can reproduce this. > > > Mark > I thought it might be easier to drop the configuration and certificate generating scripts into a ZIP on dropbox: https://www.dropbox.com/s/ib98y6ti2bem53v/TomcatCertsIssue.zip?dl=0 In the root of the ZIP contains two scripts, run the create-cert.sh, to generate them. Our Java installation has the Java Cryptography Extension (JCE) installed, and generally we run with the java security manager enabled, but I've tested running without it doesn't seem to affect the error we get. -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 20/03/18 14:49, Richard Tearle wrote: > Hello > > On 20 March 2018 at 11:29, Mark Thomaswrote: >> >> On 20/03/18 07:52, Richard Tearle wrote: >>> Hello >>> >>> We have 4 applications built on the same architecture with a web UI >>> and camel based ESB running in separate Tomcat's, using REST/XML to >>> communicate between the two. This is all deployed within separate >>> Docker containers but on the same VM (at least for test), either on >>> Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've >>> been upgrading to Tomcat 8.5.x since November last year, but this has >>> been hampered by what looks to be random connection closed errors when >>> our UI communicates to the ESB. We have a series of Selenium based >>> autotests which will fail in different places, but with the same >>> error: >> >> >> >> There are rather too many factors at play here. It would be good to try >> and eliminate some of them. >> >> What are the known working 8.0.x versions? >> >> I looks like you are using JSSE with 8.0.x. It should be possible to use >> the exact same configuration with 8.5.x. No need to use the native >> library and no need to switch to the new configuration style. >> >> Lets try and get 8.5.x working with JSSE. That should help narrow down >> the root cause. What happens when you transfer the working 8.0.x config >> to 8.5.x? > > On startup we get: > > 20-Mar-2018 14:43:18.908 SEVERE [main] > org.apache.catalina.util.LifecycleBase.handleSubClassException Failed > to initialize component [Connector[HTTP/1.1-4001]] > org.apache.catalina.LifecycleException: Protocol handler initialization > failed > at > org.apache.catalina.connector.Connector.initInternal(Connector.java:935) > at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) > at > org.apache.catalina.core.StandardService.initInternal(StandardService.java:530) > at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) > at > org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:852) > at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) > at org.apache.catalina.startup.Catalina.load(Catalina.java:633) > at org.apache.catalina.startup.Catalina.load(Catalina.java:656) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309) > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492) > Caused by: java.lang.IllegalArgumentException: the trustAnchors > parameter must be non-empty > at > org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114) > at > org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:85) > at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216) > at > org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1043) > at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:540) > at > org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74) > at > org.apache.catalina.connector.Connector.initInternal(Connector.java:932) > ... 13 more > Caused by: java.security.InvalidAlgorithmParameterException: the > trustAnchors parameter must be non-empty > at > java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200) > at java.security.cert.PKIXParameters.(PKIXParameters.java:157) > at > java.security.cert.PKIXBuilderParameters.(PKIXBuilderParameters.java:130) > at > org.apache.tomcat.util.net.jsse.JSSEUtil.getParameters(JSSEUtil.java:389) > at > org.apache.tomcat.util.net.jsse.JSSEUtil.getTrustManagers(JSSEUtil.java:313) > at > org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112) > ... 19 more OK. Can you share you configuration and the steps you used to create the self-signed certificate. I'd like to see if I can reproduce this. >> Also, anything you can do to reduce the complexity of the test >> application (ideally reducing it to simple Servlets/JSPs) would make it >> easier for others to reproduce the issue. > > I can ZIP my code and drop it somewhere if that will help. Yes. But lets get the JSSE issue fixed first. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
On 20 March 2018 at 14:49, Richard Tearlewrote: > Hello > > On 20 March 2018 at 11:29, Mark Thomas wrote: >> >> >> >> There are rather too many factors at play here. It would be good to try >> and eliminate some of them. >> >> What are the known working 8.0.x versions? >> Sorry I missed these in my previous reply: 8.0.50, 8.0.48 to 8.0.46, 8.0.41. 8.0.33, 8.0.26 -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and intended solely for the addressee(s). If you are not the intended recipient of this email you must: (i) not disclose, copy or distribute its contents to any other person nor use its contents in any way or you may be acting unlawfully; (ii) contact Northgate Public Services immediately on +44(0)1442 768445 quoting the name of the sender and the addressee then delete it from your system. Northgate Public Services has taken reasonable precautions to ensure that no viruses are contained in this email, but does not accept any responsibility once this email has been transmitted. You should scan attachments (if any) for viruses. Northgate Public Services (UK) Limited, registered in England and Wales under number 00968498 with a registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India) Pvt Limited, registered in India under number 117068 with a registered address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 41. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Hello On 20 March 2018 at 11:29, Mark Thomaswrote: > > On 20/03/18 07:52, Richard Tearle wrote: > > Hello > > > > We have 4 applications built on the same architecture with a web UI > > and camel based ESB running in separate Tomcat's, using REST/XML to > > communicate between the two. This is all deployed within separate > > Docker containers but on the same VM (at least for test), either on > > Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've > > been upgrading to Tomcat 8.5.x since November last year, but this has > > been hampered by what looks to be random connection closed errors when > > our UI communicates to the ESB. We have a series of Selenium based > > autotests which will fail in different places, but with the same > > error: > > > > There are rather too many factors at play here. It would be good to try > and eliminate some of them. > > What are the known working 8.0.x versions? > > I looks like you are using JSSE with 8.0.x. It should be possible to use > the exact same configuration with 8.5.x. No need to use the native > library and no need to switch to the new configuration style. > > Lets try and get 8.5.x working with JSSE. That should help narrow down > the root cause. What happens when you transfer the working 8.0.x config > to 8.5.x? On startup we get: 20-Mar-2018 14:43:18.908 SEVERE [main] org.apache.catalina.util.LifecycleBase.handleSubClassException Failed to initialize component [Connector[HTTP/1.1-4001]] org.apache.catalina.LifecycleException: Protocol handler initialization failed at org.apache.catalina.connector.Connector.initInternal(Connector.java:935) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardService.initInternal(StandardService.java:530) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:852) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.startup.Catalina.load(Catalina.java:633) at org.apache.catalina.startup.Catalina.load(Catalina.java:656) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492) Caused by: java.lang.IllegalArgumentException: the trustAnchors parameter must be non-empty at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114) at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:85) at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1043) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:540) at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74) at org.apache.catalina.connector.Connector.initInternal(Connector.java:932) ... 13 more Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200) at java.security.cert.PKIXParameters.(PKIXParameters.java:157) at java.security.cert.PKIXBuilderParameters.(PKIXBuilderParameters.java:130) at org.apache.tomcat.util.net.jsse.JSSEUtil.getParameters(JSSEUtil.java:389) at org.apache.tomcat.util.net.jsse.JSSEUtil.getTrustManagers(JSSEUtil.java:313) at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112) ... 19 more > Also, anything you can do to reduce the complexity of the test > application (ideally reducing it to simple Servlets/JSPs) would make it > easier for others to reproduce the issue. I can ZIP my code and drop it somewhere if that will help. > Hmm. That looks like a controlled shutdown. Random thought, does setting > disableSessionTickets="true" help at all when using OpenSSL? > I'm afraid that didn't work, but thanks for the suggestion. > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- Richard -- This email is sent on behalf of Northgate Public Services (UK) Limited and its associated companies including Rave Technologies (India) Pvt Limited (together "Northgate Public Services") and is strictly confidential and
Re: Connection closed error and certificateVerification="required"
On 20/03/18 07:52, Richard Tearle wrote: > Hello > > We have 4 applications built on the same architecture with a web UI > and camel based ESB running in separate Tomcat's, using REST/XML to > communicate between the two. This is all deployed within separate > Docker containers but on the same VM (at least for test), either on > Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've > been upgrading to Tomcat 8.5.x since November last year, but this has > been hampered by what looks to be random connection closed errors when > our UI communicates to the ESB. We have a series of Selenium based > autotests which will fail in different places, but with the same > error: > We've seen this on all of the versions of Tomcat 8.5 we've tried from > 8.5.23 onwards - and I've just tried Tomcat 9.0.6 with the same error. > We're using Oracle Java version 1.8u162. > > Development have looked at this off and on over the last couple of > months, but without success, so I was asked to reproduce this with a > simple test case (I'm devops usually, but with a development > background). The client stack uses a > PoolingNHttpClientConnectionManager, CloseableHttpAsyncClient and > AsyncRestTemplate to communicate to the ESB. My simple test case > replaces the web UI with a simple jar based application, but it still > replicates the issue. The ESB is a simple Camel route that adds two > supplied numbers, waits for a time (supplied by the UI), and returns > the result. The UI repeatedly calls the ESB (on a single thread, with > no overlapping calls), with two random numbers to be summed, and a set > amount of time to wait before the ESB returns. Between each call to > the ESB there's a set amount of time to wait before the next call. It > seems that when the ESB wait is >= 5secs the time take to fail is > greatly decreased. > > The connection between the UI and the ESB is TLSv1.2 with the ESB > configured with certificateVerification="required". The connector > configuration with the ESB's server.xml is: > > protocol="org.apache.coyote.http11.Http11NioProtocol" >maxThreads="150" SSLEnabled="true" scheme="https" > secure="true" server="Apache" maxPostSize="10"> > caCertificateFile="/usr/local/tomcat/ssl/ca-cert.pem" > > certificateFile="/usr/local/tomcat/ssl/server-cert.pem" > > certificateKeyFile="/usr/local/tomcat/ssl/server-key.pem" > certificateKeyPassword="${keystore.pass}"> > > > > > We are using PEM encoded certificates, and using Tomcat Native Library > version 1.2.16. If I take out certificateVerification="required", we > don't get the error (with my simple test case running for an hour) > I've tried using PKCS12 and JKS certificates, but I can't get Tomcat > 8.5.x to work with our self signed trust store (we get the > "trustAnchors parameter must be non-empty" errors), although Tomcat > 8.0.x seems happy enough with them. So I can't tell if it's a > difference between using OpenSSL and JSSE. There are rather too many factors at play here. It would be good to try and eliminate some of them. What are the known working 8.0.x versions? I looks like you are using JSSE with 8.0.x. It should be possible to use the exact same configuration with 8.5.x. No need to use the native library and no need to switch to the new configuration style. Lets try and get 8.5.x working with JSSE. That should help narrow down the root cause. What happens when you transfer the working 8.0.x config to 8.5.x? Also, anything you can do to reduce the complexity of the test application (ideally reducing it to simple Servlets/JSPs) would make it easier for others to reproduce the issue. > Enabling SSL debug traces on the client shows that in the error case, > after the initial successful handshaking between the client and the > server, the ESB shuts down the connection just after the client has > sent the data: > > *** CertificateVerify > Signature Algorithm SHA512withRSA > I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 520 > I/O dispatcher 1, WRITE: TLSv1.2 Change Cipher Spec, length = 1 > *** Finished > verify_data: { 197, 39, 73, 181, 14, 87, 139, 81, 247, 181, 32, 17 } > *** > I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 40 > I/O dispatcher 1, READ: TLSv1.2 Change Cipher Spec, length = 1 > I/O dispatcher 1, READ: TLSv1.2 Handshake, length = 40 > *** Finished > verify_data: { 84, 164, 177, 160, 235, 23, 31, 20, 252, 149, 214, 245 } > *** > %% Cached client session: [Session-101, > TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384] > I/O dispatcher 1, WRITE: TLSv1.2 Application Data, length = 391 > I/O dispatcher 1, READ: TLSv1.2 Alert, length = 26 > I/O dispatcher 1, RECV TLSv1.2 ALERT: warning, close_notify > I/O dispatcher 1, closeInboundInternal() > I/O dispatcher 1, closeOutboundInternal() > I/O dispatcher 1, SEND TLSv1.2 ALERT: warning,
Connection closed error and certificateVerification="required"
Hello We have 4 applications built on the same architecture with a web UI and camel based ESB running in separate Tomcat's, using REST/XML to communicate between the two. This is all deployed within separate Docker containers but on the same VM (at least for test), either on Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've been upgrading to Tomcat 8.5.x since November last year, but this has been hampered by what looks to be random connection closed errors when our UI communicates to the ESB. We have a series of Selenium based autotests which will fail in different places, but with the same error: 09:30:46.057 [main] ERROR test.ui.Main - Catching java.util.concurrent.ExecutionException: org.apache.http.ConnectionClosedException: Connection closed at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:70) ~[test-ui.jar:?] at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:80) ~[test-ui.jar:?] at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70) ~[test-ui.jar:?] at org.springframework.util.concurrent.FutureAdapter.get(FutureAdapter.java:81) ~[test-ui.jar:?] at org.springframework.util.concurrent.FutureAdapter.get(FutureAdapter.java:81) ~[test-ui.jar:?] at test.ui.Main.post(Main.java:182) ~[test-ui.jar:?] at test.ui.Main.callServer(Main.java:148) [test-ui.jar:?] at test.ui.Main.main(Main.java:109) [test-ui.jar:?] Caused by: org.apache.http.ConnectionClosedException: Connection closed at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:345) ~[test-ui.jar:?] at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261) ~[test-ui.jar:?] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[test-ui.jar:?] at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[test-ui.jar:?] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[test-ui.jar:?] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_162] We've seen this on all of the versions of Tomcat 8.5 we've tried from 8.5.23 onwards - and I've just tried Tomcat 9.0.6 with the same error. We're using Oracle Java version 1.8u162. Development have looked at this off and on over the last couple of months, but without success, so I was asked to reproduce this with a simple test case (I'm devops usually, but with a development background). The client stack uses a PoolingNHttpClientConnectionManager, CloseableHttpAsyncClient and AsyncRestTemplate to communicate to the ESB. My simple test case replaces the web UI with a simple jar based application, but it still replicates the issue. The ESB is a simple Camel route that adds two supplied numbers, waits for a time (supplied by the UI), and returns the result. The UI repeatedly calls the ESB (on a single thread, with no overlapping calls), with two random numbers to be summed, and a set amount of time to wait before the ESB returns. Between each call to the ESB there's a set amount of time to wait before the next call. It seems that when the ESB wait is >= 5secs the time take to fail is greatly decreased. The connection between the UI and the ESB is TLSv1.2 with the ESB configured with certificateVerification="required". The connector configuration with the ESB's server.xml is: We are using PEM encoded certificates, and using Tomcat Native Library version 1.2.16. If I take out certificateVerification="required", we don't get the error (with my simple test case running for an hour) I've tried using PKCS12 and JKS certificates, but I can't get Tomcat 8.5.x to work with our self signed trust store (we get the "trustAnchors parameter must be non-empty" errors), although Tomcat 8.0.x seems happy enough with them. So I can't tell if it's a difference between using OpenSSL and JSSE. Enabling SSL debug traces on the client shows that in the error case, after the initial successful handshaking between the client and the server, the ESB shuts down the connection just after the client has sent the data: *** CertificateVerify