Re: Slow startup of webapps (e.g., manager) after migration
Am 2019-06-21 um 14:33 schrieb Mark Thomas: On 21/06/2019 13:13, Osipov, Michael wrote: Folks, we're migrating off old hosts to new ones, but remain at 8.5.x (.34 to .42 at the moment) and Java 8. Surprisingly, an empty Tomcat wih just manager installed takes way too long too start. More precisely, the Tomcat is fast, deploying manager app is horribly slow: That sounds like insufficient entropy. That was my first idea too... 2019-06-21T13:50:35.233 INFORMATION [deblndw024v.ad001.siemens.net-startStop-3] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in [47,818] ms 2019-06-21T13:07:40.484 INFORMATION [deblndw024v.ad001.siemens.net-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in [126,041] ms 2019-06-21T13:17:35.885 INFORMATION [deblndw024v.ad001.siemens.net-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in [80,368] ms Which class do I need to set to FINEST to narrow down the cause? org.apache.catalina.startup.HostConfig.level = FINEST did not help. If nothing helps, I'd debug the Tomcat code. If it is happening per webapp then I'd take a look at SessionIdGeneratorBase but it should log a warning if it takes more than 100ms to start. A series of 3 thread dumps ~5s apart while the app is starting may also shed some light. I often find doing a diff between them very enlightening. I have now done the undeploy/redeploy at least 50 times, the line from SessionIdGeneratorBase never appeared in the log file and the issue seems to be non-deterministic after a container (HP-UX) restart now. Restarting the JVM process doesn't make a difference. I have the feeling that this might be related to the HP-UX container which ran for weeks now and had to be restarted due to the OpenSSL update. I will keep on eye on this and re-report when I am able to trace this properly. I do have the dumps, but they are pointless because the start takes mostly < 2000 ms now. Thanks for the pointers, Michael - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Slow startup of webapps (e.g., manager) after migration
On 21/06/2019 13:13, Osipov, Michael wrote: > Folks, > > we're migrating off old hosts to new ones, but remain at 8.5.x (.34 to > .42 at the moment) and Java 8. Surprisingly, an empty Tomcat wih just > manager installed takes way too long too start. More precisely, the > Tomcat is fast, deploying manager app is horribly slow: That sounds like insufficient entropy. >> 2019-06-21T13:50:35.233 INFORMATION >> [deblndw024v.ad001.siemens.net-startStop-3] >> org.apache.catalina.startup.HostConfig.deployDirectory Deployment of >> web application directory [/var/opt/tomcat-smartld/webapps/manager] >> has finished in [47,818] ms >> 2019-06-21T13:07:40.484 INFORMATION >> [deblndw024v.ad001.siemens.net-startStop-1] >> org.apache.catalina.startup.HostConfig.deployDirectory Deployment of >> web application directory [/var/opt/tomcat-smartld/webapps/manager] >> has finished in [126,041] ms >> 2019-06-21T13:17:35.885 INFORMATION >> [deblndw024v.ad001.siemens.net-startStop-1] >> org.apache.catalina.startup.HostConfig.deployDirectory Deployment of >> web application directory [/var/opt/tomcat-smartld/webapps/manager] >> has finished in [80,368] ms > > Which class do I need to set to FINEST to narrow down the cause? >> org.apache.catalina.startup.HostConfig.level = FINEST > did not help. If nothing helps, I'd debug the Tomcat code. If it is happening per webapp then I'd take a look at SessionIdGeneratorBase but it should log a warning if it takes more than 100ms to start. A series of 3 thread dumps ~5s apart while the app is starting may also shed some light. I often find doing a diff between them very enlightening. Mark > Note: I haven't tried any of our own apps for now, but this slow > deployment gives me creeps. > > FWIW, I am running off: >> 2019-06-21T13:16:14.371 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log Server >> version: Apache Tomcat/8.5.42 >> 2019-06-21T13:16:14.376 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log Server >> built: Jun 4 2019 20:29:04 UTC >> 2019-06-21T13:16:14.377 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log Server >> number: 8.5.42.0 >> 2019-06-21T13:16:14.377 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log OS >> Name: HP-UX >> 2019-06-21T13:16:14.377 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log OS >> Version: B.11.31 >> 2019-06-21T13:16:14.377 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log >> Architecture: IA64N >> 2019-06-21T13:16:14.378 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log Java >> Home: /opt/java8/jre >> 2019-06-21T13:16:14.378 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log JVM >> Version: 1.8.0.17-hp-ux-b1 >> 2019-06-21T13:16:14.379 INFORMATION [main] >> org.apache.catalina.startup.VersionLoggerListener.log JVM >> Vendor: Hewlett Packard Enterprise Company >> 2019-06-21T13:16:14.386 INFORMATION [main] >> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded >> APR based Apache Tomcat Native library [1.2.21] using APR version >> [1.6.6]. >> 2019-06-21T13:16:14.386 INFORMATION [main] >> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR >> capabilities: IPv6 [true], sendfile [true], accept filters [false], >> random [true]. >> 2019-06-21T13:16:14.386 INFORMATION [main] >> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent >> APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] >> 2019-06-21T13:16:14.401 INFORMATION [main] >> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL >> successfully initialized [OpenSSL 1.0.2k 26 Jan 2017] > > I have compiled all native components myself, except for Java and > OpenSSL. They are provided by HPE in binary form. > Note: OpenSSL update to 1.0.2r is pending, but this should not be cause > here.> > Michael > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Slow startup of webapps (e.g., manager) after migration
Folks, we're migrating off old hosts to new ones, but remain at 8.5.x (.34 to .42 at the moment) and Java 8. Surprisingly, an empty Tomcat wih just manager installed takes way too long too start. More precisely, the Tomcat is fast, deploying manager app is horribly slow: 2019-06-21T13:50:35.233 INFORMATION [deblndw024v.ad001.siemens.net-startStop-3] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in [47,818] ms 2019-06-21T13:07:40.484 INFORMATION [deblndw024v.ad001.siemens.net-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in [126,041] ms 2019-06-21T13:17:35.885 INFORMATION [deblndw024v.ad001.siemens.net-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in [80,368] ms Which class do I need to set to FINEST to narrow down the cause? > org.apache.catalina.startup.HostConfig.level = FINEST did not help. If nothing helps, I'd debug the Tomcat code. Note: I haven't tried any of our own apps for now, but this slow deployment gives me creeps. FWIW, I am running off: 2019-06-21T13:16:14.371 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.5.42 2019-06-21T13:16:14.376 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Jun 4 2019 20:29:04 UTC 2019-06-21T13:16:14.377 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.5.42.0 2019-06-21T13:16:14.377 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: HP-UX 2019-06-21T13:16:14.377 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: B.11.31 2019-06-21T13:16:14.377 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: IA64N 2019-06-21T13:16:14.378 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /opt/java8/jre 2019-06-21T13:16:14.378 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0.17-hp-ux-b1 2019-06-21T13:16:14.379 INFORMATION [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Hewlett Packard Enterprise Company 2019-06-21T13:16:14.386 INFORMATION [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.6.6]. 2019-06-21T13:16:14.386 INFORMATION [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 2019-06-21T13:16:14.386 INFORMATION [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] 2019-06-21T13:16:14.401 INFORMATION [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.0.2k 26 Jan 2017] I have compiled all native components myself, except for Java and OpenSSL. They are provided by HPE in binary form. Note: OpenSSL update to 1.0.2r is pending, but this should not be cause here. Michael - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org