Re: Slow startup of webapps (e.g., manager) after migration

2019-06-21 Thread Osipov, Michael




Am 2019-06-21 um 14:33 schrieb Mark Thomas:

On 21/06/2019 13:13, Osipov, Michael wrote:

Folks,

we're migrating off old hosts to new ones, but remain at 8.5.x (.34 to
.42 at the moment) and Java 8. Surprisingly, an empty Tomcat wih just
manager installed takes way too long too start. More precisely, the
Tomcat is fast, deploying manager app is horribly slow:


That sounds like insufficient entropy.


That was my first idea too...


2019-06-21T13:50:35.233 INFORMATION
[deblndw024v.ad001.siemens.net-startStop-3]
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of
web application directory [/var/opt/tomcat-smartld/webapps/manager]
has finished in [47,818] ms
2019-06-21T13:07:40.484 INFORMATION
[deblndw024v.ad001.siemens.net-startStop-1]
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of
web application directory [/var/opt/tomcat-smartld/webapps/manager]
has finished in [126,041] ms
2019-06-21T13:17:35.885 INFORMATION
[deblndw024v.ad001.siemens.net-startStop-1]
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of
web application directory [/var/opt/tomcat-smartld/webapps/manager]
has finished in [80,368] ms


Which class do I need to set to FINEST to narrow down the cause?

org.apache.catalina.startup.HostConfig.level = FINEST

did not help. If nothing helps, I'd debug the Tomcat code.


If it is happening per webapp then I'd take a look at
SessionIdGeneratorBase but it should log a warning if it takes more than
100ms to start.

A series of 3 thread dumps ~5s apart while the app is starting may also
shed some light. I often find doing a diff between them very enlightening.


I have now done the undeploy/redeploy at least 50 times, the line from 
SessionIdGeneratorBase never appeared in the log file and the issue 
seems to be non-deterministic after a container (HP-UX) restart now. 
Restarting the JVM process doesn't make a difference. I have the feeling 
that this might be related to the HP-UX container which ran for weeks 
now and had to be restarted due to the OpenSSL update.


I will keep on eye on this and re-report when I am able to trace this 
properly. I do have the dumps, but they are pointless because the start 
takes mostly < 2000 ms now.


Thanks for the pointers,

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Slow startup of webapps (e.g., manager) after migration

2019-06-21 Thread Mark Thomas
On 21/06/2019 13:13, Osipov, Michael wrote:
> Folks,
> 
> we're migrating off old hosts to new ones, but remain at 8.5.x (.34 to
> .42 at the moment) and Java 8. Surprisingly, an empty Tomcat wih just
> manager installed takes way too long too start. More precisely, the
> Tomcat is fast, deploying manager app is horribly slow:

That sounds like insufficient entropy.

>> 2019-06-21T13:50:35.233 INFORMATION
>> [deblndw024v.ad001.siemens.net-startStop-3]
>> org.apache.catalina.startup.HostConfig.deployDirectory Deployment of
>> web application directory [/var/opt/tomcat-smartld/webapps/manager]
>> has finished in [47,818] ms
>> 2019-06-21T13:07:40.484 INFORMATION
>> [deblndw024v.ad001.siemens.net-startStop-1]
>> org.apache.catalina.startup.HostConfig.deployDirectory Deployment of
>> web application directory [/var/opt/tomcat-smartld/webapps/manager]
>> has finished in [126,041] ms
>> 2019-06-21T13:17:35.885 INFORMATION
>> [deblndw024v.ad001.siemens.net-startStop-1]
>> org.apache.catalina.startup.HostConfig.deployDirectory Deployment of
>> web application directory [/var/opt/tomcat-smartld/webapps/manager]
>> has finished in [80,368] ms
> 
> Which class do I need to set to FINEST to narrow down the cause?
>> org.apache.catalina.startup.HostConfig.level = FINEST
> did not help. If nothing helps, I'd debug the Tomcat code.

If it is happening per webapp then I'd take a look at
SessionIdGeneratorBase but it should log a warning if it takes more than
100ms to start.

A series of 3 thread dumps ~5s apart while the app is starting may also
shed some light. I often find doing a diff between them very enlightening.

Mark


> Note: I haven't tried any of our own apps for now, but this slow
> deployment gives me creeps.
> 
> FWIW, I am running off:
>> 2019-06-21T13:16:14.371 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Server
>> version:    Apache Tomcat/8.5.42
>> 2019-06-21T13:16:14.376 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Server
>> built:  Jun 4 2019 20:29:04 UTC
>> 2019-06-21T13:16:14.377 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Server
>> number: 8.5.42.0
>> 2019-06-21T13:16:14.377 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log OS
>> Name:   HP-UX
>> 2019-06-21T13:16:14.377 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log OS
>> Version:    B.11.31
>> 2019-06-21T13:16:14.377 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log
>> Architecture:  IA64N
>> 2019-06-21T13:16:14.378 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log Java
>> Home: /opt/java8/jre
>> 2019-06-21T13:16:14.378 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log JVM
>> Version:   1.8.0.17-hp-ux-b1
>> 2019-06-21T13:16:14.379 INFORMATION [main]
>> org.apache.catalina.startup.VersionLoggerListener.log JVM
>> Vendor:    Hewlett Packard Enterprise Company
>> 2019-06-21T13:16:14.386 INFORMATION [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
>> APR based Apache Tomcat Native library [1.2.21] using APR version
>> [1.6.6].
>> 2019-06-21T13:16:14.386 INFORMATION [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
>> capabilities: IPv6 [true], sendfile [true], accept filters [false],
>> random [true].
>> 2019-06-21T13:16:14.386 INFORMATION [main]
>> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
>> APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]
>> 2019-06-21T13:16:14.401 INFORMATION [main]
>> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
>> successfully initialized [OpenSSL 1.0.2k  26 Jan 2017]
> 
> I have compiled all native components myself, except for Java and
> OpenSSL. They are provided by HPE in binary form.
> Note: OpenSSL update to 1.0.2r is pending, but this should not be cause
> here.>
> Michael
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
> 


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Slow startup of webapps (e.g., manager) after migration

2019-06-21 Thread Osipov, Michael

Folks,

we're migrating off old hosts to new ones, but remain at 8.5.x (.34 to 
.42 at the moment) and Java 8. Surprisingly, an empty Tomcat wih just 
manager installed takes way too long too start. More precisely, the 
Tomcat is fast, deploying manager app is horribly slow:



2019-06-21T13:50:35.233 INFORMATION [deblndw024v.ad001.siemens.net-startStop-3] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in 
[47,818] ms
2019-06-21T13:07:40.484 INFORMATION [deblndw024v.ad001.siemens.net-startStop-1] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in 
[126,041] ms
2019-06-21T13:17:35.885 INFORMATION [deblndw024v.ad001.siemens.net-startStop-1] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/var/opt/tomcat-smartld/webapps/manager] has finished in 
[80,368] ms


Which class do I need to set to FINEST to narrow down the cause?
> org.apache.catalina.startup.HostConfig.level = FINEST
did not help. If nothing helps, I'd debug the Tomcat code.

Note: I haven't tried any of our own apps for now, but this slow 
deployment gives me creeps.


FWIW, I am running off:

2019-06-21T13:16:14.371 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version:
Apache Tomcat/8.5.42
2019-06-21T13:16:14.376 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server built:  
Jun 4 2019 20:29:04 UTC
2019-06-21T13:16:14.377 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server number: 
8.5.42.0
2019-06-21T13:16:14.377 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Name:   
HP-UX
2019-06-21T13:16:14.377 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Version:
B.11.31
2019-06-21T13:16:14.377 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log Architecture:  
IA64N
2019-06-21T13:16:14.378 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log Java Home: 
/opt/java8/jre
2019-06-21T13:16:14.378 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:   
1.8.0.17-hp-ux-b1
2019-06-21T13:16:14.379 INFORMATION [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
Hewlett Packard Enterprise Company
2019-06-21T13:16:14.386 INFORMATION [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based 
Apache Tomcat Native library [1.2.21] using APR version [1.6.6].
2019-06-21T13:16:14.386 INFORMATION [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: 
IPv6 [true], sendfile [true], accept filters [false], random [true].
2019-06-21T13:16:14.386 INFORMATION [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL 
configuration: useAprConnector [true], useOpenSSL [true]
2019-06-21T13:16:14.401 INFORMATION [main] 
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL 
successfully initialized [OpenSSL 1.0.2k  26 Jan 2017]


I have compiled all native components myself, except for Java and 
OpenSSL. They are provided by HPE in binary form.
Note: OpenSSL update to 1.0.2r is pending, but this should not be cause 
here.


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org