On 29/09/2022 04:05, Jerry Malcolm wrote:
Hi, Mark, Thanks for the info. Getting several thread dumps is not going to be easy in this specific situation. This problem only occurs on initial boot up of a newly-imaged linux EC2 which is built and launched automatically by the AWS autoscaling service. By the time i can see that an image has been created and launched, get the newly assigned IP address from AWS, then go through the initialization of a puTTY session, then get the PID for tomcat, I'm pretty much out of time to try to get meaningful thread dumps. I have no doubt that the thread dump might tell us something. But short of some major restructuring and special scaffolding to obtain useful dumps, I want to exhaust all other alternatives to find the problem prior to going down that hole.
In your initial post you stated you also observed the problem in your local Windows development environment. Take the thread dumps there.
Mark
1) Are there any finer log levels I can turn on to get more data for initializing webapps?2) In general, what occurs on init of a webapp? I know it does tld scanning of jar files. But I'm not getting tld scan warnings. And any tld scans that do happen occur on every TC boot, right?3) Are the datasource pools being initialized such that there could be some database timeout wait occurring? (Not sure why there would be db timeouts, though... the db is up and running and not approaching connection limits)4) Does it load/cache every jar file in the lib folder on init5) Does it perhaps precompile all of the JSPs and cache them somewhere that the jvm might still have them on a TC reboot? When it's doing slow-load, the time varies between 13 and 19 seconds, and the time appears to be correlated to the number of JSPs in the web app.I know I'm just grasping at straws. But hopefully I can figure out something that is causing this super long load time without having to figure out the thread-dump on system boot procedure.Thanks again. Jerry On 9/28/2022 11:33 AM, Mark Thomas wrote:Lots of ways. I'd try starting with the jstack tool which should be provided as part of the JDK you are using.Mark On 28/09/2022 17:16, Jerry Malcolm wrote:Thanks, Mark, I'm not familiar with how to take a thread dump in TC. Can you point me to documentation on that?Jerry On 9/28/2022 2:56 AM, Mark Thomas wrote:Take three thread dumps 5s apart during the slowness and then diff them to see what is moving and what is not.Mark On 28/09/2022 07:30, Thomas Hoffmann (Speed4Trade GmbH) wrote:Hello, could the underlying hardware cause the delay?Maybe the OS has cached some data and don’t need to read it from disk again?Maybe you can check the IO and CPU load during startup and compare.-----Ursprüngliche Nachricht----- Von: Jerry Malcolm <techst...@malcolms.com> Gesendet: Mittwoch, 28. September 2022 04:54 An: users@tomcat.apache.org Betreff: Re: Slow startup first time after reboot Neil,Sadly, that line doesn't appear either with or without the java.security.egdoption. That appears to be a lump sum of 4-5 minutes for the SecureRandom seed thing. I'm getting a total accumulation of ~5minutes. But it's made up of a bunch of ~15-sec web starts. See example below. Same two lines from Catalina.out from the first boot and a secondboot. One other clue I've noted. The quick boot occurs on an immediatestop/start. I have noticed a couple of times in Eclipse that if I stop TC and leave it stopped for a while, it goes back to the long startup. It sounds like something related to loading modules/jarFiles. On an immediate restart theJVM wouldn't have time to unload modules before they are neededagain. But a new loading of some module on each web app might be what'skilling it (??) JVM Version: 11.0.16+8-LTS JVM Vendor: Red Hat, Inc. Auto-startup of TC on EC2 startup: 28-Sep-2022 00:15:55.612 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory[/var/domains/wridz.com/webapps/idmanager] has finished in [16,991] ms[1 example... ~20 more like it] 28-Sep-2022 00:18:40.818 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [306666] milliseconds Reboot of TC 28-Sep-2022 02:28:45.854 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory[/var/domains/wridz.com/webapps/idmanager] has finished in [737] ms [1example... ~20 more like it] 28-Sep-2022 02:28:51.476 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [19795] milliseconds On 9/27/2022 8:32 PM, Neil Aggarwal wrote:Are there perhaps some log levels I could change that would provide more detailedinformation about what step it's hung up on for loading these web apps?I just tested this on a dev sever. I removed the java.security.egd option and rebooted my server.Once I waited for Tomcat to finish starting up, catalina.out had this line:WARNING [main] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [222,741] milliseconds. Check if your log has something similar. Thank you, Neil -- Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com We offer 30 year loans on single family houses!---------------------------------------------------------------------To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
--------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
