RE: Speeding up tomcat
To follow up with my request for help. we made 3 adjustments. added in org.apache.jasper.servlet.JspServlet checkInterval 300 development false server.xml context.xml added Response times on the website went from average of 3seconds per click to average of 1.5seconds per click to complete the page load. -Original Message- From: Christopher Schultz Sent: Monday, March 25, 2024 6:37 PM To: users@tomcat.apache.org Subject: Re: Speeding up tomcat [You don't often get email from ch...@christopherschultz.net. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] [EXTERNAL] Alex, On 3/25/24 13:06, Alex Hatcher wrote: > Java version: 11.0.20 > Tomcat version: 9.0.59 Are you able to upgrade these? Your Tomcat is nearly 2 years old and your Java is ... I'm not sure how old. This will have nothing to do with performance. Just security and general stability. > OS Version: Windows Server 2022 Datacenter Azure Edition > > Azure VM Type and Specs: D8s v3 > > 8 CPU 32 GiB Mem > > VM Generation V2 > > VM Architecture x64 > > Location East US Zone 1 > > > > We have a traditional client/server application from a third-party > vendor that has a couple second delay when accessing certain items > (tabs) in their Web UI. The delay has been traced down to the > webserver, which runs Tomcat. The application and database servers do > not appear to have any significant delays. > > When an item is clicked inside the web UI, a call from the web server > is made to the app and DB server, which come back fairly quickly. > It's at the point where data is delivered from the app server to the > web server that tomcat on the webserver CPU usage spikes. In > reviewing the web server with procmon running, we noticed that tomcat > is reading a lot of class files during the time we are > waiting for the task to complete to render the page. Approximately > 55,000 (yes 55000) classes read each click. Ouch. > The vendor has reviewed this and said it's nothing to worry about, but > we cannot find any other significant task that tomcat is doing during > this wait state that a web client experiences. Are you able to see the duration of that 55000 class file read? How have you observed that Tomcat is reading all those files? Once an application is started and humming-along, Tomcat shouldn't be reading too many class files at all. Reasons I can think of for reading lots of class files: 1. Auto-reload is enabled, and your class files have timestamps in the future. This would basically restart your application completely every few seconds. I think you'd notice. 2. You have disabled caching of your and, for some reason, you are constantly searching through them for something. 3. You have an application-level scan for classes which is not using Tomcat's (usually cached) loader to locate classes. In this case, I would say that your application should (a) be loading though Tomcat and (b) probably not scanning classes all the time :) > We have sql tracing showing microsecond response times. > > Developer console in chrome shows it waiting for 1.9 (Avg) seconds per click. > > > > We would like to get to the root cause of this slowness, whether it is the > operating system, Azure VM, webserver or vendor application causing the issue. Well, if your application is triggering a 55000-file scan with every request, that's obviously the place to start. That simply should not be happening. Even a fast VM, OS, web server, etc. wasting that much time won't be "fast". So start there. If you are able to observe the files being read, you can probably arrange to have one of them refuse to be read which should produce a FileNotFoundException or similar in a log. Then you can see what component is actually triggering that file-read and try to find out what it's being read with every request. I would make the vendor do this. It's what you pay them for. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Notice: This e-mail message is confidential and is intended only for the use of the individual and/or entity identified in the address line of this message. If you have received this message in error, or are not the named recipient(s), please notify us immediately by telephone (888-479-9111) M_LEGAL_NOTICE - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Speeding up tomcat
Alex, On 3/25/24 13:06, Alex Hatcher wrote: Java version: 11.0.20 Tomcat version: 9.0.59 Are you able to upgrade these? Your Tomcat is nearly 2 years old and your Java is ... I'm not sure how old. This will have nothing to do with performance. Just security and general stability. OS Version: Windows Server 2022 Datacenter Azure Edition Azure VM Type and Specs: D8s v3 8 CPU 32 GiB Mem VM Generation V2 VM Architecture x64 Location East US Zone 1 We have a traditional client/server application from a third-party vendor that has a couple second delay when accessing certain items (tabs) in their Web UI. The delay has been traced down to the webserver, which runs Tomcat. The application and database servers do not appear to have any significant delays. When an item is clicked inside the web UI, a call from the web server is made to the app and DB server, which come back fairly quickly. It's at the point where data is delivered from the app server to the web server that tomcat on the webserver CPU usage spikes. In reviewing the web server with procmon running, we noticed that tomcat is reading a lot of class files during the time we are waiting for the task to complete to render the page. Approximately 55,000 (yes 55000) classes read each click. Ouch. The vendor has reviewed this and said it's nothing to worry about, but we cannot find any other significant task that tomcat is doing during this wait state that a web client experiences. Are you able to see the duration of that 55000 class file read? How have you observed that Tomcat is reading all those files? Once an application is started and humming-along, Tomcat shouldn't be reading too many class files at all. Reasons I can think of for reading lots of class files: 1. Auto-reload is enabled, and your class files have timestamps in the future. This would basically restart your application completely every few seconds. I think you'd notice. 2. You have disabled caching of your and, for some reason, you are constantly searching through them for something. 3. You have an application-level scan for classes which is not using Tomcat's (usually cached) loader to locate classes. In this case, I would say that your application should (a) be loading though Tomcat and (b) probably not scanning classes all the time :) We have sql tracing showing microsecond response times. Developer console in chrome shows it waiting for 1.9 (Avg) seconds per click. We would like to get to the root cause of this slowness, whether it is the operating system, Azure VM, webserver or vendor application causing the issue. Well, if your application is triggering a 55000-file scan with every request, that's obviously the place to start. That simply should not be happening. Even a fast VM, OS, web server, etc. wasting that much time won't be "fast". So start there. If you are able to observe the files being read, you can probably arrange to have one of them refuse to be read which should produce a FileNotFoundException or similar in a log. Then you can see what component is actually triggering that file-read and try to find out what it's being read with every request. I would make the vendor do this. It's what you pay them for. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Speeding up tomcat
Alex > -Original Message- > From: Alex Hatcher > Sent: Monday, March 25, 2024 12:06 PM > To: users@tomcat.apache.org > Subject: Speeding up tomcat > > > > Java version: 11.0.20 > > Tomcat version: 9.0.59 > > > OS Version: Windows Server 2022 Datacenter Azure Edition > > Azure VM Type and Specs: D8s v3 > > 8 CPU 32 GiB Mem > > VM Generation V2 > > VM Architecture x64 > > Location East US Zone 1 > > > > We have a traditional client/server application from a third-party vendor that > has a couple second delay when accessing certain items (tabs) in their Web UI. > The delay has been traced down to the webserver, which runs Tomcat. The > application and database servers do not appear to have any significant delays. > > > > When an item is clicked inside the web UI, a call from the web server is made > to the app and DB server, which come back fairly quickly. It's at the point > where data is delivered from the app server to the web server that tomcat on > the webserver CPU usage spikes. In reviewing the web server with procmon > running, we noticed that tomcat is reading a lot of class files during the > time > we are waiting for the task to complete to render the page. Approximately > 55,000 (yes 55000) classes read each click. > > > > The vendor has reviewed this and said it's nothing to worry about, but we > cannot find any other significant task that tomcat is doing during this wait > state that a web client experiences. > > > > We have sql tracing showing microsecond response times. > > > > Developer console in chrome shows it waiting for 1.9 (Avg) seconds per click. > > > > We would like to get to the root cause of this slowness, whether it is the > operating system, Azure VM, webserver or vendor application causing the > issue. > > > > > > > Notice: This e-mail message is confidential and is intended only for the use > of > the individual and/or entity identified in the address line of this message. > If > you have received this message in error, or are not the named recipient(s), > please notify us immediately by telephone (888-479-9111) M_LEGAL_NOTICE For this level of detail, I would use a profiler like the java flight recorder, JProfiler, etc. Your description of classes being re-read sounds like the code is recreating one of the many factory classes that use the java service locator under the covers, like DocumentBuilderFactory, etc. If you can confirm that, I can tell you (or the vendor) what to do about it. Thanks - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
