Re: tomcat thread incurring CPU load
On 18/11/2019 14:14, Mark Thomas wrote: > On 18/11/2019 12:06, M. Manna wrote: >> Mark and others, >> >> On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: >> >>> OK, it looks like I can reproduce this. >>> >>> Steps to reproduce: >>> >>> - Windows 2016 Server fully patched >>> - Java 1.8.0u144 >>> - Install Tomcat 8.5.45 from windows installer >>> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 >>> - Modify server.xml to use Http11AprProtocol on port 8080 >>> - Make a single request >>> >>> I then see 1 core running at 100% until the connection times out after >>> 20s. Make another request and a core goes back up to 100% for 20s (the >>> default keep-alive time out). >>> >> >> I have also successfully reproduced this with making a single request >> (sorry for not replying in the weekend). Not sure how your graph looked >> like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as >> the request hit the server. and it didn't go down at all. > > I see similar behaviour on Windows 7 but the the CPU usage drops after ~5s. > > A binary search indicates that the issue was introduced with this commit: > > https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 > > (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) > > However, that code was removed when APR was switched to a single poll > set. Ah ha. It was removed in 9.0.x but not in 8.5.x (only 9.0.x switched to a single Poller) so it does look like this change is responsible. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Re: Tomcat 8.5.48: NullPointerException in ApplicationMapping.getHttpServletMapping()
On 11/17/2019 9:01 AM, Mark Thomas wrote: On 16/11/2019 22:08, Adam Rauch wrote: While testing 8.5.48, we now see NullPointerExceptions when our ImageServlet code attempts to forward a request to a new location. In 8.5.47, the code works fine. Thanks for reporting this. I can see what the problem is. I updated the servlet4preview API in 8.5.x to align it with what was released in Servlet 4 (some names changed). As part of that I aligned the 8.5.x implementation code with 9.0.x. That removed a null check that 8.5.x needs. I'll cancel the 8.5.48 vote, get that fixed and roll an 8.5.49 release. Mark FYI: I've tested on 8.5.49 and everything looks good to me. Thanks, once again, for the quick turnaround, Mark! Adam - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat thread incurring CPU load
On 18/11/2019 12:06, M. Manna wrote: > Mark and others, > > On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > >> OK, it looks like I can reproduce this. >> >> Steps to reproduce: >> >> - Windows 2016 Server fully patched >> - Java 1.8.0u144 >> - Install Tomcat 8.5.45 from windows installer >> - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 >> - Modify server.xml to use Http11AprProtocol on port 8080 >> - Make a single request >> >> I then see 1 core running at 100% until the connection times out after >> 20s. Make another request and a core goes back up to 100% for 20s (the >> default keep-alive time out). >> > > I have also successfully reproduced this with making a single request > (sorry for not replying in the weekend). Not sure how your graph looked > like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as > the request hit the server. and it didn't go down at all. I see similar behaviour on Windows 7 but the the CPU usage drops after ~5s. A binary search indicates that the issue was introduced with this commit: https://github.com/apache/tomcat/commit/fffb08790e642e03f00c5f96a3a61ee09a2c8342 (this is for 9.0.x - 8.5.x and 7.0.x had similar commits) However, that code was removed when APR was switched to a single poll set. Also, that the issue does not happens on all platforms (I can't repeat this on Linux) suggests the issue may not be in Java code. More research required... Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat 8.5.48: NullPointerException in ApplicationMapping.getHttpServletMapping()
-Original Message-
From: Mark Thomas
Sent: Monday, November 18, 2019 3:41 AM
To: users@tomcat.apache.org
Subject: Re: Tomcat 8.5.48: NullPointerException in
ApplicationMapping.getHttpServletMapping()
On 18/11/2019 04:36, jonmcalexan...@wellsfargo.com.INVALID wrote:
> Oh yippee. Another release. :-D
>I'm not sure how to take that. My impression is that you think another release
>is a bad thing. My expectation was that regular releases would be viewed as a
>good thing. I apologise if I have misinterpreted your comment but, if I
>haven't, could you explain why another release is a bad thing?
Don't take it the wrong way. It just means work on my side when these come out.
I was being sarcastic, but not trying to be mean sarcastic. :-D
> Do you have an ETA on 8.5.49 Mark?
> Voting started yesterday. Votes typically run for 72 hours. While we can use
> shorter / longer voting periods, I'm expecting this one to be the typical 72
> hours. Assuming the vote passes, I'll move everything to the release area. It
> then takes ~24 hours for the mirrors to pick up the release and we announce
> shortly afterwards.
> That said, the actual bits don't change. So you can get a copy of 8.5.49
> whenever you like and can consider it released as soon as the VOTE passes on
> the dev@ list. I'd expect that some time Wednesday evening / Thursday morning
> (UTC).
> Mark
Thanks Mark,
Dream * Excel * Explore * Inspire
Jon McAlexander
Asst Vice President
Middleware Product Engineering
Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions
Upcoming PTO: 11/8, 11/11, 11/15, 11/22, 11/28, 11/29, 12/2, 12/6, 12/13, 12/20
– 12/31
8080 Cobblestone Rd | Urbandale, IA 50322
MAC: F4469-010
Tel 515-988-2508 | Cell 515-988-2508
jonmcalexan...@wellsfargo.com
This message may contain confidential and/or privileged information. If you are
not the addressee or authorized to receive this for the addressee, you must not
use, copy, disclose, or take any action based on this message or any
information herein. If you have received this message in error, please advise
the sender immediately by reply e-mail and delete this message. Thank you for
your cooperation.
>
>
> Dream * Excel * Explore * Inspire
> Jon McAlexander
> Asst Vice President
>
> Middleware Product Engineering
> Enterprise CIO | Platform Services | Middleware | Infrastructure
> Solutions
>
> Upcoming PTO: 11/8, 11/11, 11/15, 11/22, 11/28, 11/29, 12/2, 12/6,
> 12/13, 12/20 – 12/31
>
> 8080 Cobblestone Rd | Urbandale, IA 50322
> MAC: F4469-010
> Tel 515-988-2508 | Cell 515-988-2508
>
> jonmcalexan...@wellsfargo.com
>
>
> This message may contain confidential and/or privileged information. If you
> are not the addressee or authorized to receive this for the addressee, you
> must not use, copy, disclose, or take any action based on this message or any
> information herein. If you have received this message in error, please advise
> the sender immediately by reply e-mail and delete this message. Thank you for
> your cooperation.
>
> -Original Message-
> From: Mark Thomas
> Sent: Sunday, November 17, 2019 11:01 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 8.5.48: NullPointerException in
> ApplicationMapping.getHttpServletMapping()
>
> On 16/11/2019 22:08, Adam Rauch wrote:
>> While testing 8.5.48, we now see NullPointerExceptions when our
>> ImageServlet code attempts to forward a request to a new location. In
>> 8.5.47, the code works fine.
>
> Thanks for reporting this. I can see what the problem is.
>
> I updated the servlet4preview API in 8.5.x to align it with what was
> released in Servlet 4 (some names changed). As part of that I aligned
> the 8.5.x implementation code with 9.0.x. That removed a null check
> that 8.5.x needs. I'll cancel the 8.5.48 vote, get that fixed and roll
> an
> 8.5.49 release.
>
> Mark
>
>
>>
>> A simplified version of our code:
>>
>> public void sendResource(HttpServletRequest request,
>> HttpServletResponse response) throws IOException, ServletException
>> {
>> request.getRequestDispatcher(getForwardLocation(request)).forward(req
>> u
>> est,
>> response);
>> }
>>
>> The NPE stack trace we see:
>>
>> java.lang.NullPointerException
>> at
>> org.apache.catalina.core.ApplicationMapping.getHttpServletMapping(App
>> l
>> icationMapping.java:36)
>>
>> at
>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationD
>> i
>> spatcher.java:380)
>>
>> at
>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDis
>> p
>> atcher.java:316)
>>
>> at
>> org.labkey.api.settings.TemplateResourceHandler.sendResource(Template
>> R
>> esourceHandler.java:202)
>>
>> at
>> org.labkey.api.attachments.ImageServlet.service(ImageServlet.java:72)
>> at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>> at
>>
Re: tomcat thread incurring CPU load
Mark and others, On Mon, 18 Nov 2019 at 12:01, Mark Thomas wrote: > OK, it looks like I can reproduce this. > > Steps to reproduce: > > - Windows 2016 Server fully patched > - Java 1.8.0u144 > - Install Tomcat 8.5.45 from windows installer > - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 > - Modify server.xml to use Http11AprProtocol on port 8080 > - Make a single request > > I then see 1 core running at 100% until the connection times out after > 20s. Make another request and a core goes back up to 100% for 20s (the > default keep-alive time out). > I have also successfully reproduced this with making a single request (sorry for not replying in the weekend). Not sure how your graph looked like, but the Jvisualvm showed me a Sinusoidal modulation curve as soon as the request hit the server. and it didn't go down at all. Thanks, > > Next steps are to try and track down the root cause. > > Mark > > > > > Mark and M, > > > > On 11/13/19 19:31, Mark Thomas wrote: > >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" > >> wrote: > >>> I see this update on Windows which may have been responsible > >>> (suspicion only, haven’t rolled it back yet) > >>> > >>> > >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > > ocode-updates > >>> > >>> > >>> > > Was 8.5.45 built on Windows 10 in presence of this update ? > > > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully > >> patched at the time of the build Windows 7 64-bit VM. > > Also it doesn't matter because binaries don't include CPU microcode. > > > > It's more likely that the target system has microcode updates such as > > these that may negatively impact performance. > > > > -chris > > > >>> > >>> Thanks, > >>> > >>> On Wed, 13 Nov 2019 at 17:55, M. Manna > >>> wrote: > >>> > Hi Chris, > > On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > >> On 11/13/19 11:20, M. Manna wrote: > >>> HI Mark, > >>> > >>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas > >>> wrote: > >>> > On 12/11/2019 19:11, M. Manna wrote: > > HI Mark, > > > > following my previous reply, we have now confirmed > > that it's indeed > 8.5.45 > > with APR 1.2.23 that's causing such high JVM CPU > > usage. We used took out 2 out of 50 servers from the > > load balancer config, reverted tomcat, and > > redeployed. With near to identical user traffic, the > > two servers are responding normally without/without > > traffic with 8.5.41. The JVM dump looks a lot better > > with 8.5.41. > > > > We do think that the recent changes in APR and some > > other tomcat jar may have caused compatibility issue > > on Windows server 2016 (64-bit) platform. But > > unfortunately, we cannot pinpoint exactly what change > > may have caused this (i.e. actual OS vs Security > > Updates). With this in mind, we are also being wary > > to move to 8.5.47 as we don't know if the same issue > > will > occur > > again. Since 8.5.41 has been packaged with previously > > accepted > application > > installer, we are more comfortable rolling back. > > Just to confirm, you see this high CPU usage with a > clean install (no additional web applications deployed, > no configuration changes) on Windows 2016 DataCenter > (64-bit)? > > If this is the case, it should be fairly easy to > reproduce. > > Mark > > We do not deploy multiple applications. In fact, Under > tomcat > >>> webapps/ROOT we only have one application (ours). Each > >>> tomcat instance is hosted on a VM (total 50) and all of > >>> them are identically configured (server.xml, web.xml, > >>> logging, CPU/RAM). We have not made any other > >>> configuration change between 8.5.41 and 8.5.45. And yes, > >>> I agree with you that it's fairly easy to reproduce. > > > >> I think the question is whether or not your application is > >> required > to > >> be deployed. Can you reproduce this issue with just the stock > >> applications bundled with Tomcat? > > > > > > My apologies, but our application needs to be deployed. We > > have not > (or > > didn't try in the past) to simply deploy tomcat with stock > application (in > > other words, simply starting the tomcat OOB) on our prod > > servers. This is the first time it has hit us with such > > disparity. I’ll try to investigate and get a stock > > application data. But we may not be able > to do > > that quite easily as it’s in our production. > > > > What I can see is that 3 Windows updates may have been > > responsible > for > > this, but we aren’t sure
Re: tomcat thread incurring CPU load
OK, it looks like I can reproduce this. Steps to reproduce: - Windows 2016 Server fully patched - Java 1.8.0u144 - Install Tomcat 8.5.45 from windows installer - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 - Modify server.xml to use Http11AprProtocol on port 8080 - Make a single request I then see 1 core running at 100% until the connection times out after 20s. Make another request and a core goes back up to 100% for 20s (the default keep-alive time out). Next steps are to try and track down the root cause. Mark > Mark and M, > > On 11/13/19 19:31, Mark Thomas wrote: >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" >> wrote: >>> I see this update on Windows which may have been responsible >>> (suspicion only, haven’t rolled it back yet) >>> >>> >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > ocode-updates >>> >>> >>> > Was 8.5.45 built on Windows 10 in presence of this update ? > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully >> patched at the time of the build Windows 7 64-bit VM. > Also it doesn't matter because binaries don't include CPU microcode. > > It's more likely that the target system has microcode updates such as > these that may negatively impact performance. > > -chris > >>> >>> Thanks, >>> >>> On Wed, 13 Nov 2019 at 17:55, M. Manna >>> wrote: >>> Hi Chris, On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < ch...@christopherschultz.net> wrote: >> On 11/13/19 11:20, M. Manna wrote: >>> HI Mark, >>> >>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas >>> wrote: >>> On 12/11/2019 19:11, M. Manna wrote: > HI Mark, > > following my previous reply, we have now confirmed > that it's indeed 8.5.45 > with APR 1.2.23 that's causing such high JVM CPU > usage. We used took out 2 out of 50 servers from the > load balancer config, reverted tomcat, and > redeployed. With near to identical user traffic, the > two servers are responding normally without/without > traffic with 8.5.41. The JVM dump looks a lot better > with 8.5.41. > > We do think that the recent changes in APR and some > other tomcat jar may have caused compatibility issue > on Windows server 2016 (64-bit) platform. But > unfortunately, we cannot pinpoint exactly what change > may have caused this (i.e. actual OS vs Security > Updates). With this in mind, we are also being wary > to move to 8.5.47 as we don't know if the same issue > will occur > again. Since 8.5.41 has been packaged with previously > accepted application > installer, we are more comfortable rolling back. Just to confirm, you see this high CPU usage with a clean install (no additional web applications deployed, no configuration changes) on Windows 2016 DataCenter (64-bit)? If this is the case, it should be fairly easy to reproduce. Mark We do not deploy multiple applications. In fact, Under tomcat >>> webapps/ROOT we only have one application (ours). Each >>> tomcat instance is hosted on a VM (total 50) and all of >>> them are identically configured (server.xml, web.xml, >>> logging, CPU/RAM). We have not made any other >>> configuration change between 8.5.41 and 8.5.45. And yes, >>> I agree with you that it's fairly easy to reproduce. > >> I think the question is whether or not your application is >> required to >> be deployed. Can you reproduce this issue with just the stock >> applications bundled with Tomcat? > > > My apologies, but our application needs to be deployed. We > have not (or > didn't try in the past) to simply deploy tomcat with stock application (in > other words, simply starting the tomcat OOB) on our prod > servers. This is the first time it has hit us with such > disparity. I’ll try to investigate and get a stock > application data. But we may not be able to do > that quite easily as it’s in our production. > > What I can see is that 3 Windows updates may have been > responsible for > this, but we aren’t sure about that. I’ll let you know if we > can get anything with the stock application instance. > > Thanks, > > - -chris > > > >>> - > >>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: > users-h...@tomcat.apache.org > > > > >> - > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > > >
Re: Support for JDK only by Windows Installer?
On 16/11/2019 09:29, Alexander Norz wrote: > Am 15.11.2019 22:23, schrieb Mark Thomas: > > That is the point: > >> Because Oracle changed the directory layout and names of the standard >> registry entries and the installer probably hasn't been updated to take >> account of that yet. > > The environment variable JAVA_HOME isn't supported actually. > > I added this as the last check after JRE64/32, JRE in JDK, now new JDK > only and then environment variable. > > We can discus to use environment variable first of all? I think this > could be more common? Thanks for the PR. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.5.48: NullPointerException in ApplicationMapping.getHttpServletMapping()
On 18/11/2019 04:36, jonmcalexan...@wellsfargo.com.INVALID wrote:
> Oh yippee. Another release. :-D
I'm not sure how to take that. My impression is that you think another
release is a bad thing. My expectation was that regular releases would
be viewed as a good thing. I apologise if I have misinterpreted your
comment but, if I haven't, could you explain why another release is a
bad thing?
> Do you have an ETA on 8.5.49 Mark?
Voting started yesterday. Votes typically run for 72 hours. While we can
use shorter / longer voting periods, I'm expecting this one to be the
typical 72 hours. Assuming the vote passes, I'll move everything to the
release area. It then takes ~24 hours for the mirrors to pick up the
release and we announce shortly afterwards.
That said, the actual bits don't change. So you can get a copy of 8.5.49
whenever you like and can consider it released as soon as the VOTE
passes on the dev@ list. I'd expect that some time Wednesday evening /
Thursday morning (UTC).
Mark
>
>
> Dream * Excel * Explore * Inspire
> Jon McAlexander
> Asst Vice President
>
> Middleware Product Engineering
> Enterprise CIO | Platform Services | Middleware | Infrastructure Solutions
>
> Upcoming PTO: 11/8, 11/11, 11/15, 11/22, 11/28, 11/29, 12/2, 12/6, 12/13,
> 12/20 – 12/31
>
> 8080 Cobblestone Rd | Urbandale, IA 50322
> MAC: F4469-010
> Tel 515-988-2508 | Cell 515-988-2508
>
> jonmcalexan...@wellsfargo.com
>
>
> This message may contain confidential and/or privileged information. If you
> are not the addressee or authorized to receive this for the addressee, you
> must not use, copy, disclose, or take any action based on this message or any
> information herein. If you have received this message in error, please advise
> the sender immediately by reply e-mail and delete this message. Thank you for
> your cooperation.
>
> -Original Message-
> From: Mark Thomas
> Sent: Sunday, November 17, 2019 11:01 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 8.5.48: NullPointerException in
> ApplicationMapping.getHttpServletMapping()
>
> On 16/11/2019 22:08, Adam Rauch wrote:
>> While testing 8.5.48, we now see NullPointerExceptions when our
>> ImageServlet code attempts to forward a request to a new location. In
>> 8.5.47, the code works fine.
>
> Thanks for reporting this. I can see what the problem is.
>
> I updated the servlet4preview API in 8.5.x to align it with what was released
> in Servlet 4 (some names changed). As part of that I aligned the 8.5.x
> implementation code with 9.0.x. That removed a null check that 8.5.x needs.
> I'll cancel the 8.5.48 vote, get that fixed and roll an
> 8.5.49 release.
>
> Mark
>
>
>>
>> A simplified version of our code:
>>
>> public void sendResource(HttpServletRequest request,
>> HttpServletResponse response) throws IOException, ServletException
>> {
>> request.getRequestDispatcher(getForwardLocation(request)).forward(requ
>> est,
>> response);
>> }
>>
>> The NPE stack trace we see:
>>
>> java.lang.NullPointerException
>> at
>> org.apache.catalina.core.ApplicationMapping.getHttpServletMapping(Appl
>> icationMapping.java:36)
>>
>> at
>> org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDi
>> spatcher.java:380)
>>
>> at
>> org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDisp
>> atcher.java:316)
>>
>> at
>> org.labkey.api.settings.TemplateResourceHandler.sendResource(TemplateR
>> esourceHandler.java:202)
>>
>> at
>> org.labkey.api.attachments.ImageServlet.service(ImageServlet.java:72)
>> at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>> at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
>> cationFilterChain.java:231)
>>
>> at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
>> lterChain.java:166)
>>
>> at
>> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
>>
>> Let me know if you'd like me to open a Bugzilla and/or provide more
>> context.
>>
>> Thanks,
>> Adam
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
