RE: FW: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi,
Christopher, Please find the answer in-line:
How... exactly?
private String getRequestBody(HttpServletRequest request) throws IOException
{
StringBuilder sb = new StringBuilder();
BufferedReader reader = request.getReader();
try
{
String line;
while( ( line = reader.readLine() ) != null )
{
sb.append( line ).append( '\n' );
}
}
finally
{
reader.close();
}
return sb.toString();
}
I am trying to reproduce the StackOverflowError with a sample application ,
Once it is reproduced I will share it across.
Thanks and Regards
Arshiya Shariff
-Original Message-
From: Christopher Schultz
Sent: Thursday, October 15, 2020 12:01 AM
To: users@tomcat.apache.org
Subject: Re: FW: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
Arshiya,
On 10/14/20 01:23, Arshiya Shariff wrote:
> Please find the answers in-line Mark.
>
> Http2 requests with message payload of 34KB are pumped from JMeter at
> 20 TPS with 700 connections to an application with Embedded tomcat
> - 9.0.39 (max-Threads : 200, all other values are the tomcat
> defaults)
>
>> What does that URL do with the POSTed content? Ignore it? Read it
>> from an InputStream? Read it via getParameter()?
>
> The posted content is read via BufferedReader reader =
> request.getReader() and processed asynchronously
How... exactly?
> Is JMeter run on the same machine as Tomcat?
> JMeter is run from a different machine.
>
> Do you use the JMeter GUI or the command line?
> Launched via Command line (JMeter heap increased to 10 GB )
>
> What are the specs of the server(s) being used?
> The server is a VM with 12 CPUs and 120 GB RAM
>
> Please let us know if you require more details.
This would probabyl be easier if you'd just provide a test-case: a sample
(simple!) web application which reproduces what you are reporting.
-chris
> -Original Message-
> From: Mark Thomas
> Sent: Monday, October 12, 2020 7:28 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the
> connections to 1000/2000 (Embedded tomcat 9.0.38)
>
> On 12/10/2020 08:02, Arshiya Shariff wrote:
>> Hi Mark ,
>>
>> The issue is reproduced with version 9.0.39 as well. Max threads in Tomcat
>> is 200.
>>
>> Please find the case:
>> Client:JMeter 5.2.1 (With http2 plugin)
>> TPS: around 20
>> No of users from JMeter : 700
>> Message payload size: 6 KB to 34 KB
>> Loop: Infinite
>> We let the loop run infinitely and see the java.lang.StackOverflowError
>> trace printed multiple times in the log within few minutes of starting the
>> test.
>
> POSTing to what URL?
>
> What does that URL do with the POSTed content? Ignore it? Read it from an
> InputStream? Read it via getParameter()?
>
> Is JMeter run on the same machine as Tomcat?
>
> Do you use the JMeter GUI or the command line?
>
> What are the specs of the server(s) being used?
>
> You need to provide the exact steps to recreate this issue on a clean install
> of Tomcat 9.0.39 as provided by the ASF.
>
> Mark
>
>
>> Please help us with this . What is the impact of StackOverflowError ?
>>
>> Thanks and Regards
>> Arshiya Shariff
>>
>> -Original Message-
>> From: Mark Thomas
>> Sent: Friday, October 9, 2020 5:31 PM
>> To: users@tomcat.apache.org
>> Subject: Re: HTTP2: memory filled up fast on increasing the
>> connections to 1000/2000 (Embedded tomcat 9.0.38)
>>
>> On 09/10/2020 12:32, Arshiya Shariff wrote:
>>> Hi,
>>>
>>> Mark , with the test runs that I performed over clean 9.0.x branch I was
>>> not able to reproduce this.
>>
>> Good. But I'd really like to understand why...
>>
>>> But with 9.0.38 and the jars built from 9.0.x with hash:
>>> c8ec2d4cde3a31b0e9df9a30e7915d77ba725545 , with 700 or 1000 users
>>> (connections) and on sending 1000 Requests per second (or even lesser) ,
>>> payload of 16K from JMeter I can see that this Exception occurs within few
>>> minutes of starting the test . The maxThreads configured in tomcat is 200 .
>>>
>>> How often do you see these errors in your test run?
>>> Randomly, at times 2 or 3 such traces.
>>
>> OK. Definitely a timing issue then.
>>
>>> Do you have the other end of that stack trace?
>>> It is only the two lines that is recursively printed till the end about
>>> ~500 times in one trace :
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
>>> at
>>> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHan
>>> d
>>> l
>>> er.completed(SocketWrapperBase.java:1100)
>>
>> Doesn't tell me much unfortunately.
>>
>>> I see the trace starting with :
>>>
Re: Recent Tomcat crash produced error messages I've never seen before
James, On 10/20/20 16:39, James H. H. Lampert wrote: > On 10/20/20 1:26 PM, Christopher Schultz wrote: >> Theoretically, it should not be possible to cause a JVM to crash with >> pure Java code. > > Thanks. > > Of course, we all know that while theory and practice are the same in > theory, they aren't always in practice. ;-P Exactly. But my point is that if your Java code is causing the JVM to crash, it's not your Java code's fault. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Recent Tomcat crash produced error messages I've never seen before
On 10/20/20 1:26 PM, Christopher Schultz wrote: Theoretically, it should not be possible to cause a JVM to crash with pure Java code. Thanks. Of course, we all know that while theory and practice are the same in theory, they aren't always in practice. ;-P -- JHHL - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Recent Tomcat crash produced error messages I've never seen before
James, On 10/20/20 13:35, James H. H. Lampert wrote: > We had a Tomcat crash on a customer box, a few hours ago (a simple > restart got them back up and running), and it produced a whole bunch of > errors in the general vein of >> >> *** Invalid JIT return address 0006E2E2E400 in 0001A83C5210 Yikes. > before finally failing with a null pointer exception, and producing a > Java dump, a Snap dump, and a JIT dump, all over the course of a couple > of seconds. > > Prior to that, our webapp was having a great deal of trouble accessing > Office365. Via HTTP(S) or via some native API? > Other than that it's almost certainly our webapp, rather than Tomcat > itself, that caused the "Invalid JIT return address" messages, does > anybody here have any insights? I've literally never seen messages of > this type before. It's more likely to be a JVM bug than anything in your code. Theoretically, it should not be possible to cause a JVM to crash with pure Java code. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
jstl jar location
I am hoping someone can shed some lights on a question. I did try to search online and SO but haven't had luck in figure it out so hopefully it is a quick answer from the people that know that stuff. We have an uber-lib folder where we keep shared libraries in our TC85-hosted app. If we put jstl-1.2.jar into that directory but not in the application /WEB-INF/lib directory, TC generates [1]. If I move jstl into the application lib folder, it works. I made sure jstl is excluded from jarsToSkip and included in jarsToScan. Is there any rule or switch that says that the JSP compiler cannot use the parent CL to resolve the jstl URIs? George [1] Type Exception Report Message The absolute uri: [http://java.sun.com/jsp/jstl/core] cannot be resolved in either web.xml or the jar files deployed with this application Description The server encountered an unexpected condition that prevented it from fulfilling the request. Exception org.apache.jasper.JasperException: The absolute uri: [http://java.sun.com/jsp/jstl/core] cannot be resolved in either web.xml or the jar files deployed with this application org.apache.jasper.compiler.DefaultErrorHandler.jspError(DefaultErrorHandler.java:55) org.apache.jasper.compiler.ErrorDispatcher.dispatch(ErrorDispatcher.java:293) org.apache.jasper.compiler.ErrorDispatcher.jspError(ErrorDispatcher.java:80) org.apache.jasper.compiler.TagLibraryInfoImpl.generateTldResourcePath(TagLibraryInfoImpl.java:251) org.apache.jasper.compiler.TagLibraryInfoImpl.(TagLibraryInfoImpl.java:122) org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:431) org.apache.jasper.compiler.Parser.parseDirective(Parser.java:489) org.apache.jasper.compiler.Parser.parseElements(Parser.java:1445) org.apache.jasper.compiler.Parser.parse(Parser.java:144) org.apache.jasper.compiler.ParserController.doParse(ParserController.java:244) org.apache.jasper.compiler.ParserController.parse(ParserController.java:105) org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:203) org.apache.jasper.compiler.Compiler.compile(Compiler.java:375) org.apache.jasper.compiler.Compiler.compile(Compiler.java:351) org.apache.jasper.compiler.Compiler.compile(Compiler.java:335) org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:597) org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399) org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:386) org.apache.jasper.servlet.JspServlet.service(JspServlet.java:330) javax.servlet.http.HttpServlet.service(HttpServlet.java:733) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168) org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:304) org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1286) org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1041) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:984) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
Recent Tomcat crash produced error messages I've never seen before
We had a Tomcat crash on a customer box, a few hours ago (a simple restart got them back up and running), and it produced a whole bunch of errors in the general vein of *** Invalid JIT return address 0006E2E2E400 in 0001A83C5210 before finally failing with a null pointer exception, and producing a Java dump, a Snap dump, and a JIT dump, all over the course of a couple of seconds. Prior to that, our webapp was having a great deal of trouble accessing Office365. Other than that it's almost certainly our webapp, rather than Tomcat itself, that caused the "Invalid JIT return address" messages, does anybody here have any insights? I've literally never seen messages of this type before. -- JHHL - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [EXTERNAL] Re: Embedded vs Standalone Tomcat
Thank you so much Igal for the inputs. Thanks, Amit Thanks, Amit From: Igal Sapir Sent: Sunday, October 18, 2020 11:36:22 AM To: Tomcat Users List Subject: [EXTERNAL] Re: Embedded vs Standalone Tomcat Amit, On Fri, Oct 16, 2020 at 8:32 AM Amit Pande wrote: > My apologies in advance if this has been already discussed in the group. I > am looking for experiences of the community, any nitpicks, etc. > > Currently we are using standalone Tomcat version (9.x) to host a web > applications which are essentially hosting REST APIs. > We plan to move to micro services model where each web application > (logical unit of related APIs) will be hosted as a different process. > > With this mind, I wanted to evaluate the embedded version of Tomcat. > The code for both is the same. The part that we refer to as "Embedded" Tomcat contains most of the implementation. The "Standalone" simply adds a wrapper around the "Embedded" code that allows you to easily run Tomcat using configuration files, startup scripts, service installers, etc. > 1. Is there any guidance around when to use what? > If you have a standard deployment then use "Standalone". If you deploy a different application and Tomcat is used as a component in it, you can consider adding the wrapper and launch code yourself, using the "Embedded" version. If you plan to write a wrapper for "Embedded" that will do exactly what the "Standalone" version does then you will likely spend much resources for no reason, possibly introducing bugs during the process. > 2. Are there any feature disparities between standalone Tomcat and > embedded version? We have fairly extended Tomcat classes for use cases like > automatic certificate renewal, not having to mention is plain-text password > in server.xml for key stores, hooking in custom security provider (for > FIPS) for Tomcat JVM. etc. So we ideally wouldn't want to lose any of such > capabilities when we move to the embedded version. > Same code, I don't see any reason to believe that something would not work as long as the wrapper code that you add to launch the "Embedded" code knows how to utilize those. > 3. Currently, we have written a Windows SCM service wrapper and manage > our Tomcat JVM via that (on Windows). Would there be any issue with > embedded Tomcat if we want to have SCM service wrapper? > Same as above. > 4. How is the upgrade process different for embedded version? > Currently, we ship Tomcat with our product and have a well-defined process > to upgrade Tomcat even with older versions of our product deployed in > customers' environment. > Very similar process - you will have to update the Tomcat JARs and restart the service. As always you will have to check and test that no breaking changes were made in the newer version, though those are rather rare IMO. > 5. Externalizing the configuration (like we can do that via web.xml, > server.xml, context.xml) - any issues, limitations here? > You can set the "Embedded" Tomcat to use the web.xml. If you want to use server.xml and context.xml then you should really revisit question/answer 1) above. > 6. Any differences with resource usages (e.g. heap)? > Nope. It's still a JVM. If you use the same JVM args for both then you should expect similar results. HTH, Igal > > Any help is greatly appreciated. > > Thanks, > Amit > >
Re: Virtual event focussed on Tomcat Security
Mark, On 10/15/20 14:01, Mark Thomas wrote: > On 29/09/2020 12:25, Mark Thomas wrote: >> Hi all, >> >> We (the Tomcat community) have some funding from Google to help us >> improve Tomcat security. Our original plan was to use the funding to >> support an in-person security focussed hackathon. As you would expect, >> those plans are on hold for now. We would, therefore, like to explore >> the possibility of doing something virtually. >> >> The purpose of this email is to gather input from the community about >> what such an event should look like. With that input we can put together >> a plan for the event. So, over to you. What would your ideal virtual >> event focussed on Tomcat Security look like? > > Summarising the suggestions so far: > - application security / OWASP > - making HTTP requests *from* Tomcat > - SSO / SAML / OpenIDConnect > > The first two are more application security focused and would not have > to be Tomcat specific. > > The third is more likely to Tomcat specific depending on the extent to > which the SSO mechanism ties into Tomcat's internals. I've built incoming single-legged SAML SSO into my own application without any external libraries, so I could led a group to work on this kind of thing. > All the suggestions so far have been for conference like presentations > (if I am reading them correctly). > > Other possibilities: > - hackathon to implement (with support from committers) new security > features (no idea what these might be - suggestions welcome) > > - hackathon to run $tool_of_choice against Tomcat code base, review the > results and fix (with committer support) those that need fixing. > Suggestions as to tools to use welcome* > > Anything else you'd like to suggest that is related to Tomcat and security. > > There hasn't been any thought given to timing yet. > > Mark > > > > * I'll note that over the years most if not all of the major static > analysis tools have been run against the Tomcat code base and the > results have been very heavy on the false positives. Most of the work is > likely to be separating the few useful results from a lot of noise. +1 It's worth running new tools against Tomcat and then having many eyes look at the list to determine false-positives. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: No thread name in AccessLogValve printed
Am 2020-10-20 um 11:44 schrieb Mark Thomas:
On 20/10/2020 10:10, Michael Osipov wrote:
Folks,
I have seen recently entried like this on our access logs:
2020-10-19T20:00:05.591 [null] xyz - "-" 400 - 0
2020-10-19T20:00:05.591 [null] abc- "-" 400 - 0
2020-10-19T20:00:05.592 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.593 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.616 [null] abc - "-" 400 - 0
with pattern:
%{-MM-dd'T'HH:mm:ss.SSS}t [%I] %h %u %r %s %b %D
While I am quite certain that these ary "security" scans at work
I wonder why RequestInfo#getWorkerThreadName() is null.
Is the request rejected because it is malformed because before it is
handled to a worker?
Have you tried looking at the source code? Callers of
RequestInfo.setWorkerThreadName() are likely to be enlightening.
Yes, I did. It happens before the thread local is used. It happens in
org.apache.coyote.http11.Http11Processor.prepareRequest() before a
servlet is invoked. So the worker name has to be null here.
I am not sure whether "info != null &&
info.getgetWorkerThreadName() != null" would be the right change here.
That would address the symptom. There may be a better fix that addresses
(or at least gets closer) to the cause.
Given the information above I am inclined to say that Thread#getName()
would make sense here. Writing '-' does not make sense becuase every
request is processed in some thread, isn't it?
Michael
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: No thread name in AccessLogValve printed
On 20/10/2020 10:10, Michael Osipov wrote:
> Folks,
>
> I have seen recently entried like this on our access logs:
>> 2020-10-19T20:00:05.591 [null] xyz - "-" 400 - 0
>> 2020-10-19T20:00:05.591 [null] abc- "-" 400 - 0
>> 2020-10-19T20:00:05.592 [null] abc - "-" 400 - 0
>> 2020-10-19T20:00:05.593 [null] abc - "-" 400 - 0
>> 2020-10-19T20:00:05.616 [null] abc - "-" 400 - 0
>
> with pattern:
>> %{-MM-dd'T'HH:mm:ss.SSS}t [%I] %h %u %r %s %b %D
>
> While I am quite certain that these ary "security" scans at work
> I wonder why RequestInfo#getWorkerThreadName() is null.
> Is the request rejected because it is malformed because before it is
> handled to a worker?
Have you tried looking at the source code? Callers of
RequestInfo.setWorkerThreadName() are likely to be enlightening.
> I am not sure whether "info != null &&
> info.getgetWorkerThreadName() != null" would be the right change here.
That would address the symptom. There may be a better fix that addresses
(or at least gets closer) to the cause.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
No thread name in AccessLogValve printed
Folks,
I have seen recently entried like this on our access logs:
2020-10-19T20:00:05.591 [null] xyz - "-" 400 - 0
2020-10-19T20:00:05.591 [null] abc- "-" 400 - 0
2020-10-19T20:00:05.592 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.593 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.616 [null] abc - "-" 400 - 0
with pattern:
%{-MM-dd'T'HH:mm:ss.SSS}t [%I] %h %u %r %s %b %D
While I am quite certain that these ary "security" scans at work
I wonder why RequestInfo#getWorkerThreadName() is null.
Is the request rejected because it is malformed because before it is
handled to a worker? I am not sure whether "info != null &&
info.getgetWorkerThreadName() != null" would be the right change here.
Does someone know better?
Michael
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
