Re: using custom classes in Tomcat configuration files
Hi everyone. I thought I would circle back and answer my own question from back in June 2020 (see message thread below). The thing that creates objects from the server.xml configuration file is the Digester (https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/tomcat/util/digester/Digester.html), and it uses the processing rules described in the digester package summary (https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/tomcat/util/digester/package-summary.html#doc.Rules). Of particular note are the ObjectCreateRule (https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/tomcat/util/digester/ObjectCreateRule.html) and the SetPropertiesRule (https://tomcat.apache.org/tomcat-9.0-doc/api/org/apache/tomcat/util/digester/SetPropertiesRule.html). You can read more about my upgrading my Guise Mummy components to be able to be used in Tomcat configuration files, as well as see related code changes, at https://globalmentor.atlassian.net/browse/GUISE-150 . Now that I have all this working, I won't be coding on Tomcat as often, so I'm going to drop off this list for a while to cut down on my inbox influx. Thanks for the opportunity to participate in ApacheCon last year, and I hope the virtual event this year is a success. I enjoyed meeting those of you I met in person, and if things go well maybe I'll meet some of you again at an in-face conference in the future. All the best and stay safe. Garret On 6/8/2020 6:59 AM, Garret Wilson wrote: Hi, all. I'm getting close to releasing a new version of my Guise Mummy static site generation software. I gave a presentation over Guise Mummy's use of embedded Tomcat to serve the generated site locally for testing before deployment. It uses a custom site root and web resource https://guise.io/mummy/present/clean-urls Although in my use case I instantiation and connect these things programmatically, someone in attendance (I think it was Remy, or maybe Chris) mentioned that I could also reference the customized versions in one of the configuration files and Tomcat would create the instance graph automatically. I want to make the library flexible, so in this version I'd like to make sure that's possible. Could you remind me on the prerequisites for this, and what documentation I could look at? Does it use JavaBean conventions or some other conventions? (I assume I just need a no-argument constructor and some getters/setters, but I want to verify.) And in which Tomcat configuration would they go? Thanks, Garret - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Exit code 6 on shutting down Tomcat service
Hi. On 30.09.2020 14:40, Jakub Moravec wrote: Hello Tomcat team, we are having an issue that we were not able to resolve ourselves or using the existing documentation, so I'd like to ask you for help. Description: During Tomcat service shutdown (using command /bin/tomcat9.exe //SS//), we sometimes receive exit code 6. You may want to have a look at these : - https://cwiki.apache.org/confluence/display/TOMCAT/Windows#Windows-Q11 - http://tomcat.apache.org/tomcat-9.0-doc/windows-service-howto.html These pages do not explain why you get an exit code 6 from tomcat9.exe. But they explain what tomcat9.exe actually is, which may help for what follows (*) The documentation page http://tomcat.apache.org/tomcat-9.0-doc/windows-service-howto.html, at the end, lists some additional tomcat9.exe command-line parameters (the ones starting with "--Log") which may enable you to find out more details about the internal error that triggers this exit code. (e.g : --LogLevel "Debug")(and where to find that logfile) The mailing list archives, at https://markmail.org/list/org.apache.commons.users/ may also help finding the reason (in the search box, enter "daemon", or "daemon exit") (*) the tomcat9.exe program is actually a renamed copy of the Apache Commons Daemon "prunsrv" program, which the tomcat team adds to the tomcat-for-Windows package, to facilitate installing and running tomcat as a Windows service. It happens underministically (or at least we don't know the exact circumstances under which the error code is returned). We were not able to find any information about this exit code or any suggestions for fixes in the documentation. Environment: Tomcat: 9.0.33 OS: Windows 2016, Version: 10.0, Flavor: Data Center Thank you for your assistance! Jakub Moravec jakub.mora...@getmanta.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas wrote:
> On 30/09/2020 16:17, Mark Thomas wrote:
> > On 30/09/2020 13:53, Martin Grigorov wrote:
> >> On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov
> >
> >
> >
> >
> >> When I load test HTTP2 with POST (with big bodies) there are many errors
> >> like:
> >>
> >> 1)
> >> Exception in thread "https-jsse-nio-8080-exec-5"
> >> java.nio.BufferOverflowException
> >> at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
> >> at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
> >> at
> >> org.apache.tomcat.util.net
> .SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
> >> at
> >> org.apache.tomcat.util.net
> .SocketWrapperBase.unRead(SocketWrapperBase.java:401)
> >> at
> >>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
> >> at
> >>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
> >> at
> >> org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
> >> at
> >> org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> >> at
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> >> at
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> >> at
> >>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >> at java.base/java.lang.Thread.run(Thread.java:832)
> >>
> >> 2)
> >> Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller
> events
> >> SEVERE: Failed to register socket with selector from poller
> >> java.nio.channels.ClosedChannelException
> >> at
> >>
> java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
> >> at
> >> org.apache.tomcat.util.net
> .NioEndpoint$Poller.events(NioEndpoint.java:609)
> >> at org.apache.tomcat.util.net
> .NioEndpoint$Poller.run(NioEndpoint.java:703)
> >> at java.base/java.lang.Thread.run(Thread.java:832)
> >
> > That is helpful. Looks like you have found a way to reproduce the buffer
> > issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>
> Can you share the command you used to trigger those errors please.
>
The Vegeta command I used is:
jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext";,
"body":"payload=Some
sdgggwwsdgssfshffheeepayload"
| @base64, header: {"Content-Type":
["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
-http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta encode >
/tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
The app is at
https://github.com/martin-g/http2-server-perf-tests/tree/master/java/tomcat.
Just start EmbeddedTomcat#main() with -Dtomcat.http2=true
Martin
>
> Thanks,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 16:17, Mark Thomas wrote: > On 30/09/2020 13:53, Martin Grigorov wrote: >> On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov > > > > >> When I load test HTTP2 with POST (with big bodies) there are many errors >> like: >> >> 1) >> Exception in thread "https-jsse-nio-8080-exec-5" >> java.nio.BufferOverflowException >> at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957) >> at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247) >> at >> org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100) >> at >> org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401) >> at >> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307) >> at >> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164) >> at >> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087) >> at >> org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) >> at >> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) >> at java.base/java.lang.Thread.run(Thread.java:832) >> >> 2) >> Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events >> SEVERE: Failed to register socket with selector from poller >> java.nio.channels.ClosedChannelException >> at >> java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222) >> at >> org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609) >> at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703) >> at java.base/java.lang.Thread.run(Thread.java:832) > > That is helpful. Looks like you have found a way to reproduce the buffer > issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710 Can you share the command you used to trigger those errors please. Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 09:32, Arshiya Shariff wrote: > Thank you for the response Mark , > >>> Are you able to test with a custom Tomcat build and/or build Tomcat 9 from >>> source for testing? > Yes Mark , we will be able to test with the jars built from Tomcat 9 source > for testing . The reduced memory footprint changes have now been back-ported to 9.0.x. Please let us know how you get on with your testing. Mark > > Thanks and Regards > Arshiya Shariff > > -Original Message- > From: Mark Thomas > Sent: Tuesday, September 29, 2020 12:25 AM > To: users@tomcat.apache.org > Subject: Re: HTTP2: memory filled up fast on increasing the connections to > 1000/2000 (Embedded tomcat 9.0.38) > > On 28/09/2020 17:58, Arshiya Shariff wrote: >> Hi All, >> With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and >> by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat >> application we did not observe any memory issue , but on sending 1000 http2 >> requests/sec with 2000 or 1000 users from JMeter , the application's heap >> space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory >> clears and comes down to 4GB (expected) . >> >> Embedded tomcat Version:9.0.38 >> Max Threads : 200 >> All other properties are the tomcat defaults. >> >> Why is tomcat not able to process many connections ? > > You haven't provided any evidence that Tomcat isn't able to process "many" > connections. > >> Why is the memory filled when the connections are increased, are there any >> parameters to tune connections ? > > It looks like users == HTTP/2 Connection. Connections are required to > maintain state for closed streams for both prioritisation and for error > handling. More connections == more state == more memory. > > Given the number of connections increased by a factor of between 12.5 and 25, > that the memory usage only increased by a factor of 5 looks to be a positive > result rather than an issue. > > There are significant improvements to memory usage in this area in Tomcat > 10.0.x that will get back-ported to 9.0.x but more testing is required. > > Are you able to test with a custom Tomcat build and/or build Tomcat 9 from > source for testing? > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 13:53, Martin Grigorov wrote: > On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov > When I load test HTTP2 with POST (with big bodies) there are many errors > like: > > 1) > Exception in thread "https-jsse-nio-8080-exec-5" > java.nio.BufferOverflowException > at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957) > at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247) > at > org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100) > at > org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401) > at > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307) > at > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > at java.base/java.lang.Thread.run(Thread.java:832) > > 2) > Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events > SEVERE: Failed to register socket with selector from poller > java.nio.channels.ClosedChannelException > at > java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222) > at > org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609) > at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703) > at java.base/java.lang.Thread.run(Thread.java:832) That is helpful. Looks like you have found a way to reproduce the buffer issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710 Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Virtual event focussed on Tomcat Security
Greetings, Folks This plan about Tomcat security is very nice. We look forward to the meetings. Could we have a session related to " Best practices for using Tomcat + (Apache Web Server) Forward Proxy (FP) combo in a real production environment " where an application hosted in Tomcat (web) container, targets a destination system in the internet, through the FP ? The application communicates with the destination system on a TLS channel. The FP is placed in a perimeter zone. The role of FP is to route the intranet traffic to the destination system in internet. If it is desired to have TLS terminated on the FP, and a SSL (or TLS) intercept is being sought - what is the best way to accomplish this interception (so that the application's communication reaches the destination system smoothly) ? The TLS intercept portion intends to decrypt the TLS transactions, check for security compliance and then re-encrypt to push the traffic to the destination system. Is there any generalized document that makes assessment (and recommendations) of a Tomcat plus a Forward Proxy combo, in a real word set up ? Thanks, -Raghu -Original Message- From: Maarten van Hulsentop Sent: Wednesday, September 30, 2020 3:10 AM To: Tomcat Users List Subject: Re: Virtual event focussed on Tomcat Security Hi Mark, This sounds like a great idea to me. Security is a very important topic, and the maturity of the Tomcat makes it a very secure choice for users. I am sure a lot of people will be interested to join in. What is not completely clear to me on this event; would this event be focussed on improving the security of Tomcat from within (as a Hackathon suggests)? Like trying to find security flaws/improvements and get them fixed. or is this meant to be an educational event where information is shared about secure setups/hardening of the Tomcat in production systems? Or a little of both? For the educational/hardening aspect, it could be nice to team up with/involve OWASP? I am surely interested to pitch in on this topic! Kind regards, Maarten van Hulsentop Op di 29 sep. 2020 om 13:26 schreef Mark Thomas : > Hi all, > > We (the Tomcat community) have some funding from Google to help us > improve Tomcat security. Our original plan was to use the funding to > support an in-person security focussed hackathon. As you would expect, > those plans are on hold for now. We would, therefore, like to explore > the possibility of doing something virtually. > > The purpose of this email is to gather input from the community about > what such an event should look like. With that input we can put > together a plan for the event. So, over to you. What would your ideal > virtual event focussed on Tomcat Security look like? > > Thanks, > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
RE: [EXTERNAL] Re: Enabling FIPS for Tomcat
Thank you Mark! Will work on getting this added to the Wiki page. Would appreciate feedback from all to confirm accuracy/usefulness. Thanks, Amit -Original Message- From: Mark Thomas Sent: Tuesday, September 29, 2020 1:40 PM To: users@tomcat.apache.org Subject: [EXTERNAL] Re: Enabling FIPS for Tomcat On 29/09/2020 16:25, Amit Pande wrote: > Dear all, > > The link below documents how to enable FIPS (using Bouncy Castle) for Tomcat. > > https://github.com/amitlpande/tomcat-9-fips > > Kindly let me know your inputs if this needs any corrections, enhancements. > > Also, a request to Tomcat leads: It is possible for these steps to be > part of (extended) Tomcat documentation? Even if currently it uses > Bouncy Castle as FIPS JCA/JCE provider, with minor changes it would > work for any other provider too (e.g. CryptoComply for Java / CCJ from > Safelogic) You can always create a page in the wiki. Maybe "FAQ > Security > FIPS". You'll need to create an account and then ask (on this list is fine) for write access. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov
wrote:
> Hi,
>
> On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas wrote:
>
>> On 30/09/2020 06:42, Arshiya Shariff wrote:
>> > Hi Martin ,
>> >
>> > Thank you for the response.
>> >
>> > With a payload of 200 bytes we were able to send 20K requests/sec with
>> 200 users from Jmeter without any memory issue . On increasing the payload
>> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
>> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
>> memory is cleared in the G1 mixed GC itself , but with 1000 users the
>> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
>> seconds to clear the memory. These cases were executed with maxThreads 200
>> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
>> still GC was struggling .
>> >
>> > When we tried with 10 instances of JMeter , each with 100 users , where
>> each instance was started with a delay of 1 minute we were able to see 1000
>> connections created in tomcat without any memory issues. But when 1000
>> users are created using single instance of JMeter in 20 seconds , tomcat's
>> memory is filling fast- 20GB in 2 minutes.
>> > We suspect that the burst of connections being opened has a problem .
>> Please help us with the same .
>> >
>> > On analyzing the heap dump we see
>> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93%
>> of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.
>>
>> You can't have high throughput, low GC pauses and small heap sizes.
>> Broadly you can have any two of those three at the expense of the third.
>>
>> The way Tomcat currently retains information about completed h2 streams
>> means you are likely to need a large heap under heavy load. There are
>> some changes already in 10.0.x that I plan to back-port to 9.0.x and
>> 8.5.x later today that should significantly reduce the heap requirements.
>>
>
> Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from
> 10.0.x with Vegeta for 3 mins:
> https://pasteboard.co/JtshrAs.png
> As you can see the GC is properly cleaning the heap. At the end the memory
> is not released until the GC kicks.
>
> Note: this is with a GET request without a body! I'm going to start a new
> email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for
> very low load.
>
When I load test HTTP2 with POST (with big bodies) there are many errors
like:
1)
Exception in thread "https-jsse-nio-8080-exec-5"
java.nio.BufferOverflowException
at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
at
org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
at
org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:832)
2)
Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events
SEVERE: Failed to register socket with selector from poller
java.nio.channels.ClosedChannelException
at
java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703)
at java.base/java.lang.Thread.run(Thread.java:832)
And the client gets these kind of error responses:
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=7, ErrCode=PROTOCOL_ERROR,
debug=\"Connection [9354], Stream [7], The content length header value
[255] does not agree with the size of the data received [0]\"",
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=31,
ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [9355], Too much overhead so
the connection will be closed\""
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=6155, ErrCode=STREAM_CLOSED,
debug=\"Connection [10048], Stream [6153], State [CLOSED_RX], Frame type
[RST]\""
"status_codes": {
"0": 286,
"200": 35373
}
, i.e. just a small po
Exit code 6 on shutting down Tomcat service
Hello Tomcat team, we are having an issue that we were not able to resolve ourselves or using the existing documentation, so I'd like to ask you for help. Description: During Tomcat service shutdown (using command /bin/tomcat9.exe //SS//), we sometimes receive exit code 6. It happens underministically (or at least we don't know the exact circumstances under which the error code is returned). We were not able to find any information about this exit code or any suggestions for fixes in the documentation. Environment: Tomcat: 9.0.33 OS: Windows 2016, Version: 10.0, Flavor: Data Center Thank you for your assistance! Jakub Moravec jakub.mora...@getmanta.com -- *This email is intended solely for the addressee(s) and all its contents, including all attachments and files transmitted with it, represent confidential information. Unauthorized distribution, modification or disclosure of its contents and unauthorized reliance on its contents are prohibited. If you have received this email in error, please notify the sender immediately by return email. Please then delete the email from your system and do not (i) copy or distribute it, (ii) rely on its contents, or (iii) disclose its contents to any person.*
Re: HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM
On Wed, Sep 30, 2020 at 1:15 PM Martin Grigorov
wrote:
>
>
> On Wed, Sep 30, 2020 at 1:01 PM Martin Grigorov
> wrote:
>
>> Hi,
>>
>> I've tried to test the scenario of Arshiya Shariff:
>> "With a payload of 200 bytes we were able to send 20K requests/sec with
>> 200 users from Jmeter without any memory issue . On increasing the payload
>> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
>> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
>> memory is cleared in the G1 mixed GC itself , but with 1000 users the
>> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
>> seconds to clear the memory. These cases were executed with maxThreads 200
>> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
>> still GC was struggling ."
>>
>> My Servlet looks like this:
>> https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
>>
>> And the Vegeta command I issue is:
>>
>> jq -ncM '{"method": "POST", "url": "
>> https://localhost:8080/testbed/plaintext";, "body":"payload=Some payload"
>> | @base64, header: {"Content-Type":
>> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
>> -http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
>> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>>
>> The outcome is:
>>
>> {
>> "latencies": {
>> "total": 114369584,
>> "mean": 38123194,
>> "50th": 45737841,
>> "90th": 62258803,
>> "95th": 62258803,
>> "99th": 62258803,
>> "max": 62258803,
>> "min": 6372940
>> },
>> "bytes_in": {
>> "total": 24,
>> "mean": 8
>> },
>> "bytes_out": {
>> "total": 40,
>> "mean": 13.334
>> },
>> "earliest": "2020-09-30T12:40:15.208111926+03:00",
>> "latest": "2020-09-30T12:40:17.208170426+03:00",
>> "end": "2020-09-30T12:40:17.253908267+03:00",
>> "duration": 258500,
>> "wait": 45737841,
>> "requests": 3,
>> "rate": 1.499956126283306,
>> "throughput": 0.9776144183650195,
>> "success": 0.,
>> "status_codes": {
>> "0": 1,
>> "200": 2
>> },
>> "errors": [
>> "Post \"https://localhost:8080/testbed/plaintext\": http2: server
>> sent GOAWAY and closed the connection; LastStreamID=3,
>> ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
>> connection will be closed\""
>> ]
>> }
>>
>> I.e. it sends 1 POST request every second for 3 seconds and this leads to
>> GOAWAY+ENHANCE_YOUR_CALM !
>>
>> From my understanding on "overhead" this should happen if there are more
>> SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
>>
>> I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
>> so I will try to debug it or to reproduce it with Firefox (to be able to
>> use Wireframe).
>>
>> Any ideas why this happens ?
>>
>
> The reason is here:
> https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L1448
> overheadThreshold is 1024 and average is 20
> It goes twice in this method and then fails at
> https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L352
>
https://bz.apache.org/bugzilla/show_bug.cgi?id=63690#c7 explains the issue.
No idea why but Vegeta/Golang sends two DATA frames and the second one
(with END_STREAM) is empty:
30-Sep-2020 14:45:35.646 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream.emitHeader Connection [1], Stream [1], HTTP
header [accept-encoding], Value [gzip]
30-Sep-2020 14:45:35.646 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream.emitHeader Connection [1], Stream [1], HTTP
header [user-agent], Value [Go-http-client/2.0]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.swallow Connection [1], Stream [1],
Swallowed [0] bytes
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.validateFrame Connection [1], Stream
[1], Frame type [DATA], Flags [0], Payload size [20]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.readDataFrame Connection [1], Stream
[1], Data length, [20], Padding length [none]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream$StreamInputBuffer.onDataAvailable Data added
to inBuffer when read thread is waiting. Signalling that thread to continue
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.validateFrame Connection [1], Stream
[1], Frame type [DATA], Flags [1], Payload size [0]
>
>
>>
>> Martin
>>
>
Re: HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM
On Wed, Sep 30, 2020 at 1:01 PM Martin Grigorov
wrote:
> Hi,
>
> I've tried to test the scenario of Arshiya Shariff:
> "With a payload of 200 bytes we were able to send 20K requests/sec with
> 200 users from Jmeter without any memory issue . On increasing the payload
> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
> memory is cleared in the G1 mixed GC itself , but with 1000 users the
> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
> seconds to clear the memory. These cases were executed with maxThreads 200
> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
> still GC was struggling ."
>
> My Servlet looks like this:
> https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
>
> And the Vegeta command I issue is:
>
> jq -ncM '{"method": "POST", "url": "
> https://localhost:8080/testbed/plaintext";, "body":"payload=Some payload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>
> The outcome is:
>
> {
> "latencies": {
> "total": 114369584,
> "mean": 38123194,
> "50th": 45737841,
> "90th": 62258803,
> "95th": 62258803,
> "99th": 62258803,
> "max": 62258803,
> "min": 6372940
> },
> "bytes_in": {
> "total": 24,
> "mean": 8
> },
> "bytes_out": {
> "total": 40,
> "mean": 13.334
> },
> "earliest": "2020-09-30T12:40:15.208111926+03:00",
> "latest": "2020-09-30T12:40:17.208170426+03:00",
> "end": "2020-09-30T12:40:17.253908267+03:00",
> "duration": 258500,
> "wait": 45737841,
> "requests": 3,
> "rate": 1.499956126283306,
> "throughput": 0.9776144183650195,
> "success": 0.,
> "status_codes": {
> "0": 1,
> "200": 2
> },
> "errors": [
> "Post \"https://localhost:8080/testbed/plaintext\": http2: server
> sent GOAWAY and closed the connection; LastStreamID=3,
> ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
> connection will be closed\""
> ]
> }
>
> I.e. it sends 1 POST request every second for 3 seconds and this leads to
> GOAWAY+ENHANCE_YOUR_CALM !
>
> From my understanding on "overhead" this should happen if there are more
> SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
>
> I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
> so I will try to debug it or to reproduce it with Firefox (to be able to
> use Wireframe).
>
> Any ideas why this happens ?
>
The reason is here:
https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L1448
overheadThreshold is 1024 and average is 20
It goes twice in this method and then fails at
https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L352
>
> Martin
>
HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM
Hi,
I've tried to test the scenario of Arshiya Shariff:
"With a payload of 200 bytes we were able to send 20K requests/sec with 200
users from Jmeter without any memory issue . On increasing the payload to
5Kb and the number of users to 1000 in Jmeter and sending 1000 requests per
second , the heap of 20GB got filled in 2 minutes . With 200 users the
memory is cleared in the G1 mixed GC itself , but with 1000 users the
memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
seconds to clear the memory. These cases were executed with maxThreads 200
in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
still GC was struggling ."
My Servlet looks like this:
https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
And the Vegeta command I issue is:
jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext";,
"body":"payload=Some payload" | @base64, header: {"Content-Type":
["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
-http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
/tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
The outcome is:
{
"latencies": {
"total": 114369584,
"mean": 38123194,
"50th": 45737841,
"90th": 62258803,
"95th": 62258803,
"99th": 62258803,
"max": 62258803,
"min": 6372940
},
"bytes_in": {
"total": 24,
"mean": 8
},
"bytes_out": {
"total": 40,
"mean": 13.334
},
"earliest": "2020-09-30T12:40:15.208111926+03:00",
"latest": "2020-09-30T12:40:17.208170426+03:00",
"end": "2020-09-30T12:40:17.253908267+03:00",
"duration": 258500,
"wait": 45737841,
"requests": 3,
"rate": 1.499956126283306,
"throughput": 0.9776144183650195,
"success": 0.,
"status_codes": {
"0": 1,
"200": 2
},
"errors": [
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=3,
ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
connection will be closed\""
]
}
I.e. it sends 1 POST request every second for 3 seconds and this leads to
GOAWAY+ENHANCE_YOUR_CALM !
>From my understanding on "overhead" this should happen if there are more
SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
so I will try to debug it or to reproduce it with Firefox (to be able to
use Wireframe).
Any ideas why this happens ?
Martin
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi, On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas wrote: > On 30/09/2020 06:42, Arshiya Shariff wrote: > > Hi Martin , > > > > Thank you for the response. > > > > With a payload of 200 bytes we were able to send 20K requests/sec with > 200 users from Jmeter without any memory issue . On increasing the payload > to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests > per second , the heap of 20GB got filled in 2 minutes . With 200 users the > memory is cleared in the G1 mixed GC itself , but with 1000 users the > memory is not cleared in the mixed GC , it takes full GCs of 7 to 10 > seconds to clear the memory. These cases were executed with maxThreads 200 > in tomcat , so we tried increasing the maxThreads from 200 to 1000, but > still GC was struggling . > > > > When we tried with 10 instances of JMeter , each with 100 users , where > each instance was started with a delay of 1 minute we were able to see 1000 > connections created in tomcat without any memory issues. But when 1000 > users are created using single instance of JMeter in 20 seconds , tomcat's > memory is filling fast- 20GB in 2 minutes. > > We suspect that the burst of connections being opened has a problem . > Please help us with the same . > > > > On analyzing the heap dump we see > org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% > of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump. > > You can't have high throughput, low GC pauses and small heap sizes. > Broadly you can have any two of those three at the expense of the third. > > The way Tomcat currently retains information about completed h2 streams > means you are likely to need a large heap under heavy load. There are > some changes already in 10.0.x that I plan to back-port to 9.0.x and > 8.5.x later today that should significantly reduce the heap requirements. > Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from 10.0.x with Vegeta for 3 mins: https://pasteboard.co/JtshrAs.png As you can see the GC is properly cleaning the heap. At the end the memory is not released until the GC kicks. Note: this is with a GET request without a body! I'm going to start a new email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for very low load. Martin > > Mark > > > > > > Thanks and Regards > > Arshiya Shariff > > > > -Original Message- > > From: Martin Grigorov > > Sent: Monday, September 28, 2020 11:44 PM > > To: Tomcat Users List > > Subject: Re: HTTP2: memory filled up fast on increasing the connections > to 1000/2000 (Embedded tomcat 9.0.38) > > > > Hi Arshiya, > > > > > > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff < > arshiya.shar...@ericsson.com.invalid> wrote: > > > >> Hi All, > >> With 200 threads(users) , ramp up duration of 2 seconds , loop count > >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an > >> embedded tomcat application we did not observe any memory issue , but > >> on sending > >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the > >> application's heap space of 20 GB is occupied in 2 minutes and after 2 > >> full GCs the memory clears and comes down to 4GB (expected) . > >> > > > > I am not sure whether you follow the other discussions at users@. > > In another email thread we discuss load testing Tomcat HTTP2 and we are > able to make around 12K reqs/s with another load testing tool - > https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f&q=1&e=6a9c569d-7da1-4394-a9ac-bf72724992fa&u=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta > > For me JMeter itself failed with OOM when increasing the number of the > virtual users above 2K. > > There are several improvements in Tomcat master and 9.0.x in the HTTP2 > area. Some of the changes are not yet downported to 9.0.x. We still test > them, trying to avoid introducing regressions in 9.0.x. > > > > > >> > >> Embedded tomcat Version:9.0.38 > >> Max Threads : 200 > >> > > > > The number of threads should be less if you do only CPU calculations > without IO/network. If your app blocks on IO/network calls then you need > more spare threads. > > With more threads there will be more context switches and less > throughput. > > That's why there is no one golden rule that applies to all applications. > > 200 is a good default that works for most of the applications. But you > need to test with different values to see which one gives the best > performance for your scenaria. > > > > > >> All other properties are the tomcat defaults. > >> > >> Why is tomcat not able to process many connections ? > >> > > > > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and > -XX:HeapDumpPath=. Once you have the .hprof file you can > examine it with Eclipse Memory Analyzer tool and see what is leaking. > > I will try to reproduce this issue tomorrow with Vegeta. > > > > > >> Why is the memory filled when the connections are increased, are there >
Re: Virtual event focussed on Tomcat Security
Hi Mark, This sounds like a great idea to me. Security is a very important topic, and the maturity of the Tomcat makes it a very secure choice for users. I am sure a lot of people will be interested to join in. What is not completely clear to me on this event; would this event be focussed on improving the security of Tomcat from within (as a Hackathon suggests)? Like trying to find security flaws/improvements and get them fixed. or is this meant to be an educational event where information is shared about secure setups/hardening of the Tomcat in production systems? Or a little of both? For the educational/hardening aspect, it could be nice to team up with/involve OWASP? I am surely interested to pitch in on this topic! Kind regards, Maarten van Hulsentop Op di 29 sep. 2020 om 13:26 schreef Mark Thomas : > Hi all, > > We (the Tomcat community) have some funding from Google to help us > improve Tomcat security. Our original plan was to use the funding to > support an in-person security focussed hackathon. As you would expect, > those plans are on hold for now. We would, therefore, like to explore > the possibility of doing something virtually. > > The purpose of this email is to gather input from the community about > what such an event should look like. With that input we can put together > a plan for the event. So, over to you. What would your ideal virtual > event focussed on Tomcat Security look like? > > Thanks, > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 06:42, Arshiya Shariff wrote: > Hi Martin , > > Thank you for the response. > > With a payload of 200 bytes we were able to send 20K requests/sec with 200 > users from Jmeter without any memory issue . On increasing the payload to 5Kb > and the number of users to 1000 in Jmeter and sending 1000 requests per > second , the heap of 20GB got filled in 2 minutes . With 200 users the memory > is cleared in the G1 mixed GC itself , but with 1000 users the memory is not > cleared in the mixed GC , it takes full GCs of 7 to 10 seconds to clear the > memory. These cases were executed with maxThreads 200 in tomcat , so we tried > increasing the maxThreads from 200 to 1000, but still GC was struggling . > > When we tried with 10 instances of JMeter , each with 100 users , where each > instance was started with a delay of 1 minute we were able to see 1000 > connections created in tomcat without any memory issues. But when 1000 users > are created using single instance of JMeter in 20 seconds , tomcat's memory > is filling fast- 20GB in 2 minutes. > We suspect that the burst of connections being opened has a problem . Please > help us with the same . > > On analyzing the heap dump we see > org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% of > 3GB live data ,the remaining 17GB is Garbage collected in the heap dump. You can't have high throughput, low GC pauses and small heap sizes. Broadly you can have any two of those three at the expense of the third. The way Tomcat currently retains information about completed h2 streams means you are likely to need a large heap under heavy load. There are some changes already in 10.0.x that I plan to back-port to 9.0.x and 8.5.x later today that should significantly reduce the heap requirements. Mark > > Thanks and Regards > Arshiya Shariff > > -Original Message- > From: Martin Grigorov > Sent: Monday, September 28, 2020 11:44 PM > To: Tomcat Users List > Subject: Re: HTTP2: memory filled up fast on increasing the connections to > 1000/2000 (Embedded tomcat 9.0.38) > > Hi Arshiya, > > > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff > wrote: > >> Hi All, >> With 200 threads(users) , ramp up duration of 2 seconds , loop count >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an >> embedded tomcat application we did not observe any memory issue , but >> on sending >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the >> application's heap space of 20 GB is occupied in 2 minutes and after 2 >> full GCs the memory clears and comes down to 4GB (expected) . >> > > I am not sure whether you follow the other discussions at users@. > In another email thread we discuss load testing Tomcat HTTP2 and we are able > to make around 12K reqs/s with another load testing tool - > https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f&q=1&e=6a9c569d-7da1-4394-a9ac-bf72724992fa&u=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta > For me JMeter itself failed with OOM when increasing the number of the > virtual users above 2K. > There are several improvements in Tomcat master and 9.0.x in the HTTP2 area. > Some of the changes are not yet downported to 9.0.x. We still test them, > trying to avoid introducing regressions in 9.0.x. > > >> >> Embedded tomcat Version:9.0.38 >> Max Threads : 200 >> > > The number of threads should be less if you do only CPU calculations without > IO/network. If your app blocks on IO/network calls then you need more spare > threads. > With more threads there will be more context switches and less throughput. > That's why there is no one golden rule that applies to all applications. > 200 is a good default that works for most of the applications. But you need > to test with different values to see which one gives the best performance for > your scenaria. > > >> All other properties are the tomcat defaults. >> >> Why is tomcat not able to process many connections ? >> > > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and > -XX:HeapDumpPath=. Once you have the .hprof file you can > examine it with Eclipse Memory Analyzer tool and see what is leaking. > I will try to reproduce this issue tomorrow with Vegeta. > > >> Why is the memory filled when the connections are increased, are there >> any parameters to tune connections ? >> Please let us know. >> >> Thanks and Regards >> Arshiya Shariff >> > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Thank you for the response Mark , >> Are you able to test with a custom Tomcat build and/or build Tomcat 9 from >> source for testing? Yes Mark , we will be able to test with the jars built from Tomcat 9 source for testing . Thanks and Regards Arshiya Shariff -Original Message- From: Mark Thomas Sent: Tuesday, September 29, 2020 12:25 AM To: users@tomcat.apache.org Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) On 28/09/2020 17:58, Arshiya Shariff wrote: > Hi All, > With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and > by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat > application we did not observe any memory issue , but on sending 1000 http2 > requests/sec with 2000 or 1000 users from JMeter , the application's heap > space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory > clears and comes down to 4GB (expected) . > > Embedded tomcat Version:9.0.38 > Max Threads : 200 > All other properties are the tomcat defaults. > > Why is tomcat not able to process many connections ? You haven't provided any evidence that Tomcat isn't able to process "many" connections. > Why is the memory filled when the connections are increased, are there any > parameters to tune connections ? It looks like users == HTTP/2 Connection. Connections are required to maintain state for closed streams for both prioritisation and for error handling. More connections == more state == more memory. Given the number of connections increased by a factor of between 12.5 and 25, that the memory usage only increased by a factor of 5 looks to be a positive result rather than an issue. There are significant improvements to memory usage in this area in Tomcat 10.0.x that will get back-ported to 9.0.x but more testing is required. Are you able to test with a custom Tomcat build and/or build Tomcat 9 from source for testing? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 30.09.2020 07:42, Arshiya Shariff wrote: Hi Martin , Thank you for the response. With a payload of 200 bytes we were able to send 20K requests/sec with 200 users from Jmeter without any memory issue . On increasing the payload to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests per second , the heap of 20GB got filled in 2 minutes . How long does it typically take (at the beginning of the test) for tomcat to *process* one of these requests ? With 200 users the memory is cleared in the G1 mixed GC itself , but with 1000 users the memory is not cleared in the mixed GC , it takes full GCs of 7 to 10 seconds to clear the memory. These cases were executed with maxThreads 200 in tomcat , so we tried increasing the maxThreads from 200 to 1000, but still GC was struggling . When we tried with 10 instances of JMeter , each with 100 users , where each instance was started with a delay of 1 minute we were able to see 1000 connections created in tomcat without any memory issues. But when 1000 users are created using single instance of JMeter in 20 seconds , tomcat's memory is filling fast- 20GB in 2 minutes. We suspect that the burst of connections being opened has a problem . Please help us with the same . On analyzing the heap dump we see org.apache.tomcat.util.collections.SynchronizedStack occupying around 93% of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump. Thanks and Regards Arshiya Shariff -Original Message- From: Martin Grigorov Sent: Monday, September 28, 2020 11:44 PM To: Tomcat Users List Subject: Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38) Hi Arshiya, On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff wrote: Hi All, With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat application we did not observe any memory issue , but on sending 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the application's heap space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory clears and comes down to 4GB (expected) . I am not sure whether you follow the other discussions at users@. In another email thread we discuss load testing Tomcat HTTP2 and we are able to make around 12K reqs/s with another load testing tool - https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f&q=1&e=6a9c569d-7da1-4394-a9ac-bf72724992fa&u=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta For me JMeter itself failed with OOM when increasing the number of the virtual users above 2K. There are several improvements in Tomcat master and 9.0.x in the HTTP2 area. Some of the changes are not yet downported to 9.0.x. We still test them, trying to avoid introducing regressions in 9.0.x. Embedded tomcat Version:9.0.38 Max Threads : 200 The number of threads should be less if you do only CPU calculations without IO/network. If your app blocks on IO/network calls then you need more spare threads. With more threads there will be more context switches and less throughput. That's why there is no one golden rule that applies to all applications. 200 is a good default that works for most of the applications. But you need to test with different values to see which one gives the best performance for your scenaria. All other properties are the tomcat defaults. Why is tomcat not able to process many connections ? You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=. Once you have the .hprof file you can examine it with Eclipse Memory Analyzer tool and see what is leaking. I will try to reproduce this issue tomorrow with Vegeta. Why is the memory filled when the connections are increased, are there any parameters to tune connections ? Please let us know. Thanks and Regards Arshiya Shariff - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
