Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 28/09/2020 17:58, Arshiya Shariff wrote: > Hi All, > With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and > by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat > application we did not observe any memory issue , but on sending 1000 http2 > requests/sec with 2000 or 1000 users from JMeter , the application's heap > space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory > clears and comes down to 4GB (expected) . > > Embedded tomcat Version:9.0.38 > Max Threads : 200 > All other properties are the tomcat defaults. > > Why is tomcat not able to process many connections ? You haven't provided any evidence that Tomcat isn't able to process "many" connections. > Why is the memory filled when the connections are increased, are there any > parameters to tune connections ? It looks like users == HTTP/2 Connection. Connections are required to maintain state for closed streams for both prioritisation and for error handling. More connections == more state == more memory. Given the number of connections increased by a factor of between 12.5 and 25, that the memory usage only increased by a factor of 5 looks to be a positive result rather than an issue. There are significant improvements to memory usage in this area in Tomcat 10.0.x that will get back-ported to 9.0.x but more testing is required. Are you able to test with a custom Tomcat build and/or build Tomcat 9 from source for testing? Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi Arshiya, On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff wrote: > Hi All, > With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 > and by sending 1000 http2 requests/sec from JMeter Client to an embedded > tomcat application we did not observe any memory issue , but on sending > 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the > application's heap space of 20 GB is occupied in 2 minutes and after 2 full > GCs the memory clears and comes down to 4GB (expected) . > I am not sure whether you follow the other discussions at users@. In another email thread we discuss load testing Tomcat HTTP2 and we are able to make around 12K reqs/s with another load testing tool - https://github.com/tsenart/vegeta For me JMeter itself failed with OOM when increasing the number of the virtual users above 2K. There are several improvements in Tomcat master and 9.0.x in the HTTP2 area. Some of the changes are not yet downported to 9.0.x. We still test them, trying to avoid introducing regressions in 9.0.x. > > Embedded tomcat Version:9.0.38 > Max Threads : 200 > The number of threads should be less if you do only CPU calculations without IO/network. If your app blocks on IO/network calls then you need more spare threads. With more threads there will be more context switches and less throughput. That's why there is no one golden rule that applies to all applications. 200 is a good default that works for most of the applications. But you need to test with different values to see which one gives the best performance for your scenaria. > All other properties are the tomcat defaults. > > Why is tomcat not able to process many connections ? > You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath=. Once you have the .hprof file you can examine it with Eclipse Memory Analyzer tool and see what is leaking. I will try to reproduce this issue tomorrow with Vegeta. > Why is the memory filled when the connections are increased, are there any > parameters to tune connections ? > Please let us know. > > Thanks and Regards > Arshiya Shariff >
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Arshiya, On 9/28/20 12:58, Arshiya Shariff wrote: > With 200 threads(users) , ramp up duration of 2 seconds , loop count > 80 and by sending 1000 http2 requests/sec from JMeter Client to an > embedded tomcat application we did not observe any memory issue , but > on sending 1000 http2 requests/sec with 2000 or 1000 users from > JMeter , the application's heap space of 20 GB is occupied in 2 > minutes and after 2 full GCs the memory clears and comes down to 4GB > (expected) . So a full GC releases the memory? > Embedded tomcat Version:9.0.38 > Max Threads : 200 > All other properties are the tomcat defaults. > > Why is tomcat not able to process many connections ? What evidence is there that Tomcat cannot process that many connections? You said above the only concern was used-heap space. If you have 200 threads, then you can only handle 200 active requests at a time. If you need to process 2000 requests *simultaneously*, then you need 2000 threads. If you only need 2000 *users* at the "same time", then 200 threads should be able to handle the load, depending upon the application's performance characteristics. > Why is the memory filled when the connections are increased, are > there any parameters to tune connections ? Are you using HttpSessions? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi All, With 200 threads(users) , ramp up duration of 2 seconds , loop count 80 and by sending 1000 http2 requests/sec from JMeter Client to an embedded tomcat application we did not observe any memory issue , but on sending 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the application's heap space of 20 GB is occupied in 2 minutes and after 2 full GCs the memory clears and comes down to 4GB (expected) . Embedded tomcat Version:9.0.38 Max Threads : 200 All other properties are the tomcat defaults. Why is tomcat not able to process many connections ? Why is the memory filled when the connections are increased, are there any parameters to tune connections ? Please let us know. Thanks and Regards Arshiya Shariff
Re: RemoteIpValve doesn't maintain context in worker thread after startAsync
On 28/09/2020 16:15, Solas, Nathan wrote:
> I'm using RemoteIpValve to capture protocolHeader x-forwarded-proto and
> upgrade the request to secure when SSL is terminated at the load balancer -
> so far so good.
>
> When using theServletRequest.startAsync() and then passing the work to a
> threadpool executor, it seems the RemoteIpValve.invoke finally{} block
> executes first and undoes all the changes to the Request object. The result
> is by the time the worker picks up the request, it's demoted back to http and
> nothing works.
>
> This was noted in 2017 with no answers, but the internet is pretty quiet
> about others facing this:
> https://marc.info/?l=tomcat-user=150971153905722=2
>
> Maybe I'm not understanding how this should work? Seems right to me... Thanks
> for any information,
> Nate
You'll probably have more luck with the Filter than the Valve. One of
the advantages the Filter has is that it can warp the request and/or
response.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ByteBuffer pooling
On Mon, Sep 28, 2020 at 5:31 PM Martin Grigorov wrote: > On Mon, Sep 28, 2020 at 6:11 PM Rémy Maucherat wrote: > > > On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov > > wrote: > > > > > Hi, > > > > > > I've profiled the memory allocation during load testing HTTP2: > > > https://pasteboard.co/Jtblqfl.png > > > > > > As you can see there are a lot of ByteBuffer.allocate(int) calls. > > > org.apache.catalina.connector.Response#Response(int) > > > org.apache.catalina.connector.Request#inputBuffer > > > org.apache.coyote.http2.Http2AsyncParser#readFrame > > > org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer > > > > > > Has it been discussed in the past to use a pool of ByteBuffer > instances ? > > > Netty provides such for its abstraction ByteBuf: > > > https://netty.io/wiki/using-as-a-generic-library.html > > > > > > Here is a simple implementation: > > > > > > > > > https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44 > > > It's give() method could be optimized to not return the buffer to the > > pool > > > if the pool size is bigger than N, so it doesn't use huge list of > buffers > > > which are used just once. > > > > > > The big unknown to me is: where to return the buffers to the pool ? > > > For HTTP2 maybe this could be done after writing the buffer to the > > socket. > > > If the Request/Response is recycled then their Input/OutputBuffer are > > > reused and everything is OK. But if recycling is not in use then new > > > allocations are done for each new request. > > > > > > I see that org.apache.tomcat.util.net.WriteBuffer does some pooling > > > already. > > > > > > What do you think? > > > > > > > I removed a lot of pooling in the connector with zero impact, so I don't > > see the point. > > > > ConcurrentDateFormat pools SimpleDateFormat instances and it does make a > difference: > > https://medium.com/@martin.grigorov/compare-performance-of-javas-simpledateformat-against-datetimeformatter-31be58cadf1d > ! > I'd give it a try for the HTTP2 Stream related allocations and let you know > how it goes! > This is not a simple object, this involves parsing and computations. A BB however is a very simple object, even if this somehow shows up in a microbenchmark the actual impact will be negative (more memory is retained, the life of the GC is harder, and most importantly contention on the cache). I mean these were the optimizations I was doing in the 2000s, but now it seems to work as the JDK people say. There's an exception for direct BBs in theory, but in practice retaining the memory is an even bigger issue. Rémy
Re: ByteBuffer pooling
On Mon, Sep 28, 2020 at 6:11 PM Rémy Maucherat wrote: > On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov > wrote: > > > Hi, > > > > I've profiled the memory allocation during load testing HTTP2: > > https://pasteboard.co/Jtblqfl.png > > > > As you can see there are a lot of ByteBuffer.allocate(int) calls. > > org.apache.catalina.connector.Response#Response(int) > > org.apache.catalina.connector.Request#inputBuffer > > org.apache.coyote.http2.Http2AsyncParser#readFrame > > org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer > > > > Has it been discussed in the past to use a pool of ByteBuffer instances ? > > Netty provides such for its abstraction ByteBuf: > > https://netty.io/wiki/using-as-a-generic-library.html > > > > Here is a simple implementation: > > > > > https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44 > > It's give() method could be optimized to not return the buffer to the > pool > > if the pool size is bigger than N, so it doesn't use huge list of buffers > > which are used just once. > > > > The big unknown to me is: where to return the buffers to the pool ? > > For HTTP2 maybe this could be done after writing the buffer to the > socket. > > If the Request/Response is recycled then their Input/OutputBuffer are > > reused and everything is OK. But if recycling is not in use then new > > allocations are done for each new request. > > > > I see that org.apache.tomcat.util.net.WriteBuffer does some pooling > > already. > > > > What do you think? > > > > I removed a lot of pooling in the connector with zero impact, so I don't > see the point. > ConcurrentDateFormat pools SimpleDateFormat instances and it does make a difference: https://medium.com/@martin.grigorov/compare-performance-of-javas-simpledateformat-against-datetimeformatter-31be58cadf1d ! I'd give it a try for the HTTP2 Stream related allocations and let you know how it goes! > > Rémy > > > > > > Martin > > >
RemoteIpValve doesn't maintain context in worker thread after startAsync
I'm using RemoteIpValve to capture protocolHeader x-forwarded-proto and upgrade
the request to secure when SSL is terminated at the load balancer - so far so
good.
When using theServletRequest.startAsync() and then passing the work to a
threadpool executor, it seems the RemoteIpValve.invoke finally{} block executes
first and undoes all the changes to the Request object. The result is by the
time the worker picks up the request, it's demoted back to http and nothing
works.
This was noted in 2017 with no answers, but the internet is pretty quiet about
others facing this:
https://marc.info/?l=tomcat-user=150971153905722=2
Maybe I'm not understanding how this should work? Seems right to me... Thanks
for any information,
Nate
Please be advised that this email may contain confidential information. If you
are not the intended recipient, please notify us by email by replying to the
sender and delete this message. The sender disclaims that the content of this
email constitutes an offer to enter into, or the acceptance of, any agreement;
provided that the foregoing does not invalidate the binding effect of any
digital or other electronic reproduction of a manual signature that is included
in any attachment.
Re: Some functions not working when using a particular dns after tomcat upgrade from 6.x to 8.5.x
Larvi, On 9/28/20 10:04, Larvi Boy wrote: > Hi, > > When I try to login to out web gui via direct link, it is working fine but > when I used the dns url, for first time it works fine as for the first time > we are redirected to our login page which redirects us back to my direct > link, but if we create another window with same dns link, some buttons in > the jsp are not working. We cleared the cache but didn't help. > > I checked the application logs but there were no logs for the actions that > should occur after click and I checked tomcat catalina.out and localhost > logs and there is no error there. > > We have 2 dns urls but we are not facing this issue with the other dns url. > > Can you please help me on this. > > Please ask if more information is needed. Can you give some examples? What happens if you: $ host [hostname] Do you get the same IP address that you are trying to use? Please post your and configuration from server.xml. Remove any secrets you may have in there. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: ByteBuffer pooling
On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov wrote: > Hi, > > I've profiled the memory allocation during load testing HTTP2: > https://pasteboard.co/Jtblqfl.png > > As you can see there are a lot of ByteBuffer.allocate(int) calls. > org.apache.catalina.connector.Response#Response(int) > org.apache.catalina.connector.Request#inputBuffer > org.apache.coyote.http2.Http2AsyncParser#readFrame > org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer > > Has it been discussed in the past to use a pool of ByteBuffer instances ? > Netty provides such for its abstraction ByteBuf: > https://netty.io/wiki/using-as-a-generic-library.html > > Here is a simple implementation: > > https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44 > It's give() method could be optimized to not return the buffer to the pool > if the pool size is bigger than N, so it doesn't use huge list of buffers > which are used just once. > > The big unknown to me is: where to return the buffers to the pool ? > For HTTP2 maybe this could be done after writing the buffer to the socket. > If the Request/Response is recycled then their Input/OutputBuffer are > reused and everything is OK. But if recycling is not in use then new > allocations are done for each new request. > > I see that org.apache.tomcat.util.net.WriteBuffer does some pooling > already. > > What do you think? > I removed a lot of pooling in the connector with zero impact, so I don't see the point. Rémy > > Martin >
ByteBuffer pooling
Hi, I've profiled the memory allocation during load testing HTTP2: https://pasteboard.co/Jtblqfl.png As you can see there are a lot of ByteBuffer.allocate(int) calls. org.apache.catalina.connector.Response#Response(int) org.apache.catalina.connector.Request#inputBuffer org.apache.coyote.http2.Http2AsyncParser#readFrame org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer Has it been discussed in the past to use a pool of ByteBuffer instances ? Netty provides such for its abstraction ByteBuf: https://netty.io/wiki/using-as-a-generic-library.html Here is a simple implementation: https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44 It's give() method could be optimized to not return the buffer to the pool if the pool size is bigger than N, so it doesn't use huge list of buffers which are used just once. The big unknown to me is: where to return the buffers to the pool ? For HTTP2 maybe this could be done after writing the buffer to the socket. If the Request/Response is recycled then their Input/OutputBuffer are reused and everything is OK. But if recycling is not in use then new allocations are done for each new request. I see that org.apache.tomcat.util.net.WriteBuffer does some pooling already. What do you think? Martin
Some functions not working when using a particular dns after tomcat upgrade from 6.x to 8.5.x
Hi, When I try to login to out web gui via direct link, it is working fine but when I used the dns url, for first time it works fine as for the first time we are redirected to our login page which redirects us back to my direct link, but if we create another window with same dns link, some buttons in the jsp are not working. We cleared the cache but didn't help. I checked the application logs but there were no logs for the actions that should occur after click and I checked tomcat catalina.out and localhost logs and there is no error there. We have 2 dns urls but we are not facing this issue with the other dns url. Can you please help me on this. Please ask if more information is needed. Thanks, Larvi
Re: Connection header override
Mark,
On 9/28/20 03:48, Mark Thomas wrote:
> On 28/09/2020 08:33, Mark Thomas wrote:
>> On 27/09/2020 00:07, Pawel Veselov wrote:
>>> Hello!
>>>
>>> Tomcat 9.0.x
>>>
>>> I'd like to force connection closure on some endpoints.
>>
>> Why? Generally, this is something that should not be an application concern.
>>
>>> I'm trying this on a simple JSP page.
>>> If I call response.setHeader("Connection","close"), I see that the
>>> response has "Connection: close, keep-alive".
>>> I assume Tomcat inserts the keep-alive part. It looks like the browsers
>>> still close the connection based on this, but I was wondering if it's
>>> possible to have Tomcat honor the header value set by the application.
>>
>> The most recent discussion on this topic was whether or not Tomcat
>> should block any attempt by an application to manipulate the Connection
>> header. The consensus was leaning towards implementing a block but
>> no-one has implemented it yet.
>>
>> See https://github.com/apache/tomcat/pull/277
>>
>> I did wonder if this was a regression introduced by some clean-up in the
>> handling of the Connection header a little while ago but it appears not.
>> Note that Tomcat will only add this header for HTTP/1.0 requests.
>> Separately, you may also see a "Keep-Alive" header for HTTP/1.1 requests.
>
> Testing shows that isn't right - I was mis-reading the code. You will
> see this with HTTP/1.1 requests where the client explicitly sends a
> "Connection: keep-alive header".
>
> I'm currently working on expanding the unit tests to cover this
> (although I'd still like to know why the app needs to close the connection).
One reason may be to "punish" a client for misbehaving in some way (e.g.
lots of failed logins, etc.). It's not much of a punishment, but forcing
a fresh TCP/IP and TLS handshake will slow down a brute-force script a bit.
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection header override
On 28/09/2020 08:33, Mark Thomas wrote:
> On 27/09/2020 00:07, Pawel Veselov wrote:
>> Hello!
>>
>> Tomcat 9.0.x
>>
>> I'd like to force connection closure on some endpoints.
>
> Why? Generally, this is something that should not be an application concern.
>
>> I'm trying this on a simple JSP page.
>> If I call response.setHeader("Connection","close"), I see that the
>> response has "Connection: close, keep-alive".
>> I assume Tomcat inserts the keep-alive part. It looks like the browsers
>> still close the connection based on this, but I was wondering if it's
>> possible to have Tomcat honor the header value set by the application.
>
> The most recent discussion on this topic was whether or not Tomcat
> should block any attempt by an application to manipulate the Connection
> header. The consensus was leaning towards implementing a block but
> no-one has implemented it yet.
>
> See https://github.com/apache/tomcat/pull/277
>
> I did wonder if this was a regression introduced by some clean-up in the
> handling of the Connection header a little while ago but it appears not.
> Note that Tomcat will only add this header for HTTP/1.0 requests.
> Separately, you may also see a "Keep-Alive" header for HTTP/1.1 requests.
Testing shows that isn't right - I was mis-reading the code. You will
see this with HTTP/1.1 requests where the client explicitly sends a
"Connection: keep-alive header".
I'm currently working on expanding the unit tests to cover this
(although I'd still like to know why the app needs to close the connection).
Mark
>
> I'm leaning towards a small fix that would prevent the keep-alive header
> being added in this case.
>
>> I was also wondering what does it mean - when multiple connection tokens
>> are specified for a connection header. Breezing through the RFCs I can't
>> find a clear statement on that except that multiple connection tokens are
>> allowed in the header...
>
> As per RFC 7230, section 6.3 if the close option is present it takes
> priority.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Connection header override
On 27/09/2020 00:07, Pawel Veselov wrote:
> Hello!
>
> Tomcat 9.0.x
>
> I'd like to force connection closure on some endpoints.
Why? Generally, this is something that should not be an application concern.
> I'm trying this on a simple JSP page.
> If I call response.setHeader("Connection","close"), I see that the
> response has "Connection: close, keep-alive".
> I assume Tomcat inserts the keep-alive part. It looks like the browsers
> still close the connection based on this, but I was wondering if it's
> possible to have Tomcat honor the header value set by the application.
The most recent discussion on this topic was whether or not Tomcat
should block any attempt by an application to manipulate the Connection
header. The consensus was leaning towards implementing a block but
no-one has implemented it yet.
See https://github.com/apache/tomcat/pull/277
I did wonder if this was a regression introduced by some clean-up in the
handling of the Connection header a little while ago but it appears not.
Note that Tomcat will only add this header for HTTP/1.0 requests.
Separately, you may also see a "Keep-Alive" header for HTTP/1.1 requests.
I'm leaning towards a small fix that would prevent the keep-alive header
being added in this case.
> I was also wondering what does it mean - when multiple connection tokens
> are specified for a connection header. Breezing through the RFCs I can't
> find a clear statement on that except that multiple connection tokens are
> allowed in the header...
As per RFC 7230, section 6.3 if the close option is present it takes
priority.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
