On 16/06/2021 15:05, Deshmukh, Kedar wrote:
Dear Tomcat users/dev team,
We are understanding the impact of HTTP/2 in our application as HTTP/2
provides better throughput and performance.
I'd be wary of making such sweeping statements. HTTP/2 has some
advantages and some disadvantages. Generally, the advantages will
outweigh the disadvantages but that will not always be the case.
Before directly tuning
HTTP/2 in application, we thought of analyzing certain use cases which
our application demands in standalone environment.
Our use case is very simple. Java based standalone client is making
simple POST request to the server and server read the file name from the
request and push the requested file to client. Here, client can request
multiple files same time by sending multiple requests concurrently, so
server should be able to send multiple files concurrently depending on
configured concurrency level.
Currently, in this test only single client is making requests to the
server with concurrency 5. Server is not overloaded and not performing
any other tasks. Machine has more than 500GB empty space and not running
any heavy applications.
Test:
We used different set of files for this test. Files with sizes between
1GB – 5GB and concurrency > 5. We are using traditional connector
protocol HTTP11NIOProtocol with HTTP/2 is turned on.
Observations:
HTTP/1.1 - With HTTP/1.1 given sample code works fine. Only drawback
here is it opens multiple TCP connections to satisfy HTTP/1.1
HTTP/2 - With HTTP/2, it is expected to be only one TCP connection and
multiple streams to handle the traffic. Tomcat HTTP/2 debug logs suggest
that only one connection being used and multiple streams are spawned as
expected. So far everything is fine. But sample code does not work
consistently with higher concurrency (> 3). We captured the stack trace
of tomcat process which is attached here. Couple of tomcat threads are
waiting to acquire semaphore for socket write operation. When write
operation is stuck servlet is not able to push any data to client and
client is also stuck waiting for more data. I don’t see any
error/exception at the client/server.
That looks / sounds like there is a code path - probably an error
condition ? - where the semaphore isn't released.
The other possibility is related to the HTTP/2 flow control windows. If
something goes wrong with the management of the flow control window for
the connection it would block everything.
streamReadTimeout and streamWriteTimeout are configured as -1 so they
are infinitely waiting for the write semaphore.
That is generally a bad idea. By all means set it high but an infinite
timeout is going to cause problems - particularly if clients just drop
off the network.
Outcome of this is client is able to receive only partial data from
server and at some point server stuck to send any more data.
We also tried IOUtils file transfer related APIs still it didn't help. I
have also tried with Async non-blocking IO but the observations are same.
Generally, the simpler you keep the test case, the easier it is for us
to work with. Non-async and no external IO libraries is better.
Our actual requirement is very similar where java based http client
would request bulk data concurrently from server and server should push
that without any trouble. But, it is not limited to files only. Server
can push serialized java bulk objects over the stream concurrently.
The content type should not make any difference to Tomcat. Static files
vs dynamic content would make a difference.
Note that sample code works fine most of the time if I enable HTTP/2
logs either in client or tomcat. So I would suggest not to turn on
HTTP/2 debug logs to conclude anything.
That suggests a timing issue of some sort.
HTTP/2 is significantly more complex than HTTP/1.1 because you have
multiple independent application threads all trying to write to the same
socket and Tomcat has to track and allocate flow control window
allocations both for individual streams and the overall connection.
Following components are used in sample code for the test
1. Client - Java 11.0.10 httpclient - (client\Client.java)
2. Server - Tomcat 9.0.46
3. Servlet - AsyncServlet - (server\Server.java)
4. Operating system - Windows 10
5. Machine specifications – 32GB RAM and 500GB open space.
5. Latency - None, client and server are running on same machine
6. Set of files - You can use any random files whose sizes are between
1GB-5GB to reproduce the issue.
Refer attachment for
1. Client side code
2. Server side servlet
3. server.xml
4. Tomcat Stacktrace
5. Tomcat server logs
Thanks. That is all useful information.
Could you please go through sample code along with server.xml. Here are
my few questions
1. Why HTTP/2 is failing for such use case where large files are
concurrently pushing to the client. I believe this is a very common use
case must have be assessed earlier.
At this point, no idea. We'll need to see if we can recreate the problem
you describe first. Then we can try and identify root causes.
2. Connector configuration in server.xml is correct for HTTP/2 ?
See previous comments regarding timeouts.
Disabling the overhead protection shouldn't be necessary but often is
and looks to have been done correctly. The Tomcat releases scheduled for
July have some significant improvements in this area.
3. Does AsyncServlet is written properly for such type of use case ? If
not, then please suggest correct way to manage it.
Looks OK but is entirely pointless. It adds unneeded complexity. All you
are doing is adding a context switch from original thread used for
doPost() to a new thread.
4. If you are already aware of similar problem, can you point out any
alternatives or recommendations.
Nothing comes to mind and a quick scan of the changelog since 9.0.46
doesn't highlight anything that might be relevant.
I have one additional question at this point. How easy is this issue to
reproduce? Does it happen every time? In 10% of requests? 1% ?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
