Embedded Tomcat 9.0.43 : WINDOW_UPDATE not sent when receiving http2 requests over unknown url

[Arshiya Shariff]

Hi All,
We are using embedded tomcat version 9.0.43 in our application to transport 
http/2 packets between 2 systems (h2c connection). All parameters used are the 
tomcat defaults.

We are facing the below issue :

  1.  Tomcat is not sending WINDOW_UPDATE when a request(payload > 65K) is 
landed on an Unknown URL(which is not deployed as a servlet and not having  a 
servlet mapping). The same is working when request landed on an known URL(which 
is deployed as a servlet and has an servlet mapping).
  2.  Tomcat is listening on 1080 port, client is sending packet size of > 
65KB, but here tomcat receives only 65KB  and not receiving full DATA, later 
tomcat sends RST_STREAM, further no WINDOW_UPDATE, after that client is not 
able to send DATA frames as there is no WINDOW_UPDATE from tomcat.


Kindly let us know whether it is a bug or not.

Working case :
Client is sending to an correct URL of size/payload > 65KB. Tomcat is sending 
WINDOW_UPDATE. 200OK from application, which is working properly. Later client 
is able to send requests along with DATA also, and tomcat is receiving those

I have placed the PCAPs for the Working(correct url) and Not Working Case 
(unknown url) in the below path:
  
https://drive.google.com/drive/folders/1IN4XVQe2cvGIIQbHmAu-XLeWhjb-rykr?usp=sharing


Thanks and Regards
Arshiya Shariff

GOAWAY sent on 404 with large payload in http2

[Doug Whitfield]

Hi folks,

It is unclear if this is a Tomcat issue, a protocol issue, or something else. I 
would like some help figuring out if it is a Tomcat issue and then resolving 
the issue if it is. We have seen this issue in Tomcat 9.0.38 through 9.0.43.

For a handful of requests, Tomcat sends GOAWAY with below reason :
GOAWAY with FRAME_SIZE_ERROR : The payload is [2105376] bytes long but the 
maximum frame size is [16384]
The payload size here from the user point of view is around 55 KB, but we’ve 
tested similar payload sizes with similar results.



Steps to reproduce:
1. client sends an request of size 90KB to an unknown URL (means 
servlet-context was not deployed) towards tomcat.
2. Tomcat sends 404 page not found which is expected.
3. Tomcats sends RST_STREAM with CANCEL as reason.
4. Tomcat doesn't send WINDOW_UPDATE after this.
5. Client was not able to send further DATA frames towards Tomcat.
6. Client is able to send only HEADERS (to an deployed URL), but no DATA, all 
requests on that connection fails.
The above flow works properly when client sends an request to an correct 
deployed URL. The obvious response is to just use the correct URLs in 
applications, but we want to make sure this is not a security vulnerability.

All properties are the tomcat defaults.
Here are the connector details:
Connector connector = new Connector();
connector.setPort(1080);
Http2Protocol http2Protocol = new Http2Protocol();
connector.addUpgradeProtocol( http2Protocol );
tomcat.setConnector(connector);


Windows 10 (but also reproduced on Red Hat 7.4)
Processor: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz 1.90GHz
RAM:16 GB
System Type : 64 bit

How we reproduced:
With JMeter as simulation client, on configuring 700 threads (700 connections) 
to connect towards Tomcat Server 9.0.43 embedded in the application and on 
sending 20 requests per second with payload of 55KB (same request with just one 
json value sent uniquely via the Random number generator) and on running the 
test in an infinite loop , few requests are getting timed out . On analyzing 
the PCAP for the requests that timed-out we see that tomcat sends GOAWAY with  
PROTOCOL ERROR / FRAME_SIZE_ERROR .

Client:
JMeter 5.3 with additional HTTP2 sampler and Xmx 5g.
No of threads: 700
Ramp-up period: 10 seconds
Loop: Infinite
Payload size: around 55KB
Constant Throughput Timer added to limit the tps to 20.
Random Variable Generator added to the JSON request to uniquely identify for 
which request the exception is printed and to map it in the PCAP collected .
Response Timeout : 5000 ms

SERVER:
The input requests are processed asynchronously with 40 threads.




Are there any logs I should look at, and at what log level? There wasn’t 
anything obviously useful at FINER.

Please let me know if there is any additional information that would be useful.


Best Regards,
--

Doug Whitfield | Enterprise Architect, 
OpenLogic
Perforce 
Software
Visit us on: 
LinkedIn
 | 
Twitter
 | 
Facebook
 | 
YouTube




This e-mail may contain information that is privileged or confidential. If you 
are not the intended recipient, please delete the e-mail and any attachments 
and notify us immediately.

CloseNowException: This stream is not writable

[DevNull]

Apache Tomcat Version 9.0.43


Hello

In an application showing hundreds of thumbnail images at the same time, 
about a hundred thumbnails are loaded but the rest are not. Some posts 
claim it may be due to the http2 connection closing due to too much 
overhead.


I've tried to set overheadDataThreshold="0"  in server.xml but I see no 
change in number of loaded images.


Is there any other configuration I should try?


I've tried running the same code in Jetty 10.0.1 and got it working by 
increasing the http2 rate control limit.



Please see stacktrace and server.xml below.

Server side, I get exceptions for each thumbnail that is not loaded. 
From catalina.log:


09-Mar-2021 05:51:32.635 FINE [https-jsse-nio-443-exec-28] 
org.apache.coyote.AbstractProcessor.setErrorState Error state 
[CLOSE_NOW] reported while processing request
    org.apache.coyote.CloseNowException: 
org.apache.coyote.http2.StreamException: Connection [{0}], Stream [{1}], 
This stream is not writable
    at 
org.apache.coyote.http2.Stream$StreamOutputBuffer.end(Stream.java:941)
    at 
org.apache.coyote.http2.Http2OutputBuffer.end(Http2OutputBuffer.java:71)
    at 
org.apache.coyote.http2.StreamProcessor.finishResponse(StreamProcessor.java:209)
    at 
org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:389)
    at 
org.apache.coyote.http2.StreamProcessor.service(StreamProcessor.java:429)
    at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at 
org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:74)
    at 
org.apache.coyote.http2.StreamRunnable.run(StreamRunnable.java:35)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

    at java.base/java.lang.Thread.run(Thread.java:835)
    Caused by: org.apache.coyote.http2.StreamException: Connection 
[{0}], Stream [{1}], This stream is not writable
    at 
org.apache.coyote.http2.Stream.doStreamCancel(Stream.java:247)
    at 
org.apache.coyote.http2.Http2UpgradeHandler.reserveWindowSize(Http2UpgradeHandler.java:856)
    at 
org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:900)
    at 
org.apache.coyote.http2.Stream$StreamOutputBuffer.doWrite(Stream.java:819)
    at 
org.apache.coyote.http2.Http2OutputBuffer.doWrite(Http2OutputBuffer.java:59)

    at org.apache.coyote.Response.doWrite(Response.java:601)
    at 
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339)
    at 
org.apache.catalina.connector.OutputBuffer.appendByteArray(OutputBuffer.java:746)
    at 
org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:675)
    at 
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386)
    at 
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364)
    at 
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)
    at 
java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
    at 
skintaby.imgar.server.web.ImageArchiveServlet.getBinary(ImageArchiveServlet.java:460)
    at 
skintaby.imgar.server.web.ImageArchiveServlet.doGet(ImageArchiveServlet.java:127)
    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at 
org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:71)
    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at 

Re: Using threadlocals on async requests

[Christopher Schultz]

Thiago,

On 3/9/21 08:17, Thiago Veronezi wrote:

I have a java agent that has its own transaction feature and it does it by
using Threadlocals. The agent transactions feature seems to work ok with
Tomcat async requests.

Does anyone know it's fine to use threadlocals on async requests?


It's definitely *not* fine to use ThreadLocal with asynchronous 
requests. If it's working for you, its only due to some very lucky 
circumstances.


If you need to store information between invocations of IO callbacks, 
you will need to store that information in the request/response objects.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Fwd: Call for Presentations for ApacheCon 2021 now open

[Christopher Schultz]

All,

FYI

I'd love to see some new presentations this year. For anyone who has 
been considering doing a talk for ApacheCon, now might be a good tie 
since the conference will (again) be virtual this year. No need to take 
(a lot of) time off work, or advocate with your employer for travel 
expenses, etc.


Please email this list if you have any questions.

Thanks,
-chris

 Forwarded Message 
Subject: Call for Presentations for ApacheCon 2021 now open
Date: Mon, 8 Mar 2021 08:40:14 -0500
From: Rich Bowen 
Organization: The Apache Software Foundation
To: committ...@apache.org

The ApacheCon Planners and the Apache Software Foundation are pleased to 
announce that ApacheCon@Home will be held online, September 21-23, 2021. 
Once again, we’ll be featuring content from dozens of our projects, as 
well as content about our community, how Apache works, business models 
around Apache software, the legal aspects of open source, and many other 
topics.


Last year’s virtual ApacheCon@Home event was a big success, with 5,745 
registrants from more than 150 countries, spanning every time zone, with 
the virtual format delivering content to attendees who would never have 
attended an in-person ApacheCon (83% of post-event poll responders in 
2020 indicated this was their first ApacheCon ever)!


Given the great participation and excitement for last year’s event, we 
are announcing the Call for Presentations is now open to presenters from 
around the world until May 1st. Talks can be focused on the topics 
above, as well as any of our amazing projects. Submit your talks today!


https://acah2021.jamhosted.net/

We look forward to reviewing your contribution to one of the most 
popular open source software events in the world!



--
Rich Bowen, VP Conferences
The Apache Software Foundation
https://apachecon.com/
@apachecon

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Using threadlocals on async requests

[Thiago Veronezi]

Hi community,

I have a java agent that has its own transaction feature and it does it by
using Threadlocals. The agent transactions feature seems to work ok with
Tomcat async requests.

Does anyone know it's fine to use threadlocals on async requests?

Thanks!
Thiago.