Mark,
Never mind the last message, I thought it was working, but I had
looked at my test balancer config and forgot I had left the
application config using http instead of h2. So it's still busted. I
am running out of time on my end to spend on this at the moment.
Since it seems to be a Windows-specific issue, it will take me a lot
longer to set up a way to reproduce it since I don't have the
flexibility of spinning up docker images. I will be moving back to
mod_jk for now since I know that works. If I can provide you anything
else as far as logs from our environment that might point to what's
going on without me having to create a way to reproduce it for a bug
report I'd be happy to do so, but for now, I have to get back to
working on other things on my plate.
--
Thanks,
Dan
On Wed, Jul 12, 2023 at 10:56 PM Dan McLaughlin <d...@danshome.net> wrote:
>
> Hey Mark,
>
> I found a workaround/fix. On the Tomcat Connector, instead of using
> protocol=HTTP/1.1, I changed it to
> protocol="org.apache.coyote.http11.Http11Nio2Protocol," I haven't had
> a single failure since. Not only that, but our application response
> times are noticeably faster.
>
> --
>
> Thanks,
>
> Dan
> On Wed, Jul 12, 2023 at 9:58 PM Dan McLaughlin <d...@danshome.net> wrote:
> >
> > Well, the deeper I get into the problem, the more complicated it gets. I
> > thought I was onto something, thinking the size of the JSON might have
> > something to do with it, so I created a Python script to call curl POSTs
> > with increasingly larger JSON thinking I would eventually hit some size
> > limit, but what I'm seeing is that it seems to fail less with smaller JSON
> > files, although it will fail on just about any size. I even changed the
> > Python script to retry failed POSTs, which will work on over half the
> > second or third attempt.
> >
> > So I decided to try to create a maven project to build the test war, then
> > start a Tomcat and Apache docker image..and I can't reproduce the issue.
> >
> > My worst fear was that I was dealing with something potentially
> > Windows-specific, back to the drawing board.
> >
> > --
> >
> > Thanks,
> >
> > Dan
> >
> > On Wed, Jul 12, 2023 at 4:05 PM Dan McLaughlin <d...@danshome.net> wrote:
> >>
> >> Mark,
> >>
> >> I'm working on a test case. I've built a simple spring boot war with a
> >> rest API "jsonInput" that accepts any JSON and responds with
> >> {"message":"OK"}. What I've determined so far is that it only happens when
> >> you are proxying the request through Apache using mod_proxy_http2, and the
> >> size of the JSON that you are sending has something to do with the
> >> problem. I can send a large JSON or a small one directly to Tomcat, which
> >> works. If I send a small JSON through mod_proxy_http2, it also works, but
> >> if I send the JSON that our client apps are sending, which is quite large,
> >> it fails.
> >>
> >> Before I spend more time on this test case, can you think of any setting
> >> in Tomcat or mod_proxy_http2 that might cause the POST of the larger JSON
> >> to fail?
> >>
> >> --
> >>
> >> Thanks,
> >>
> >> Dan
> >>
> >> On Wed, Jul 12, 2023 at 2:36 PM Mark Thomas <ma...@apache.org> wrote:
> >>>
> >>>
> >>> 12 Jul 2023 13:40:18 Dan McLaughlin <d...@danshome.net.INVALID>:
> >>>
> >>> > I can confirm that if I switch h2 to http, everything works as
> >>> > expected, change it back to h2 or h2c, and it breaks.
> >>>
> >>> That makes me think it is an h2 bug in Tomcat.
> >>>
> >>> > Mark, Please let me know if the http2 logs weren't enough to tell you
> >>> > what's happening; if not, I'll work on creating a simple standalone
> >>> > reproduction using docker.
> >>>
> >>> I've looked through those logs and don't see anything. Enabling debug
> >>> logs for org.apache.tomcat.util.net might help but a reproducible test
> >>> case is probably the easiest for us to work with.
> >>>
> >>> If you can avoid using docker that helps as it is one less thing for us
> >>> to unpick when digging for the root cause but we can work with a docker
> >>> based reproducible test case if needed.
> >>>
> >>> Mark
> >>>
> >>> >
> >>> > --
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Dan
> >>> >
> >>> > On Wed, Jul 12, 2023 at 6:00 AM Dan McLaughlin <d...@danshome.net>
> >>> > wrote:
> >>> >>
> >>> >> Hi Mark,
> >>> >>
> >>> >> I already provided the output from org.apache.coyote.http2.level =
> >>> >> FINE in the very first post to this thread. I didn't include
> >>> >> everything because all the header information includes things I don't
> >>> >> necessarily want to post publicly and because it would take a while
> >>> >> for me to obfuscate. I will see if it's reproducible with a curl
> >>> >> command and if I can reproduce it in a standalone docker image.
> >>> >>
> >>> >> I will also try with mod_proxy_http, as suggested by Chris.
> >>> >>
> >>> >> Let me know if there is more logging I truncated that you need to see
> >>> >> that might tell you where the problem is; if I can provide it, I will.
> >>> >>
> >>> >> --
> >>> >>
> >>> >> Thanks,
> >>> >>
> >>> >> Dan
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Wed, Jul 12, 2023 at 3:34 AM Mark Thomas <ma...@apache.org> wrote:
> >>> >>>
> >>> >>> On 11/07/2023 19:10, Dan McLaughlin wrote:
> >>> >>>> One other note, is I can switch to h2c, and it still fails, and a
> >>> >>>> packet
> >>> >>>> capture shows the entire JSON is delivered to Tomcat, and when I put
> >>> >>>> the
> >>> >>>> JSON from the packet inspection together (Packets 10199 --> 10208)
> >>> >>>> and
> >>> >>>> compare it to what the browser says was sent, they are identical.
> >>> >>>> There are
> >>> >>>> no signs of TCP retransmissions or other things you might expect to
> >>> >>>> see if
> >>> >>>> there was a networking related issue.
> >>> >>>
> >>> >>> Hi Dan,
> >>> >>>
> >>> >>> This looks like a possible Tomcat bug to me.
> >>> >>>
> >>> >>> To debug futher I'd suggest the following:
> >>> >>>
> >>> >>> Enable http2 debug logging by adding the following to
> >>> >>> $CATALINA_BASE/conf/logging.properties
> >>> >>>
> >>> >>> org.apache.coyote.http2.level = FINE
> >>> >>>
> >>> >>> (that line should already be there, it just needs to be uncommented).
> >>> >>>
> >>> >>> If you can provide a curl command or similar that triggers this issue
> >>> >>> then please feel free to open a Bugzilla issue and attached the
> >>> >>> script
> >>> >>> and any relevant configuration snippets for httpd etc and we can try
> >>> >>> and
> >>> >>> reproduce it.
> >>> >>>
> >>> >>> Thanks,
> >>> >>>
> >>> >>> Mark
> >>> >>>
> >>> >>> ---------------------------------------------------------------------
> >>> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>> >>>
> >>> >
> >>> > --
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > *NOTICE:* This e-mail message and all attachments transmitted with
> >>> > it are for the sole use of the intended recipient(s) and may contain
> >>> > confidential and privileged information. Any unauthorized review, use,
> >>> > disclosure, or distribution is strictly prohibited. The contents of
> >>> > this
> >>> > e-mail are confidential and may be subject to work product privileges.
> >>> > If
> >>> > you are not the intended recipient, please contact the sender by reply
> >>> > e-mail and destroy all copies of the original message.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > ---------------------------------------------------------------------
> >>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org> For
> >>> > additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
--
*NOTICE:* This e-mail message and all attachments transmitted with
it are for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure, or distribution is strictly prohibited. The contents of this
e-mail are confidential and may be subject to work product privileges. If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
