-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 All,
On 6/29/20 12:22, Christopher Schultz wrote: > I believe I have determined the cause (or at least the fix) for > this: > > Despite the mod_proxy_ajp documentation, it is using packets > larger than 8192 bytes. > > Setting this on the Tomcat <Connector> resolves the problem: > > packetSize="65536" > > I don't see anywhere you can set the maximum packet size on > mod_proxy_ajp so I think this is the only recourse. That's because this is a property of mod_proxy and not mod_proxy_ajp. ProxyIOBufferSize [1] > Is anyone familiar enough with the mod_proxy_ajp code to know how > to look for this? > > It was trivial to reproduce: just send a large POST message > through mod_proxy_ajp to Tomcat. Evidently, I've been down this road before and I had *intentionally* set the high-water mark of ProxyIOBufferSize to 65535. So this was entirely self-inflicted. Thanks, - -chris [1] https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxyiobuffersi ze > On 6/25/20 10:28, Christopher Schultz wrote: >> All, > >> This issue is apparently trivially reproducible in my dev >> environment. > >> Do I have to get a protocol-trace to get any more helpful >> information? > >> Thanks, -chris > >> On 6/24/20 10:46, Christopher Schultz wrote: >>> All, > >>> On 6/24/20 10:29, Christopher Schultz wrote: >>>> All, > >>>> I'm slowly switching from mod_jk to mod_proxy_ajp and I have >>>> a development environment where I'm getting Bad Gateway >>>> responses sent to clients along with this exception in my >>>> Tomcat log file: > >>>> java.lang.IllegalArgumentException: Header message of length >>>> [8,194] received but the packetSize is only [8,192] at >>>> org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:68 5 > >>>> ) > >>>> >>>> > >>> at >>>> org.apache.coyote.ajp.AjpProcessor.receive(AjpProcessor.java:626) >>>> >>>> > >>>> >>>> > at >>>> org.apache.coyote.ajp.AjpProcessor.refillReadBuffer(AjpProcessor.ja v > >>>> a > >>>> > : > >>>> >> 73 > > >>> 4) >>>> at >>>> org.apache.coyote.ajp.AjpProcessor$SocketInputBuffer.doRead(AjpProc e > >>>> s > >>>> > s > >>>> >> or > > >>> .java:1456) >>>> at org.apache.coyote.Request.doRead(Request.java:581) at >>>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer . > >>>> j > >>>> > a > >>>> >> va > > >>> :344) >>>> at >>>> org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputB u > >>>> f > >>>> > f > >>>> >> er > > >>> .java:663) >>>> at >>>> org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java : > >>>> 3 > >>>> > 5 > >>>> >> 8) > > >>> at >>>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStr e > >>>> a > >>>> > m > >>>> >> .j > > >>> ava:93) >>>> at >>>> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream. j > >>>> a > >>>> > v > >>>> >> a: > > >>> 53) >>>> at >>>> org.apache.commons.io.input.TeeInputStream.read(TeeInputStream.java : > >>>> 1 > >>>> > 0 > >>>> >> 6) > > >>> at java.io.FilterInputStream.read(FilterInputStream.java:83) >>>> at my.product.MacInputStream.read(MacInputStream.java:29) at >>>> java.io.FilterInputStream.read(FilterInputStream.java:83) at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$Rewindable I > >>>> n > >>>> > p > >>>> >> ut > > >>> Stream.read(XMLEntityManager.java:2890) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurre n > >>>> t > >>>> > E > >>>> >> nt > > >>> ity(XMLEntityManager.java:674) >>>> at >>>> com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determin e > >>>> D > >>>> > o > >>>> >> cV > > >>> ersion(XMLVersionDetector.java:148) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse ( > >>>> X > >>>> > M > >>>> >> L1 > > >>> 1Configuration.java:806) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse ( > >>>> X > >>>> > M > >>>> >> L1 > > >>> 1Configuration.java:771) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParse r > >>>> . > >>>> > j > >>>> >> av > > >>> a:141) >>>> at >>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse( A > >>>> b > >>>> > s > >>>> >> tr > > >>> actSAXParser.java:1213) >>>> at >>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser . > >>>> p > >>>> > a > >>>> >> rs > > >>> e(SAXParserImpl.java:643) >>>> at >>>> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXPars e > >>>> r > >>>> > I > >>>> >> mp > > >>> l.java:327) >>>> at javax.xml.parsers.SAXParser.parse(SAXParser.java:195) > >>>> This is a web service which is reading the request with a >>>> SAXParser. It's been running in production (and dev!) for >>>> years without any issues. It''s been running for a few >>>> months in development, now, with mod_proxy_ajp without any >>>> errors. > >>>> I know about the "max packet size" and the default is 8192 >>>> bytes. I haven't changed the default. Here's my <Connector> >>>> configuration: > >>>> <Connector port="8245" address="127.0.0.1" >>>> secretRequired="false" redirectPort="443" protocol="AJP/1.3" >>>> URIEncoding="UTF-8" executor="tomcatThreadPool" /> > >>>> Here's the configuration in httpd.conf: > >>>> <Proxy "balancer://my-api"> BalancerMember >>>> "ajp://localhost:8245" timeout=300 ping=5 ttl=60 </Proxy> > >>>> ProxyPass "/my-api/" "balancer://my-api/my-api/" >>>> ProxyPassReverse "/my-api/" "balancer://my-api/my-api/" > >>>> The documentation for mod_proxy_ajp[1] seems to indicate >>>> that the "Packet Size" for AJP is fixed at 8192 bytes: > >>>> " Packet Size > >>>> According to much of the code, the max packet size is 8 * >>>> 1024 bytes (8K). The actual length of the packet is encoded >>>> in the header. > >>>> Packet Headers > >>>> Packets sent from the server to the container begin with >>>> 0x1234. Packets sent from the container to the server begin >>>> with AB (that's the ASCII code for A followed by the ASCII >>>> code for B). After those first two bytes, there is an >>>> integer (encoded as above) with the length of the payload. >>>> Although this might suggest that the maximum payload could be >>>> as large as 2^16, in fact, *the code sets the maximum to be >>>> 8K*. " (emphasis mine) > >>>> Does anyone know under what circumstances mod_proxy_ajp >>>> might send more than 8192 bytes? It looks like mod_proxy_ajp >>>> doesn't have any way to set the max packet size like mod_jk >>>> does. > >>>> I should probably be able to set the max packet size on the >>>> Tomcat side to something higher than 8192 to catch this kind >>>> of thing... but it looks like it might be a bug in >>>> mod_proxy_ajp. > >>>> Versions are Apache httpd 2.4.25 (Debian) and Tomcat >>>> 8.5.trunk (8.5.55). mod_jk is not being used. > >>>> Any ideas? > >>>> -chris > >>>> [1] https://httpd.apache.org/docs/2.4/mod/mod_proxy_ajp.html > > >>> Some additional information: > >>> 1. The headers of the HTTP request seem to be arriving in a >>> correct packet before this error occurs. The headers are only >>> a few hundred bytes (~340) and the request line should be >>> relatively short (~50 bytes or so). Method is POST, protocol >>> is HTTP/1.1. > >>> 2. Apache httpd is terminating TLS. I have no configuration >>> for forwarding TLS information over to Tomcat, so I'm assuming >>> it's not being sent as part of the first packet. > >>> 3. Before I get the packet-too-large error, I get another >>> error: > >>> org.apache.catalina.connector.ClientAbortException: >>> java.io.IOException: Invalid message received with length [-1] >>> at >>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer. j > >>> a > >>> > va > > >> :348) >>> at >>> org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBu f > >>> f > >>> > er > > >> .java:663) >>> at >>> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:370) > >>> >>> > >> at >>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStre a > >>> m > >>> > .j > > >> ava:183) >>> at >>> org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.j a > >>> v > >>> > a: > > >> 75) >>> at >>> org.apache.commons.io.input.TeeInputStream.read(TeeInputStream.java: 1 > >>> 2 > >>> > 4) > > >> at java.io.FilterInputStream.read(FilterInputStream.java:133) >>> at my.product.MacInputStream.read(MacInputStream.java:49) at >>> java.io.FilterInputStream.read(FilterInputStream.java:107) at >>> my.product.XMLMessageProcessor.validate(XMLMessageProcessor.java:326 ) > >>> >>> > >> at com.chadis.servlet.APIServlet.doPost(APIServlet.java:291) >>> at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at >>> javax.servlet.http.HttpServlet.service(HttpServlet.java:741) > >>> I haven't changed any configuration, yet. But if the first >>> error is size=-1 then it's unlikely that the problem will be >>> solved by increasing my max packet size on the Tomcat end. > >>> I'm working with my client to see if we can reproduce this >>> problem reliably. I'm sure I can get more information once I >>> do that. > >>> -chris > > > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl76RyIACgkQHPApP6U8 pFhRbA//YNTn0vsQ5TKUUmkpCEaG99n6rm4aF2Tb7NbPoZqjuCQtNxmiKRp8Q+SP rIF3m1AN6TUaaXosGXxGVUoOLRbZESC5QAXW/oU36zL8ioAf+XUyMLfq5lJxzokQ lfyPNspYUKk37+CUr3ocVtaatgH2LX1Z81bMuY4LgU2eliNv9byUpe+bu+4hpEVj r5EJalctwKw34A3ffUNspIXfUNlg0NUuzODcFiWlhE3MKujg6dEXjTY0fKwbz3W2 iGSkfofD8BDHDdBjjF3/pBodwunqSTsv0R/R/fDotm+CC5i59DPmfJf1j+JM4XUr XeXZv4UTzVVScS7o3AaAELWnck1DUEG9eFC7Pv6hN3DbNgDqybu7tzVUH4/CCand 11GFzCFFqOTF7A+VbKY6D2zB206+nSbhJBXdOpJTJAHOqgFepItEHlLompoIIs7t uuBIA5uzGUwi/b/UTPZdzMRh3qCvkWlz42GMKcCoDF47PGnV2+VpYP2qRwfucaMX eY6kzq76Qbd0tCqr/O+Aqr0375afEPPLu10dlRu48uOcvqLWFlF1R7mAaSDWdRab kweKjyj9P0+c51+2MiSZuNqEb2ayu7+is1o58WBCL9eVkoWDfWuCbgHn4O3vMkxj ZRe8WITYJhXwutDU0scRWaCqE7W4r2IrFtPm2F8Q6IPhMnR7ySU= =tH2L -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org