AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello, the spec https://datatracker.ietf.org/doc/html/rfc7231#page-25 says in chapter 4.3.2: " The HEAD method is identical to GET except that the server MUST NOT send a message body in the response (i.e., the response terminates at the end of the header section)." Greetings, Thomas -Ursprüngliche Nachricht- Von: Stefan Mayr Gesendet: Sonntag, 13. Februar 2022 18:37 An: users@tomcat.apache.org Betreff: Re: mod_jk interference with ErrorDocument/Alias on HEAD request Hi, looking at the source code https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.0/mod_jk.c#L2954#L2973 I did some more testing: Variant 1: JkMount /demo/* ajp13_worker;use_server_errors=401 Variant 2: JkMount /demo/* ajp13_worker ignoring what variant 2 changes for regular request: reading the source comment my understanding is, that for both variants a HEAD request (by definition must have an empty response body) should let Apache httpd handle the error code. But the return code for jk_handler looks different: Variant 1: s.http_response_status Variant 2: r->status The response only seems correct for variant 1 - which is configured to let Apache httpd handle all responses for status codes >= 401. For variant 2 mod_jk seems to handle the response itself - contrary to what the comment explains. Am 12.02.2022 um 14:24 schrieb Stefan Mayr: > Hello Tomcat users, > > this week we were debugging a strange connection issue which I tracked > down to an interference between Apache httpd and mod_jk. > > For the full picture, the infrastructure setup contains > > 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. > 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat > mit AJP-Connector > > We have an application doing many different HEAD requests against an > application running in the Tomcat server. The requests contain an > Authorization header for Basic authentication. Expected response is a > HTTP 200 OK or HTTP 401 if this particular user is not allowed to > access that ressource. Because this is a HEAD request there must not > be a response body according to RFC 2616. > > If there is a response body in the response to the HEAD request our > loadbalancer does strange things: aborts the connection if the clients > uses HTTP/2, SSL errors if using HTTP/1.1. But this is an issue on its > own which we might have to solve with the vendor. > > Now comes the point where I need your help. We have a httpd > configuration with mod_jk which generates these invalid response > bodies on HEAD requests. I have a gut feeling this could be a bug with mod_jk. > > For demonstration purpose i created a minimal demo app which only is a > WEB-INF/web.xml xmlns="https://jakarta.ee/xml/ns/jakartaee"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee > https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd"; > version="5.0"> > > > Login > /* > > > manager > > > > manager > > > BASIC > > > > Then I place a JkMount in my Apache httpd configuration (+ minimal > worker.properties): > > JkMount /demo/* ajp13_worker > > Testing this with curl works like expected: > > root@1ae8973f1b6b:~# curl -I -v localhost/demo/ > * Trying 127.0.0.1:80... > * TCP_NODELAY set > * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ > HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > > > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 > HTTP/1.1 401 401 > < Date: Sat, 12 Feb 2022 12:57:33 GMT > Date: Sat, 12 Feb 2022 12:57:33 GMT > < Server: Apache/2.4.41 (Ubuntu) > Server: Apache/2.4.41 (Ubuntu) > < Cache-Control: private > Cache-Control: private > < WWW-Authenticate: Basic realm="Authentication required" > WWW-Authenticate: Basic realm="Authentication required" > < Content-Language: en > Content-Language: en > < Content-Type: text/html;charset=utf-8 > Content-Type: text/html;charset=utf-8 > > < > * Connection #0 to host localhost left intact > > But our default setup always includes custom error pages: > > Alias /error/ "/usr/share/apache2/error/" > ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var > > If both of those lines are added this results in a response body for > the HEAD request. > > root@1ae8973f1b6b:~# curl -I -v localhost/demo/ > * Trying 127.0.0.1:80... > * TCP_NODELAY set > * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ > HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > > > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 > HTTP/1.1 401 401 > < Date: Sat, 12 Feb 2022 12:56:27 GMT > Date: Sat, 12 Feb 2022 12:56:27 GMT > < Server
Re: mod_jk interference with ErrorDocument/Alias on HEAD request
Hi, looking at the source code https://github.com/apache/tomcat-connectors/blob/main/native/apache-2.0/mod_jk.c#L2954#L2973 I did some more testing: Variant 1: JkMount /demo/* ajp13_worker;use_server_errors=401 Variant 2: JkMount /demo/* ajp13_worker ignoring what variant 2 changes for regular request: reading the source comment my understanding is, that for both variants a HEAD request (by definition must have an empty response body) should let Apache httpd handle the error code. But the return code for jk_handler looks different: Variant 1: s.http_response_status Variant 2: r->status The response only seems correct for variant 1 - which is configured to let Apache httpd handle all responses for status codes >= 401. For variant 2 mod_jk seems to handle the response itself - contrary to what the comment explains. Am 12.02.2022 um 14:24 schrieb Stefan Mayr: Hello Tomcat users, this week we were debugging a strange connection issue which I tracked down to an interference between Apache httpd and mod_jk. For the full picture, the infrastructure setup contains 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat mit AJP-Connector We have an application doing many different HEAD requests against an application running in the Tomcat server. The requests contain an Authorization header for Basic authentication. Expected response is a HTTP 200 OK or HTTP 401 if this particular user is not allowed to access that ressource. Because this is a HEAD request there must not be a response body according to RFC 2616. If there is a response body in the response to the HEAD request our loadbalancer does strange things: aborts the connection if the clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an issue on its own which we might have to solve with the vendor. Now comes the point where I need your help. We have a httpd configuration with mod_jk which generates these invalid response bodies on HEAD requests. I have a gut feeling this could be a bug with mod_jk. For demonstration purpose i created a minimal demo app which only is a WEB-INF/web.xml https://jakarta.ee/xml/ns/jakartaee"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd"; version="5.0"> Login /* manager manager BASIC Then I place a JkMount in my Apache httpd configuration (+ minimal worker.properties): JkMount /demo/* ajp13_worker Testing this with curl works like expected: root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:57:33 GMT Date: Sat, 12 Feb 2022 12:57:33 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Connection #0 to host localhost left intact But our default setup always includes custom error pages: Alias /error/ "/usr/share/apache2/error/" ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var If both of those lines are added this results in a response body for the HEAD request. root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:56:27 GMT Date: Sat, 12 Feb 2022 12:56:27 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Excess found: excess = 589 url = /demo/ (zero-length body) * Connection #0 to host localhost left intact Checking with tcpdump on port 8009 we see the expected response without a body from the Tomcat AJP connector. The tcpdump von port 80 reveals httpd is adding the configured ErrorDocument as response body. If we comment out either the A
Re: mod_jk interference with ErrorDocument/Alias on HEAD request
Hi Thomas, Am 13.02.2022 um 17:27 schrieb Thomas Hoffmann (Speed4Trade GmbH): Hello, maybe you can try to set an environment variable which skips interpreting the content-length: https://tomcat.apache.org/connectors-doc/reference/apache.html#Advanced%20Environment%20Variables --> JK_IGNORE_CL To get more information, you can also set the logfile and log-level to debug: JkLogLevel (same reference page as above). Greetings, Thomas I added both options: Alias /error/ "/usr/share/apache2/error/" ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var SetEnv JK_IGNORE_CL JkLogLevel Debug JkMount /demo/* ajp13_worker This produces the following debug output: [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 005001 00 17 74 65 78 74 2F 68 74 6D 6C 3B 63 68 61 - ...text/html;cha [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 006072 73 65 74 3D 75 74 66 2D 38 00 A0 02 00 02 65 - rset=utf-8.e [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 00706E 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - n... [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_unmarshal_response::jk_ajp_common.c (736): (ajp13_worker) status = 401 [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_unmarshal_response::jk_ajp_common.c (743): Number of headers is = 4 [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13_worker) Header[0] [Cache-Control] = [private] [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13_worker) Header[1] [WWW-Authenticate] = [Basic realm="Authentication required"] [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13_worker) Header[2] [Content-Type] = [text/html;charset=utf-8] [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_unmarshal_response::jk_ajp_common.c (799): (ajp13_worker) Header[3] [Content-Language] = [en] [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1459): received from ajp13 pos=0 len=2 max=8192 [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_connection_tcp_get_message::jk_ajp_common.c (1459): 05 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_process_callback::jk_ajp_common.c (2132): (ajp13_worker) AJP13 protocol: Reuse is OK [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_reset_endpoint::jk_ajp_common.c (848): (ajp13_worker) resetting endpoint with socket 15 [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [debug] ajp_done::jk_ajp_common.c (3288): recycling connection pool for worker ajp13_worker and socket 15 [Sun Feb 13 16:37:02.591 2022] [212:140388647335680] [info] jk_handler::mod_jk.c (2968): No body with status=401 for worker=ajp13_worker [Sun Feb 13 16:37:02.592 2022] [212:140388647335680] [debug] jk_servlet_normalize::jk_util.c (2185): URI on entering jk_servlet_normalize: [/error/HTTP_UNAUTHORIZED.html.var] [Sun Feb 13 16:37:02.592 2022] [212:140388647335680] [debug] jk_servlet_normalize::jk_util.c (2279): URI on exiting jk_servlet_normalize: [/error/HTTP_UNAUTHORIZED.html.var] [Sun Feb 13 16:37:02.592 2022] [212:140388647335680] [debug] map_uri_to_worker_ext::jk_uri_worker_map.c (1168): Attempting to map URI '/error/HTTP_UNAUTHORIZED.html.var' from 1 maps [Sun Feb 13 16:37:02.592 2022] [212:140388647335680] [debug] find_match::jk_uri_worker_map.c (978): Attempting to map context URI '/demo/*=ajp13_worker' source 'JkMount' [Sun Feb 13 16:37:02.592 2022] [212:140388647335680] [debug] jk_map_to_storage::mod_jk.c (4058): no match for /error/HTTP_UNAUTHORIZED.html.var found The last lines are irritating. They don't change when I try to add a JkUnMount: JkMount /demo/* ajp13_worker JkUnMount /error/* ajp13_worker -Ursprüngliche Nachricht- Von: Stefan Mayr Gesendet: Samstag, 12. Februar 2022 14:24 An: Tomcat Users List Betreff: mod_jk interference with ErrorDocument/Alias on HEAD request Hello Tomcat users, this week we were debugging a strange connection issue which I tracked down to an interference between Apache httpd and mod_jk. For the full picture, the infrastructure setup contains 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat mit AJP-Connector We have an application doing many different HEAD requests against an application running in the Tomcat server. The requests contain an Authorization header for Basic authentication. Expected response is a HTTP 200 OK or HTTP 401 if this particular
AW: mod_jk interference with ErrorDocument/Alias on HEAD request
Hello, maybe you can try to set an environment variable which skips interpreting the content-length: https://tomcat.apache.org/connectors-doc/reference/apache.html#Advanced%20Environment%20Variables --> JK_IGNORE_CL To get more information, you can also set the logfile and log-level to debug: JkLogLevel (same reference page as above). Greetings, Thomas -Ursprüngliche Nachricht- Von: Stefan Mayr Gesendet: Samstag, 12. Februar 2022 14:24 An: Tomcat Users List Betreff: mod_jk interference with ErrorDocument/Alias on HEAD request Hello Tomcat users, this week we were debugging a strange connection issue which I tracked down to an interference between Apache httpd and mod_jk. For the full picture, the infrastructure setup contains 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat mit AJP-Connector We have an application doing many different HEAD requests against an application running in the Tomcat server. The requests contain an Authorization header for Basic authentication. Expected response is a HTTP 200 OK or HTTP 401 if this particular user is not allowed to access that ressource. Because this is a HEAD request there must not be a response body according to RFC 2616. If there is a response body in the response to the HEAD request our loadbalancer does strange things: aborts the connection if the clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an issue on its own which we might have to solve with the vendor. Now comes the point where I need your help. We have a httpd configuration with mod_jk which generates these invalid response bodies on HEAD requests. I have a gut feeling this could be a bug with mod_jk. For demonstration purpose i created a minimal demo app which only is a WEB-INF/web.xml https://jakarta.ee/xml/ns/jakartaee"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd"; version="5.0"> Login /* manager manager BASIC Then I place a JkMount in my Apache httpd configuration (+ minimal worker.properties): JkMount /demo/* ajp13_worker Testing this with curl works like expected: root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:57:33 GMT Date: Sat, 12 Feb 2022 12:57:33 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Connection #0 to host localhost left intact But our default setup always includes custom error pages: Alias /error/ "/usr/share/apache2/error/" ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var If both of those lines are added this results in a response body for the HEAD request. root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:56:27 GMT Date: Sat, 12 Feb 2022 12:56:27 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Excess found: excess = 589 url = /demo/ (zero-length body) * Connection #0 to host localhost left intact Checking with tcpdump on port 8009 we see the expected response without a body from the Tomcat AJP connector. The tcpdump von port 80 reveals httpd is adding the configured ErrorDocument as response body. If we comment out either the Alias or ErrorDocument directive the response is correct again. Doing similar tests with CGI/PHP applications always show the correct response without a response body. This only affects requests which use mod_jk. So far I could reproduce this on SLES12 SP5 and SLES15 SP3 running Apache httpd 2.4.51 and a self compile mod_jk 1.2.46 (same wit