Hi, We've deployed Apache 2.2 as a reverse ssl proxy in front of Jetty-6.1.11. The number of users has increaded in the last couple of weeks and we are now seeing proxy errors in the Apache error log 3-5 times daily:
[Wed Jul 29 10:45:40 2009] [error] [client xxx.xxx.xxx.xxx] proxy: error reading status line from remote server 172.16.101.156, referer: https://xxx.xxxxx.com/app/xxx [Wed Jul 29 10:45:40 2009] [error] [client xxx.xxx.xxx.xxx] proxy: Error reading from remote server returned by /xxx, referer:https://xxx.xxxxx.com/app/xxx The request in question doesn't appear in Jetty's request log. We captured the TCP stream using Wireshark and found that the request is forwarded by Apache but coincides with a FIN,ACK from Jetty to close the connection because 30 seconds have passed without activity. Jetty responds to the request with a RST, but Apache continues to use the connection. Here is the entire connection from Jetty's point of view: No. Time Source Destination Dest Port Protocol Info 21379 2009-07-29 10:44:40.685505 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=4182231540 TSER=0 WS=7 21380 2009-07-29 10:44:40.685517 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=4182015031 TSER=4182231540 WS=7 21381 2009-07-29 10:44:40.685624 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=4182231541 TSER=4182015031 21382 2009-07-29 10:44:40.685868 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1 (application/x-www-form-urlencoded) 21383 2009-07-29 10:44:40.685876 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [ACK] Seq=1 Ack=1173 Win=8192 Len=0 TSV=4182015031 TSER=4182231541 21384 2009-07-29 10:44:40.736972 172.16.101.156 172.16.101.155 47295 HTTP HTTP/1.1 200 OK [Unreassembled Packet [incorrect TCP checksum]] 21385 2009-07-29 10:44:40.736987 172.16.101.156 172.16.101.155 47295 HTTP Continuation or non-HTTP traffic 21386 2009-07-29 10:44:40.737323 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1173 Ack=1449 Win=8832 Len=0 TSV=4182231592 TSER=4182015082 21387 2009-07-29 10:44:40.737343 172.16.101.156 172.16.101.155 47295 HTTP Continuation or non-HTTP traffic 21388 2009-07-29 10:44:40.737579 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1173 Ack=2897 Win=11648 Len=0 TSV=4182231592 TSER=4182015082 21389 2009-07-29 10:44:40.737584 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1173 Ack=4345 Win=14592 Len=0 TSV=4182231592 TSER=4182015082 21390 2009-07-29 10:44:40.737605 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1173 Ack=4634 Win=17536 Len=0 TSV=4182231593 TSER=4182015083 21391 2009-07-29 10:44:40.738115 172.16.101.156 172.16.101.155 47295 HTTP Continuation or non-HTTP traffic 21392 2009-07-29 10:44:40.738210 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=1173 Ack=4639 Win=17536 Len=0 TSV=4182231593 TSER=4182015084 21519 2009-07-29 10:44:48.710869 172.16.101.155 172.16.101.156 8080 HTTP GET /xxx/xxxx HTTP/1.1 21520 2009-07-29 10:44:48.710876 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [ACK] Seq=4639 Ack=2081 Win=10496 Len=0 TSV=4182023056 TSER=4182239565 21521 2009-07-29 10:44:48.714462 172.16.101.156 172.16.101.155 47295 HTTP HTTP/1.1 304 21522 2009-07-29 10:44:48.714580 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=2081 Ack=4716 Win=17536 Len=0 TSV=4182239569 TSER=4182023060 21578 2009-07-29 10:45:09.878681 172.16.101.155 172.16.101.156 8080 TCP [TCP segment of a reassembled PDU] 21579 2009-07-29 10:45:09.878794 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx.xxxx HTTP/1.1 (application/x-www-form-urlencoded) 21580 2009-07-29 10:45:09.878803 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [ACK] Seq=4716 Ack=4862 Win=16384 Len=0 TSV=4182044223 TSER=4182260733 21581 2009-07-29 10:45:11.237149 172.16.101.156 172.16.101.155 47295 HTTP HTTP/1.1 302 Found 21582 2009-07-29 10:45:11.237304 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [ACK] Seq=4862 Ack=5013 Win=20352 Len=0 TSV=4182262091 TSER=4182045582 21701 2009-07-29 10:45:40.877019 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [FIN, ACK] Seq=5013 Ack=4862 Win=16384 Len=0 TSV=4182075221 TSER=4182262091 21702 2009-07-29 10:45:40.880273 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx.xxxx HTTP/1.1 (application/x-www-form-urlencoded) 21703 2009-07-29 10:45:40.880305 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [RST] Seq=5014 Win=0 Len=0 21704 2009-07-29 10:45:40.880312 172.16.101.155 172.16.101.156 8080 TCP 47295 > http-alt [FIN, ACK] Seq=6107 Ack=5014 Win=20352 Len=0 TSV=4182291733 TSER=4182075221 21705 2009-07-29 10:45:40.880317 172.16.101.156 172.16.101.155 47295 TCP http-alt > 47295 [RST] Seq=5014 Win=0 Len=0 and here's another connection with the same error but a slightly different conclusion: No. Time Source Destination Dest Port Protocol Info 259350 2009-07-28 12:25:34.242754 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [SYN] Seq=0 Win=5840 Len=0 MSS=1460 TSV=4101887388 TSER=0 WS=7 259351 2009-07-28 12:25:34.242767 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=4101671152 TSER=4101887388 WS=7 259352 2009-07-28 12:25:34.242856 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=4101887388 TSER=4101671152 ... 306028 2009-07-28 12:32:28.458704 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1 (application/x-www-form-urlencoded) 306038 2009-07-28 12:32:28.471810 172.16.101.156 172.16.101.155 42530 HTTP HTTP/1.1 200 OK (application/json) 306039 2009-07-28 12:32:28.471961 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=34931 Ack=237709 Win=81280 Len=0 TSV=4102301605 TSER=4102085368 306040 2009-07-28 12:32:28.472982 172.16.101.156 172.16.101.155 42530 HTTP Continuation or non-HTTP traffic 306041 2009-07-28 12:32:28.473078 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=34931 Ack=237714 Win=81280 Len=0 TSV=4102301606 TSER=4102085369 307087 2009-07-28 12:32:51.321792 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1 (application/x-www-form-urlencoded) 307097 2009-07-28 12:32:51.334466 172.16.101.156 172.16.101.155 42530 HTTP HTTP/1.1 200 OK (application/json) 307098 2009-07-28 12:32:51.334615 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=36015 Ack=237956 Win=81280 Len=0 TSV=4102324467 TSER=4102108230 307099 2009-07-28 12:32:51.335616 172.16.101.156 172.16.101.155 42530 HTTP Continuation or non-HTTP traffic 307100 2009-07-28 12:32:51.335713 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=36015 Ack=237961 Win=81280 Len=0 TSV=4102324468 TSER=4102108231 307512 2009-07-28 12:33:09.577293 172.16.101.155 172.16.101.156 8080 HTTP GET /xxx/xxxxx HTTP/1.1 307564 2009-07-28 12:33:09.616699 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [ACK] Seq=237961 Ack=36892 Win=64128 Len=0 TSV=4102126512 TSER=4102342709 307582 2009-07-28 12:33:09.626212 172.16.101.156 172.16.101.155 42530 HTTP HTTP/1.1 200 OK 307583 2009-07-28 12:33:09.626369 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=36892 Ack=238260 Win=81280 Len=0 TSV=4102342758 TSER=4102126521 307584 2009-07-28 12:33:09.626425 172.16.101.156 172.16.101.155 42530 HTTP Continuation or non-HTTP traffic 307585 2009-07-28 12:33:09.626428 172.16.101.156 172.16.101.155 42530 HTTP Continuation or non-HTTP traffic 307587 2009-07-28 12:33:09.626768 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=36892 Ack=239708 Win=81280 Len=0 TSV=4102342759 TSER=4102126521 307588 2009-07-28 12:33:09.626842 172.16.101.156 172.16.101.155 42530 HTTP Continuation or non-HTTP traffic 307589 2009-07-28 12:33:09.627013 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=36892 Ack=240880 Win=81280 Len=0 TSV=4102342759 TSER=4102126521 307590 2009-07-28 12:33:09.627019 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=36892 Ack=240885 Win=81280 Len=0 TSV=4102342759 TSER=4102126522 309125 2009-07-28 12:33:30.245862 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1 (application/x-www-form-urlencoded) 309126 2009-07-28 12:33:30.245870 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [ACK] Seq=240885 Ack=37985 Win=64128 Len=0 TSV=4102147140 TSER=4102363377 309137 2009-07-28 12:33:30.258721 172.16.101.156 172.16.101.155 42530 HTTP HTTP/1.1 200 OK (application/json) 309138 2009-07-28 12:33:30.258873 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=37985 Ack=241140 Win=81280 Len=0 TSV=4102363390 TSER=4102147153 309139 2009-07-28 12:33:30.259867 172.16.101.156 172.16.101.155 42530 HTTP Continuation or non-HTTP traffic 309140 2009-07-28 12:33:30.259961 172.16.101.155 172.16.101.156 8080 TCP 42530 > http-alt [ACK] Seq=37985 Ack=241145 Win=81280 Len=0 TSV=4102363391 TSER=4102147154 310858 2009-07-28 12:34:00.245039 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [FIN, ACK] Seq=241145 Ack=37985 Win=64128 Len=0 TSV=4102177138 TSER=4102363391 310860 2009-07-28 12:34:00.259377 172.16.101.155 172.16.101.156 8080 TCP [TCP segment of a reassembled PDU] 310861 2009-07-28 12:34:00.259394 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [RST] Seq=241146 Win=0 Len=0 310862 2009-07-28 12:34:00.259485 172.16.101.155 172.16.101.156 8080 TCP [TCP segment of a reassembled PDU] 310863 2009-07-28 12:34:00.259491 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [RST] Seq=241146 Win=0 Len=0 310864 2009-07-28 12:34:00.259628 172.16.101.155 172.16.101.156 8080 TCP [TCP segment of a reassembled PDU] 310865 2009-07-28 12:34:00.259632 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [RST] Seq=241146 Win=0 Len=0 310866 2009-07-28 12:34:00.259634 172.16.101.155 172.16.101.156 8080 HTTP POST /xxx/xxxxx HTTP/1.1 (application/x-www-form-urlencoded) 310867 2009-07-28 12:34:00.259639 172.16.101.156 172.16.101.155 42530 TCP http-alt > 42530 [RST] Seq=241146 Win=0 Len=0 Does anyone know what's going on here? Regards, thanks, Patrick --------------------------------------------------------------------- The official User-To-User support forum of the Apache HTTP Server Project. See <URL:http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org " from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
