On 2 March 2017 at 20:56:37, Christopher Faulet (cfau...@haproxy.com) wrote:
Le 02/03/2017 à 16:46, Jon Simpson a écrit : > Hi all, > > Further to my previous message to the list I can now reliably reproduce > the behaviour where certain requests using authentication headers > receive 503 responses from HAProxy 1.7.3. > > We’re making a request twice, once without basic auth (checking that > basic auth is prompted on the URL) and then a second, identical, request > but with the the basic auth Authorization header set. The request body > is textual file content, so these requests are larger than average > GET/POST requests (Content-Length > 3500). We don't see this problem > with smaller requests. > > The two requests happen on a single connection to HAProxy as we have > keepalive enabled. The first request is always sent to the backend > successfully and but the second request receives a 503 status from > HAProxy whenever the request size is over a Content-Length of 3500. Just > below this size the 503’s are sporadic and at very small body sizes it > doesn’t happen at all - the request is sent to the backend succesfully. > > There are no errors visible by catting the stats socket, and the second > request/503 response is not present in logs at all (using option httplog). > > We don't get this behaviour when keeping all other variables the same > (client, request body & HAProxy configuration) using HAProxy 1.6.11. > > Thanks in advance for any help anyone can provide. Hi, I've quickly tried to reproduce your bug without success. Please, share you configuration to be sure to do test in the same situation. It could also be helpful to have a tcpdump for data exchanged between your client and HAProxy and another one for data exchanged between HAProxy and your webserver. If you cannot do a tcpdump, you can use the trace filter by adding the following line in your frontend/listener section: filer trace Then run HAProxy in foreground. -- Christopher Faulet Hi Christopher, Thanks for your response. I’ve attached the requested configuration and HAProxy trace logs below (TCPdump appeared to be much noisier). I have an input file which reproduces the 503 case but not predictably, so I’ve included one log which generated the correct 401 response from HAProxy and one which returned a 503 response. The 503 response comes back every time with larger requests than this. I’ve stripped down the configuration to a minimum which reproduces the issue. The behaviour only seems to occur with proxy-protocol enabled (accept-proxy). I initially tested directly sending requests directly to HAProxy and that works without the 503 responses. We run this HAProxy configuration behind Amazon ELB - a TCP load balancer with proxy protocol enabled, where the issue occurs (and it occurred in my test environment once proxy protocol was enabled and connections went through the TCP load balancer). Thanks, Jon
1488550830.939713 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00000000 0x00000000] trace_attach : filter-type=frontend 1488550830.939713 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00000000 0x00000000] trace_stream_start 00000006:https-in.accept(0005)=0008 from [34.250.167.137:38280] 1488550830.939713 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034035 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550830.939713 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 00000006:https-in.clireq[0008:ffffffff]: POST /v1/expost/check HTTP/1.1 00000006:https-in.clihdr[0008:ffffffff]: Content-Length: 3632 00000006:https-in.clihdr[0008:ffffffff]: Content-Type: application/xml 00000006:https-in.clihdr[0008:ffffffff]: Host: [REDACTED] 00000006:https-in.clihdr[0008:ffffffff]: Connection: Keep-Alive 00000006:https-in.clihdr[0008:ffffffff]: User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034030 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=POST 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034030 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_HTTP_PROCESS_FE - step=PRE 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034020 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_HTTP_PROCESS_FE - step=POST 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034020 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_SWITCHING_RULES - step=PRE 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00034000 0x00000000] trace_attach : filter-type=backend 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034000 0x00000000] trace_stream_set_backend : backend=web-02 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00034000 0x00000000] trace_stream_set_backend : backend=web-02 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034940 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_SWITCHING_RULES - step=POST 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00034940 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (backend) 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034900 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=PRE 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00034900 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=PRE 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=POST 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=POST 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=PRE 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=PRE 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=POST 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=POST 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] trace_http_headers : channel=REQUEST - mode=HTTP (backend) 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] POST /v1/expost/check HTTP/1.1 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] Content-Length: 3632 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] Content-Type: application/xml 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] Host: [REDACTED] 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] X-Forwarded-For: 34.250.167.137 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] trace_http_headers : channel=REQUEST - mode=HTTP (backend) 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] POST /v1/expost/check HTTP/1.1 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] Content-Length: 3632 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] Content-Type: application/xml 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] Host: [REDACTED] 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x0002c000 0x00000000] X-Forwarded-For: 34.250.167.137 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=194 - fwd=0 - avail=0 - consume=0 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=194 - fwd=0 - avail=0 - consume=0 1488550830.976938 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=194 - nxt=194 - fwd=0 - forward=194 1488550830.976938 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=194 - nxt=194 - fwd=0 - forward=194 1488550830.981614 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=3632 - consume=3632 1488550830.981614 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=3632 - consume=3632 1488550830.981614 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=3632 - nxt=3632 - fwd=0 - forward=3632 1488550830.981614 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=3632 - nxt=3632 - fwd=0 - forward=3632 1488550830.981614 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_end : channel=REQUEST - mode=HTTP (backend) 1488550830.981614 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x00000000] trace_http_end : channel=REQUEST - mode=HTTP (backend) 1488550830.982432 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0dac0000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550830.982432 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0dac0000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550830.982432 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0da80000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550830.982432 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550830.982432 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 00000006:web-02.srvrep[0008:0009]: HTTP/1.1 401 Unauthorized 00000006:web-02.srvhdr[0008:0009]: Server: Microsoft-IIS/8.5 00000006:web-02.srvhdr[0008:0009]: WWW-Authenticate: Basic realm=FundApps API 00000006:web-02.srvhdr[0008:0009]: X-Powered-By: ASP.NET 00000006:web-02.srvhdr[0008:0009]: X-FA-Server: WEB-02A 00000006:web-02.srvhdr[0008:0009]: Date: Fri, 03 Mar 2017 14:20:28 GMT 00000006:web-02.srvhdr[0008:0009]: Content-Length: 0 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=POST 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=POST 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=PRE 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=PRE 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=POST 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=POST 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] trace_http_headers : channel=RESPONSE - mode=HTTP (backend) 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] HTTP/1.1 401 Unauthorized 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] Server: Microsoft-IIS/8.5 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] WWW-Authenticate: Basic realm=FundApps API 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] X-Powered-By: ASP.NET 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] X-FA-Server: WEB-02A 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] Date: Fri, 03 Mar 2017 14:20:28 GMT 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] Content-Length: 0 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] trace_http_headers : channel=RESPONSE - mode=HTTP (backend) 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] HTTP/1.1 401 Unauthorized 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] Server: Microsoft-IIS/8.5 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] WWW-Authenticate: Basic realm=FundApps API 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] X-Powered-By: ASP.NET 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] X-FA-Server: WEB-02A 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] Date: Fri, 03 Mar 2017 14:20:28 GMT 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0b000000] Content-Length: 0 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0a000000] trace_http_data : channel=RESPONSE - mode=HTTP (backend) - chunk_len=0 - next=201 - fwd=0 - avail=0 - consume=0 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0a000000] trace_http_data : channel=RESPONSE - mode=HTTP (backend) - chunk_len=0 - next=201 - fwd=0 - avail=0 - consume=0 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0a000000] trace_http_forward_data : channel=RESPONSE - mode=HTTP (backend) - len=201 - nxt=201 - fwd=0 - forward=201 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0a000000] trace_http_forward_data : channel=RESPONSE - mode=HTTP (backend) - len=201 - nxt=201 - fwd=0 - forward=201 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(6) 0x00028000 0x0a000000] trace_http_end : channel=RESPONSE - mode=HTTP (backend) 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(6) 0x00028000 0x0a000000] trace_http_end : channel=RESPONSE - mode=HTTP (backend) 1488550830.983488 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00020000 0x08000000] trace_chn_end_analyze : channel=RESPONSE - mode=HTTP (frontend) 1488550830.983488 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00020000 0x08000000] trace_chn_end_analyze : channel=RESPONSE - mode=HTTP (frontend) 1488550830.985684 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00020000 0x00000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550830.985684 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00020000 0x00000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550830.985684 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034035 0x00000000] trace_detach : filter-type=backend 1488550830.985684 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034035 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550830.985684 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 00000007:https-in.clireq[0008:0009]: POST /v1/expost/check HTTP/1.1 00000007:https-in.clihdr[0008:0009]: Content-Length: 3632 00000007:https-in.clihdr[0008:0009]: Content-Type: application/xml 00000007:https-in.clihdr[0008:0009]: Host: [REDACTED] 00000007:https-in.clihdr[0008:0009]: Connection: Keep-Alive 00000007:https-in.clihdr[0008:0009]: User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 00000007:https-in.clihdr[0008:0009]: Authorization: Basic dGVzdDp0ZXN0 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034030 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=POST 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034030 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_HTTP_PROCESS_FE - step=PRE 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034020 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_HTTP_PROCESS_FE - step=POST 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034020 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_SWITCHING_RULES - step=PRE 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034000 0x00000000] trace_attach : filter-type=backend 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034000 0x00000000] trace_stream_set_backend : backend=web-02 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034000 0x00000000] trace_stream_set_backend : backend=web-02 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034940 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_SWITCHING_RULES - step=POST 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034940 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (backend) 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034900 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=PRE 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034900 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=PRE 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=POST 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=POST 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=PRE 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=PRE 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=POST 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=POST 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] trace_http_headers : channel=REQUEST - mode=HTTP (backend) 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] POST /v1/expost/check HTTP/1.1 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Content-Length: 3632 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Content-Type: application/xml 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Host: [REDACTED] 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Authorization: Basic dGVzdDp0ZXN0 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] X-Forwarded-For: 34.250.167.137 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] trace_http_headers : channel=REQUEST - mode=HTTP (backend) 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] POST /v1/expost/check HTTP/1.1 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Content-Length: 3632 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Content-Type: application/xml 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Host: [REDACTED] 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] Authorization: Basic dGVzdDp0ZXN0 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x0002c000 0x00000000] X-Forwarded-For: 34.250.167.137 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=229 - fwd=0 - avail=0 - consume=0 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=229 - fwd=0 - avail=0 - consume=0 1488550831.022211 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=229 - nxt=229 - fwd=0 - forward=229 1488550831.022211 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=229 - nxt=229 - fwd=0 - forward=229 1488550831.026857 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=3632 - consume=3632 1488550831.026857 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=3632 - consume=3632 1488550831.026857 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=3632 - nxt=3632 - fwd=0 - forward=3632 1488550831.026857 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=3632 - nxt=3632 - fwd=0 - forward=3632 1488550831.026857 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_http_end : channel=REQUEST - mode=HTTP (backend) 1488550831.026857 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_http_end : channel=REQUEST - mode=HTTP (backend) 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0dac0000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0da80000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 00000007:web-02.srvrep[0008:0009]: HTTP/1.1 401 Unauthorized 00000007:web-02.srvhdr[0008:0009]: Server: Microsoft-IIS/8.5 00000007:web-02.srvhdr[0008:0009]: WWW-Authenticate: Basic realm=FundApps API 00000007:web-02.srvhdr[0008:0009]: X-Powered-By: ASP.NET 00000007:web-02.srvhdr[0008:0009]: X-FA-Server: WEB-02A 00000007:web-02.srvhdr[0008:0009]: Date: Fri, 03 Mar 2017 14:20:28 GMT 00000007:web-02.srvhdr[0008:0009]: Content-Length: 0 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=POST 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=POST 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=PRE 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=PRE 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=POST 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=POST 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] trace_http_headers : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] HTTP/1.1 401 Unauthorized 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] Server: Microsoft-IIS/8.5 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] WWW-Authenticate: Basic realm=FundApps API 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] X-Powered-By: ASP.NET 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] X-FA-Server: WEB-02A 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] Date: Fri, 03 Mar 2017 14:20:28 GMT 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] Content-Length: 0 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] trace_http_headers : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] HTTP/1.1 401 Unauthorized 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] Server: Microsoft-IIS/8.5 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] WWW-Authenticate: Basic realm=FundApps API 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] X-Powered-By: ASP.NET 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] X-FA-Server: WEB-02A 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] Date: Fri, 03 Mar 2017 14:20:28 GMT 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0b000000] Content-Length: 0 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0a000000] trace_http_data : channel=RESPONSE - mode=HTTP (backend) - chunk_len=0 - next=201 - fwd=0 - avail=0 - consume=0 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0a000000] trace_http_data : channel=RESPONSE - mode=HTTP (backend) - chunk_len=0 - next=201 - fwd=0 - avail=0 - consume=0 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0a000000] trace_http_forward_data : channel=RESPONSE - mode=HTTP (backend) - len=201 - nxt=201 - fwd=0 - forward=201 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0a000000] trace_http_forward_data : channel=RESPONSE - mode=HTTP (backend) - len=201 - nxt=201 - fwd=0 - forward=201 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(7) 0x00028000 0x0a000000] trace_http_end : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(7) 0x00028000 0x0a000000] trace_http_end : channel=RESPONSE - mode=HTTP (backend) 1488550831.033378 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00020000 0x08000000] trace_chn_end_analyze : channel=RESPONSE - mode=HTTP (frontend) 1488550831.033378 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(8) 0x00020000 0x08000000] trace_chn_end_analyze : channel=RESPONSE - mode=HTTP (frontend) 1488550831.035268 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00020000 0x00000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550831.035268 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(8) 0x00020000 0x00000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550831.035268 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(8) 0x00034035 0x00000000] trace_detach : filter-type=backend 1488550831.035268 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00034035 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550831.035268 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 1488550831.056425 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 1488550831.056425 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00020000 0x00000000] trace_http_reply : channel=- - mode=HTTP (frontend) 1488550831.056425 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00020000 0x00000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 00000008:https-in.clicls[0008:0009] 00000008:https-in.closed[0008:0009] 1488550831.056425 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00000000 0x00000000] trace_stream_stop 1488550831.056425 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(8) 0x00000000 0x00000000] trace_detach : filter-type=frontend
repro-haproxy.cfg
Description: Binary data
1488550869.455670 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00000000 0x00000000] trace_attach : filter-type=frontend 1488550869.455670 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00000000 0x00000000] trace_stream_start 0000000e:https-in.accept(0005)=0008 from [34.250.167.137:38284] 1488550869.455670 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034035 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550869.455670 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 0000000e:https-in.clireq[0008:ffffffff]: POST /v1/expost/check HTTP/1.1 0000000e:https-in.clihdr[0008:ffffffff]: Content-Length: 3632 0000000e:https-in.clihdr[0008:ffffffff]: Content-Type: application/xml 0000000e:https-in.clihdr[0008:ffffffff]: Host: [REDACTED] 0000000e:https-in.clihdr[0008:ffffffff]: Connection: Keep-Alive 0000000e:https-in.clihdr[0008:ffffffff]: User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034030 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=POST 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034030 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_HTTP_PROCESS_FE - step=PRE 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034020 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_HTTP_PROCESS_FE - step=POST 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034020 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_SWITCHING_RULES - step=PRE 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00034000 0x00000000] trace_attach : filter-type=backend 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034000 0x00000000] trace_stream_set_backend : backend=web-02 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00034000 0x00000000] trace_stream_set_backend : backend=web-02 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034940 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_SWITCHING_RULES - step=POST 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00034940 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (backend) 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034900 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=PRE 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00034900 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=PRE 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=POST 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_PROCESS_BE - step=POST 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=PRE 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00034800 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=PRE 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=POST 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (backend) - analyzer=AN_REQ_HTTP_INNER - step=POST 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] trace_http_headers : channel=REQUEST - mode=HTTP (backend) 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] POST /v1/expost/check HTTP/1.1 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] Content-Length: 3632 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] Content-Type: application/xml 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] Host: [REDACTED] 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] X-Forwarded-For: 34.250.167.137 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] trace_http_headers : channel=REQUEST - mode=HTTP (backend) 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] POST /v1/expost/check HTTP/1.1 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] Content-Length: 3632 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] Content-Type: application/xml 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] Host: [REDACTED] 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] User-Agent: Apache-HttpClient/4.2.3 (java 1.5) 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x0002c000 0x00000000] X-Forwarded-For: 34.250.167.137 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=194 - fwd=0 - avail=0 - consume=0 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x00000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=194 - fwd=0 - avail=0 - consume=0 1488550869.491238 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=194 - nxt=194 - fwd=0 - forward=194 1488550869.491238 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x00000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=194 - nxt=194 - fwd=0 - forward=194 1488550869.491958 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0dac0000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=0 - consume=0 1488550869.491958 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0dac0000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=0 - consume=0 1488550869.491958 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0dac0000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=0 - nxt=0 - fwd=0 - forward=0 1488550869.491958 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0dac0000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=0 - nxt=0 - fwd=0 - forward=0 1488550869.491958 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0dac0000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550869.491958 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0dac0000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550869.491958 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0da80000] trace_chn_start_analyze : channel=RESPONSE - mode=HTTP (backend) 1488550869.491958 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550869.491958 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0da00000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=PRE 0000000e:web-02.srvrep[0008:0009]: HTTP/1.1 401 Unauthorized 0000000e:web-02.srvhdr[0008:0009]: Server: Microsoft-IIS/8.5 0000000e:web-02.srvhdr[0008:0009]: WWW-Authenticate: Basic realm=FundApps API 0000000e:web-02.srvhdr[0008:0009]: X-Powered-By: ASP.NET 0000000e:web-02.srvhdr[0008:0009]: X-FA-Server: WEB-02A 0000000e:web-02.srvhdr[0008:0009]: Date: Fri, 03 Mar 2017 14:21:06 GMT 0000000e:web-02.srvhdr[0008:0009]: Content-Length: 0 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=POST 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_WAIT_HTTP - step=POST 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=PRE 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0d800000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=PRE 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=POST 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] trace_chn_analyze : channel=RESPONSE - mode=HTTP (backend) - analyzer=AN_RES_HTTP_PROCESS_FE/BE - step=POST 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] trace_http_headers : channel=RESPONSE - mode=HTTP (backend) 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] HTTP/1.1 401 Unauthorized 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] Server: Microsoft-IIS/8.5 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] WWW-Authenticate: Basic realm=FundApps API 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] X-Powered-By: ASP.NET 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] X-FA-Server: WEB-02A 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] Date: Fri, 03 Mar 2017 14:21:06 GMT 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] Content-Length: 0 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] trace_http_headers : channel=RESPONSE - mode=HTTP (backend) 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] HTTP/1.1 401 Unauthorized 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] Server: Microsoft-IIS/8.5 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] WWW-Authenticate: Basic realm=FundApps API 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] X-Powered-By: ASP.NET 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] X-FA-Server: WEB-02A 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] Date: Fri, 03 Mar 2017 14:21:06 GMT 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0b000000] Content-Length: 0 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_data : channel=RESPONSE - mode=HTTP (backend) - chunk_len=0 - next=201 - fwd=0 - avail=0 - consume=0 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_data : channel=RESPONSE - mode=HTTP (backend) - chunk_len=0 - next=201 - fwd=0 - avail=0 - consume=0 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_forward_data : channel=RESPONSE - mode=HTTP (backend) - len=201 - nxt=201 - fwd=0 - forward=201 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_forward_data : channel=RESPONSE - mode=HTTP (backend) - len=201 - nxt=201 - fwd=0 - forward=201 1488550869.492754 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_end : channel=RESPONSE - mode=HTTP (backend) 1488550869.492754 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_end : channel=RESPONSE - mode=HTTP (backend) 1488550869.494912 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=3632 - consume=3632 1488550869.494912 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_data : channel=REQUEST - mode=HTTP (backend) - chunk_len=3632 - next=0 - fwd=0 - avail=3632 - consume=3632 1488550869.494912 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=3632 - nxt=3632 - fwd=0 - forward=3632 1488550869.494912 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_forward_data : channel=REQUEST - mode=HTTP (backend) - len=3632 - nxt=3632 - fwd=0 - forward=3632 1488550869.494912 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_end : channel=REQUEST - mode=HTTP (backend) 1488550869.494912 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(e) 0x00028000 0x0a000000] trace_http_end : channel=REQUEST - mode=HTTP (backend) 1488550869.495170 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00020000 0x08000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550869.495170 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(f) 0x00020000 0x08000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550869.495170 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00000000 0x08000000] trace_http_reply : channel=- - mode=HTTP (frontend) 1488550869.495170 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(f) 0x00000000 0x08000000] trace_http_reply : channel=- - mode=HTTP (frontend) 1488550869.495170 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00000000 0x08000000] trace_chn_end_analyze : channel=RESPONSE - mode=HTTP (frontend) 1488550869.495170 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(f) 0x00000000 0x08000000] trace_chn_end_analyze : channel=RESPONSE - mode=HTTP (frontend) 1488550869.495170 [BE-BEFORE-COMP/web-02] [strm 0x55ec556ab060(f) 0x00034035 0x00000000] trace_detach : filter-type=backend 1488550869.495170 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00034035 0x00000000] trace_chn_start_analyze : channel=REQUEST - mode=HTTP (frontend) 1488550869.495170 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00034034 0x00000000] trace_chn_analyze : channel=REQUEST - mode=HTTP (frontend) - analyzer=AN_REQ_WAIT_HTTP - step=PRE 1488550869.495170 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00020000 0x00000000] trace_http_reply : channel=- - mode=HTTP (frontend) 1488550869.495514 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00020000 0x00000000] trace_chn_end_analyze : channel=REQUEST - mode=HTTP (frontend) 0000000f:https-in.clicls[0008:0009] 0000000f:https-in.closed[0008:0009] 1488550869.495514 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00000000 0x00000000] trace_stream_stop 1488550869.495514 [FE-BEFORE-COMP/https-in] [strm 0x55ec556ab060(f) 0x00000000 0x00000000] trace_detach : filter-type=frontend
