Thanks Lukas! I'll capture the dump and reply with additional info ASAP.
Regards, Alexey. 13 окт. 2017 г. 20:07 пользователь "Lukas Tribus" <lu...@ltri.eu> написал: Hello Alexey, > Details of error 502 from Haproxy: > > client_ip=<redacted> client_port=32668 backend=http_data times=0/0/0/-1/91 > http_status=502 bytes_uploaded=478 bytes_read=1807 termination_state=PH-- > connections=1/0/0/0/0 queues=0/0 "PATCH /incoming_data HTTP/1.1" > > Termination state PH means error happened prematurely on headers processing > so I've checked out the output of 'show errors' socket command and saw the > following: > > Total events captured on [11/Oct/2017:07:27:22.352] : 1 > > [10/Oct/2017:18:34:51.923] backend http-data (#8): invalid response > frontend http-ssl (#3), server incoming_data_bk1 (#1), event #1 > src <redacted>:24565, session #12, session flags 0x000004ce > HTTP msg state 26, msg flags 0x00000000, tx flags 0x08000000 > HTTP chunk len 0 bytes, HTTP body len 0 bytes > buffer flags 0x00008002, out 0 bytes, total 1453 bytes > pending 1453 bytes, wrapping at 16384, error at position 0: > > 00000 \x17\x03\x03\x05 That's all invalid data that haproxy sees here. The question is where is this data coming from. > And there is 100+ more lines of binary data. > I've inspected the actual reply from web-server (it's logged before sending) > and it was only 6 bytes with all proper headers. But you can see that there is a disparity with what haproxy believes it is receiving and what your backend believes it is sending. Somebody is lying ... So, a packet capture with tcpdump is the only way to find out what happens. Capture something like this, replace <interface> and <backendip> as appropriate: tcpdump -i <interface> -nps0 -w backendtraffic.cap host <backendip> and port 80 This capture has to be taken on the box/VM/instance that is running haproxy on. > Then I've started Haproxy in debug mode on foreground to see all info about > all requests and responses on screen. > After I've send the PATCH request Haproxy have stuck for about 40 seconds > (literally) but after this pause it continued with proper response from > backend! > > I have no idea what Haproxy did in that 40 seconds - there was no messages > on screen about anything. After this 'freeze' the normal response from > backend have appeared. > All headers was in place and Content-Length and Content-Type was OK when > response have transmitted. That may be a different problem. > So it looks like Haproxy somehow intercepts data exchange between backend > and DynamoDB and treat this data as reply. > Does anyone saw something like this before? Any suggestions on how to > debug/fix that? What might be the cause of that? > > I've checked this out with all main releases of Haproxy - 1.5, 1.6, 1.7 - > all have the same behavior with that. We need the actual configuration and the *exact* haproxy release, as well as the output of "haproxy -vv" to check if you are hitting any known bugs. But I believe your time is better invested in taking that tcpdump capture. Regards, Lukas
