Hello again! So, I've used tcpdump to capture the traffic on both sides - Haproxy and on backend and ruled out the possible issue with Haproxy. It's clearly not the load balancer issue, it looks like some module on backend incorrectly sends a packet of binary service data (which most likely should go to Amazon I guess) to the Haproxy using active session and backend port, instead of sending it using separate port and session as it should. So the Haproxy gets this wrong data which shouldn't be passed to it.
Thanks a lot for the tip, and sorry for the false alarm. I'm now going to debug the backend to find the source of this rogue packet. With best regards, Alexey. 2017-10-13 20:07 GMT+03:00 Lukas Tribus <lu...@ltri.eu>: > Hello Alexey, > > > > > > Details of error 502 from Haproxy: > > > > client_ip=<redacted> client_port=32668 backend=http_data > times=0/0/0/-1/91 > > http_status=502 bytes_uploaded=478 bytes_read=1807 termination_state=PH-- > > connections=1/0/0/0/0 queues=0/0 "PATCH /incoming_data HTTP/1.1" > > > > Termination state PH means error happened prematurely on headers > processing > > so I've checked out the output of 'show errors' socket command and saw > the > > following: > > > > Total events captured on [11/Oct/2017:07:27:22.352] : 1 > > > > [10/Oct/2017:18:34:51.923] backend http-data (#8): invalid response > > frontend http-ssl (#3), server incoming_data_bk1 (#1), event #1 > > src <redacted>:24565, session #12, session flags 0x000004ce > > HTTP msg state 26, msg flags 0x00000000, tx flags 0x08000000 > > HTTP chunk len 0 bytes, HTTP body len 0 bytes > > buffer flags 0x00008002, out 0 bytes, total 1453 bytes > > pending 1453 bytes, wrapping at 16384, error at position 0: > > > > 00000 \x17\x03\x03\x05 > > That's all invalid data that haproxy sees here. The question is where is > this > data coming from. > > > > > And there is 100+ more lines of binary data. > > I've inspected the actual reply from web-server (it's logged before > sending) > > and it was only 6 bytes with all proper headers. > > But you can see that there is a disparity with what haproxy believes > it is receiving > and what your backend believes it is sending. Somebody is lying ... > > So, a packet capture with tcpdump is the only way to find out what happens. > > > Capture something like this, replace <interface> and <backendip> as > appropriate: > > tcpdump -i <interface> -nps0 -w backendtraffic.cap host <backendip> and > port 80 > > > > This capture has to be taken on the box/VM/instance that is running > haproxy on. > > > > > > Then I've started Haproxy in debug mode on foreground to see all info > about > > all requests and responses on screen. > > After I've send the PATCH request Haproxy have stuck for about 40 seconds > > (literally) but after this pause it continued with proper response from > > backend! > > > > I have no idea what Haproxy did in that 40 seconds - there was no > messages > > on screen about anything. After this 'freeze' the normal response from > > backend have appeared. > > All headers was in place and Content-Length and Content-Type was OK when > > response have transmitted. > > That may be a different problem. > > > > > So it looks like Haproxy somehow intercepts data exchange between backend > > and DynamoDB and treat this data as reply. > > Does anyone saw something like this before? Any suggestions on how to > > debug/fix that? What might be the cause of that? > > > > I've checked this out with all main releases of Haproxy - 1.5, 1.6, 1.7 - > > all have the same behavior with that. > > We need the actual configuration and the *exact* haproxy release, as well > as the > output of "haproxy -vv" to check if you are hitting any known bugs. > > But I believe your time is better invested in taking that tcpdump capture. > > > > Regards, > Lukas >
