Hello,
We encounter a nasty bug when htx is enabled.
Under certain conditions, the incoming client data can overwrite part of
the buffer with data prepared for backend servers.
I was able to reproduce the issue by using the attached script
('request.py') to generate http requests.
I did a wireshark capture of the traffic originating from haproxy toward
the backend server.
In random cases, I observe a corrupted x-forwarded-for header in the
requests sent to the backend server.
For example :
==================================
POST / HTTP/1.1
content-length: 14700
accept-encoding: gzip, deflate
accept: */*
user-agent: python-requests/2.6.0 CPython/2.7.5 Linux
host: a.exemple.com
222222222233333: 3333333344444
abcdefgh<.... the rest of the request body....>
==================================
I wasn't able to reproduce the issue by doing requests to a haproxy
located on localhost.
I believe there are timing requirements to reproduce it:
- Haproxy should start processing part of the HTTP request before
receiving the rest of the body.
- Moreover, the rest of the request body should arrive after this
partial processing (but before sending the first request to backend
servers).
Having some network latency between the client and haproxy helps
reproduce the issue. In my case: ~7ms.
The issue only occurs for requests with content-length>1448.
Moreover, If content-length == 1450, only the first 2 byte of the
x-forwarded-for header name will be corrupted with the last 2 bytes of
the request.
I was able to reproduce the issue with all haproxy 1.9.* release
versions, and a fairly minimal haproxy.cfg (see attached).
The bug still persists with a freshly compiled git version (head commit:
0b4b27efde653a4b7bbafb56df3796bffa4722ae).
Please tell me if you need more details.
Sincerely,
Radu
#!/usr/bin/env python
import requests
import time
import string
data = b''
for i in range(1,50):
for j in list(string.ascii_letters) + [str(x) for x in range(0,10)]:
data+=bytes(j*i, encoding='ascii')
data = data[:14700]
print(data)
for i in range(0, 200):
try:
requests.post("http://10.236.100.19";, data=data, headers={'host': 'a.exemple.com'})
except Exception as e:
print(e)
print(i)
global
user haproxy
group haproxy
nbproc 1
nbthread 1
defaults
mode http
option forwardfor
option http-use-htx
frontend fe_main
bind *:80 name http
use_backend be_main
backend be_main
server srv0 10.236.72.23:31044 weight 10
