Hi Willy and Lukas,
Here's snippets of the new config:
-----------------------------
global
maxconn 645000
maxpipes 645000
ulimit-n 645120
user haproxy
group haproxy
tune.bufsize 49152
spread-checks 10
daemon
quiet
stats socket /var/run/haproxy.sock level admin
pidfile /var/run/haproxy.pid
defaults
log global
mode http
option accept-invalid-http-request
option accept-invalid-http-response
option httplog
option dontlognull
option dontlog-normal
option log-separate-errors
option http-server-close
option tcp-smart-connect
option tcp-smart-accept
option forwardfor except 127.0.0.1
option dontlog-normal
retries 3
option redispatch
maxconn 2000000
contimeout 5000
clitimeout 60000
srvtimeout 60000
listen www 0.0.0.0:80
mode http
capture response header Via len 20
capture response header Content-Length len 10
capture response header Cache-Control len 8
capture response header Location len 40
balance roundrobin
# Haproxy status page
stats uri /haproxy-status
stats auth fb:phoo
# when cookie persistence is required
cookie SERVERID insert indirect nocache
# When internal servers support a status page
option httpchk GET /xyzzyx.php
bind 0.0.0.0:443 ssl crt /etc/lighttpd/ssl_certs/xxxx.co.pem
http-request add-header X-FORWARDED-PROTO https if { ssl_fc }
server app1 10.1.1.6:85 check inter 40000 rise 2 fall 3 maxconn
16384
server app2 10.1.1.7:85 check inter 40000 rise 2 fall 3 maxconn
16384
-----------------------------
The old config did NOT have the following items, and had about 500x more
errors:
-----------------------------
tune.bufsize 49152
option accept-invalid-http-request
option accept-invalid-http-response
-----------------------------
Here's what the 'show errors' shows on a sampling of the server. It looks
like 90% of the errors are the second error (25/Nov/2014:00:06:30.753):
Total events captured on [24/Nov/2014:23:31:52.468] : 151
[22/Nov/2014:21:55:56.597] frontend www (#2): invalid request
backend www (#2), server <NONE> (#-1), event #150
src 166.137.247.239:8949, session #3883610, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 1183 bytes
pending 1183 bytes, wrapping at 49152, error at position 227:
00000
sited%22%3A1416713764%2C%22times_visited%22%3A6%2C%22device%22%3A%22We
00070+
b%22%2C%22lastsource%22%3A%22bottomxpromo%22%2C%22language%22%3A%22en%
00140+
22%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22di
00210+ d_watch%22%3A1%7D; yXXXXX=5167136038811769837; _vhist=%7B%22visito
00280+
r_id%22%3A%225024165909427731336%22%2C%22seen_articles%22%3A%22%7B%5C%
00350+
22950%5C%22%3A1402416590%2C%5C%22685%5C%22%3A1402416675%2C%5C%22799%5C
00420+
%22%3A1402416789%2C%5C%22954%5C%22%3A1402416997%2C%5C%22939%5C%22%3A14
00490+
02417098%2C%5C%222334%5C%22%3A1407162586%2C%5C%222055%5C%22%3A14071626
00560+
91%2C%5C%223888%5C%22%3A1409938121%2C%5C%223020%5C%22%3A1409938211%2C%
00630+
5C%223773%5C%22%3A1409938340%2C%5C%222163%5C%22%3A1409938389%2C%5C%222
00700+
569%5C%22%3A1409938872%2C%5C%222426%5C%22%3A1409938959%2C%5C%2213984%5
00770+
C%22%3A1411916274%2C%5C%221675%5C%22%3A1411916466%2C%5C%2214950%5C%22%
00840+
3A1412432461%2C%5C%2219759%5C%22%3A1416714580%7D%22%2C%22num_articles%
00910+ 22%3A17%7D; lastlargeleaderboard=1416713603;
lastlike-264755123648776=
00980+ 1416713764; lastlike-703681396382652=1416713919;
lastlike-939184096095
01050+ 676=1416714070; lastlike-293377080869393=1416714282;
lastlike-14790568
01120+ 05705686=1416714412; popped_19731=1\r\n
01157 Connection: keep-alive\r\n
01181 \r\n
-----------
Total events captured on [25/Nov/2014:00:06:30.753] : 480
[25/Nov/2014:00:06:26.417] frontend www (#2): invalid request
backend www (#2), server <NONE> (#-1), event #479
src 50.92.157.165:53649, session #14629125, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x84000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 45 bytes
pending 45 bytes, wrapping at 49152, error at position 0:
00000
\x17\x03\x03\x00(t\xF42\x9D\x96\x19[(\xF6\xFEK\xB7\x9Br\x00C\xDBn<\x98
00025+ \x12"\xC2\xBC\x91\x91\x0E5\xEA,\xCE0t\xF3\x90\x16\x1Dd\x99\xC4
------------------
Total events captured on [25/Nov/2014:00:08:45.356] : 128
[24/Nov/2014:22:38:34.626] frontend www (#2): invalid request
backend www (#2), server <NONE> (#-1), event #127
src 157.55.39.106:29900, session #4147419, session flags 0x00000080
HTTP msg state 27, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 304 bytes
pending 304 bytes, wrapping at 49152, error at position 15:
00000 GET
/?id=551&sr\xC3\x83\xC2\xAF\xC3\x82\xC2\xBF\xC3\x82\xC2\xBD=share_
00034+ fb_new_551 HTTP/1.1\r\n
00055 Cache-Control: no-cache\r\n
00080 Connection: Keep-Alive\r\n
00104 Pragma: no-cache\r\n
00122 Accept: */*\r\n
00135 Accept-Encoding: gzip, deflate\r\n
00167 From: bingbot(at)microsoft.com\r\n
00199 Host: xxxxxx.co\r\n
00217 User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +
http://www.bing.com
00287+ /bingbot.htm)\r\n
00302 \r\n
------------------------
Total events captured on [25/Nov/2014:00:11:16.030] : 557
[25/Nov/2014:00:08:19.250] frontend www (#2): invalid request
backend www (#2), server <NONE> (#-1), event #556
src 199.59.148.211:40148, session #14688643, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x84000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 140 bytes
pending 140 bytes, wrapping at 49152, error at position 5:
00000 GET /\x00/?id=16278&src=shorturl_16278 HTTP/1.1\r\n
00046 Host: xxxxxx.com\r\n
00065 User-Agent: Twitterbot/1.0\r\n
00093 Accept-Encoding: gzip, deflate\r\n
00125 Accept: */*\r\n
00138 \r\n
---------------------
Total events captured on [25/Nov/2014:00:12:57.050] : 577
[25/Nov/2014:00:09:20.036] frontend www (#2): invalid request
backend www (#2), server <NONE> (#-1), event #576
src 173.209.211.205:51013, session #15108367, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 60 bytes
pending 60 bytes, wrapping at 49152, error at position 6:
00000 action=video_ready&value=1&time_on_page=140.87&article=18539
----------------
Total events captured on [25/Nov/2014:00:14:54.259] : 526
[25/Nov/2014:00:08:22.991] frontend www (#2): invalid request
backend www (#2), server <NONE> (#-1), event #525
src 67.231.32.73:59810, session #15100093, session flags 0x00000080
HTTP msg state 26, msg flags 0x00000000, tx flags 0x00000000
HTTP chunk len 0 bytes, HTTP body len 0 bytes
buffer flags 0x00808002, out 0 bytes, total 1460 bytes
pending 1460 bytes, wrapping at 49152, error at position 6:
00000 Cookie:
_vis=%7B%22visitor_id%22%3A%225108882918757752765%22%2C%22sour
00070+
ce%22%3A%22fbfan_2081%22%2C%22joined%22%3A1410888291%2C%22last_visited
00140+
%22%3A1412459722%2C%22times_visited%22%3A23%2C%22language%22%3A%22en%2
00210+
2%2C%22extra%22%3A%22%7B%5C%22tr%5C%22%3A%5C%22en%5C%22%7D%22%2C%22dev
00280+
ice%22%3A%22Web%22%2C%22lastsource%22%3A%22fbfan_15040%22%2C%22did_wat
00350+ ch%22%3A1%7D; yXXXXX=5108882918757752765; _vhist=%7B%22visitor_id%
00420+
22%3A%225108882918757752765%22%2C%22seen_articles%22%3A%22%7B%5C%22208
00490+
1%5C%22%3A1410888291%2C%5C%2213102%5C%22%3A1410888488%2C%5C%2213225%5C
00560+
%22%3A1410956287%2C%5C%2213369%5C%22%3A1410996782%2C%5C%2212535%5C%22%
00630+
3A1410997026%2C%5C%223020%5C%22%3A1410997306%2C%5C%222913%5C%22%3A1410
00700+
997471%2C%5C%222426%5C%22%3A1410997631%2C%5C%2213208%5C%22%3A141099779
00770+
0%2C%5C%222350%5C%22%3A1410997918%2C%5C%2212995%5C%22%3A1410998005%2C%
00840+
5C%2212777%5C%22%3A1410998345%2C%5C%222996%5C%22%3A1410998730%2C%5C%22
00910+
3471%5C%22%3A1410999158%2C%5C%222778%5C%22%3A1410999435%2C%5C%221840%5
00980+
C%22%3A1410999880%2C%5C%222662%5C%22%3A1411000322%2C%5C%222655%5C%22%3
01050+
A1411000534%2C%5C%222777%5C%22%3A1411000586%2C%5C%222398%5C%22%3A14110
01120+
00887%2C%5C%223787%5C%22%3A1411001258%2C%5C%2212977%5C%22%3A1411001402
01190+
%2C%5C%222447%5C%22%3A1411001612%2C%5C%222450%5C%22%3A1411001901%2C%5C
01260+
%222463%5C%22%3A1411002161%2C%5C%2213213%5C%22%3A1411029331%2C%5C%2236
01330+
83%5C%22%3A1411029512%2C%5C%22655%5C%22%3A1411029963%2C%5C%222217%5C%2
01400+ 2%3A1411030200%2C%5C%222464%5C%22%3A1411030432%2C%5C%222767%
-----------------------------
Thanks!
-Alex
On Mon, Nov 24, 2014 at 12:52 AM, Willy Tarreau <w...@1wt.eu> wrote:
> On Sun, Nov 23, 2014 at 11:57:50PM +0800, Alexey Zilber wrote:
> > Hi Willy,
> >
> > I already do "option dontlognull". These get logged anyway. We've
> > actually seen a high amount of 400 errors happening. I'll post the
> configs
> > in a day or two (the pre/post) configs. I've been able to knock down the
> > 400 and 502 errors to about 10%, with 400 errors being around .6% of
> total
> > connections.
>
> Then there are data received over these connections, so please follow
> Lukas' advice and issue "show errors" on the stats socket, you'll see
> what is sent there. Maybe some bogus browsers sending an SSL hello over
> this connection or something like this...
>
> Willy
>
>
