> Is this happen randomly or can you pin point this to specifc requests, maybe > errors/timeouts? How can we reproduce this?
Nevermind, its easily reproducible (just generate some syslog messages). The whole thing seems random: most of the times, the syslog msg ends with \n\0\0, other times with \n\n\0 or even \n"\n. I've tracked this down to: 2a4a44f0f9f08d130a9a7211746728c38f670032 is the first bad commit commit 2a4a44f0f9f08d130a9a7211746728c38f670032 Author: William Lallemand <wlallem...@exceliance.fr> Date: Mon Feb 6 16:00:33 2012 +0100 REORG: log: split send_log function Heres a tcpdump output: 10:11:45.854741 IP 10.0.0.55.39079> 10.0.0.3.514: SYSLOG syslog.notice, length: 64 0x0000: 4500 005c 0000 4000 4011 2658 0a00 0037 E..\..@.@.&X...7 0x0010: 0a00 0003 98a7 0202 0048 1493 3c34 353e .........H..<45> 0x0020: 4175 6720 3232 2031 303a 3131 3a34 3520 Aug.22.10:11:45. 0x0030: 6861 7072 6f78 795b 3135 3034 315d 3a20 haproxy[15041]:. 0x0040: 5072 6f78 7920 6d79 6672 6f6e 7465 6e64 Proxy.myfrontend 0x0050: 2073 7461 7274 6564 2e0a 0000 .started.... 10:11:50.113766 IP 10.0.0.55.39079> 10.0.0.3.514: SYSLOG syslog.info, length: 170 0x0000: 4500 00c6 0000 4000 4011 25ee 0a00 0037 E.....@.@.%....7 0x0010: 0a00 0003 98a7 0202 00b2 14fd 3c34 363e ............<46> 0x0020: 4175 6720 3232 2031 303a 3131 3a35 3020 Aug.22.10:11:50. 0x0030: 6861 7072 6f78 795b 3135 3034 315d 3a20 haproxy[15041]:. 0x0040: 3a3a 6666 6666 3a31 302e 302e 302e 333a ::ffff:10.0.0.3: 0x0050: 3536 3437 3020 5b32 322f 4175 672f 3230 56470.[22/Aug/20 0x0060: 3133 3a31 303a 3131 3a35 302e 3130 385d 13:10:11:50.108] 0x0070: 206d 7966 726f 6e74 656e 6420 6d79 6261 .myfrontend.myba 0x0080: 636b 656e 642f 7333 2030 2f30 2f30 2f34 ckend/s3.0/0/0/4 0x0090: 2f34 2034 3034 2031 3434 3320 2d20 2d20 /4.404.1443.-.-. 0x00a0: 2d2d 2d2d 2031 2f31 2f30 2f30 2f30 2030 ----.1/1/0/0/0.0 0x00b0: 2f30 2022 4745 5420 2f20 4854 5450 2f31 /0."GET./.HTTP/1 0x00c0: 2e31 220a 0000 .1"... 10:11:53.595881 IP 10.0.0.55.39079> 10.0.0.3.514: SYSLOG syslog.info, length: 171 0x0000: 4500 00c7 0000 4000 4011 25ed 0a00 0037 E.....@.@.%....7 0x0010: 0a00 0003 98a7 0202 00b3 14fe 3c34 363e ............<46> 0x0020: 4175 6720 3232 2031 303a 3131 3a35 3320 Aug.22.10:11:53. 0x0030: 6861 7072 6f78 795b 3135 3034 315d 3a20 haproxy[15041]:. 0x0040: 3a3a 6666 6666 3a31 302e 302e 302e 333a ::ffff:10.0.0.3: 0x0050: 3536 3437 3120 5b32 322f 4175 672f 3230 56471.[22/Aug/20 0x0060: 3133 3a31 303a 3131 3a35 332e 3538 315d 13:10:11:53.581] 0x0070: 206d 7966 726f 6e74 656e 6420 6d79 6261 .myfrontend.myba 0x0080: 636b 656e 642f 7333 2039 2f30 2f31 2f34 ckend/s3.9/0/1/4 0x0090: 2f31 3420 3430 3420 3134 3433 202d 202d /14.404.1443.-.- 0x00a0: 202d 2d2d 2d20 312f 312f 302f 312f 3020 .----.1/1/0/1/0. 0x00b0: 302f 3020 2247 4554 202f 2048 5454 502f 0/0."GET./.HTTP/ 0x00c0: 312e 3122 0a00 00 1.1"... 10:11:57.830360 IP 10.0.0.55.39079> 10.0.0.3.514: SYSLOG syslog.info, length: 170 0x0000: 4500 00c6 0000 4000 4011 25ee 0a00 0037 E.....@.@.%....7 0x0010: 0a00 0003 98a7 0202 00b2 14fd 3c34 363e ............<46> 0x0020: 4175 6720 3232 2031 303a 3131 3a35 3720 Aug.22.10:11:57. 0x0030: 6861 7072 6f78 795b 3135 3034 315d 3a20 haproxy[15041]:. 0x0040: 3a3a 6666 6666 3a31 302e 302e 302e 333a ::ffff:10.0.0.3: 0x0050: 3536 3437 3220 5b32 322f 4175 672f 3230 56472.[22/Aug/20 0x0060: 3133 3a31 303a 3131 3a35 372e 3832 365d 13:10:11:57.826] 0x0070: 206d 7966 726f 6e74 656e 6420 6d79 6261 .myfrontend.myba 0x0080: 636b 656e 642f 7333 2030 2f30 2f30 2f33 ckend/s3.0/0/0/3 0x0090: 2f33 2034 3034 2031 3434 3320 2d20 2d20 /3.404.1443.-.-. 0x00a0: 2d2d 2d2d 2031 2f31 2f30 2f31 2f30 2030 ----.1/1/0/1/0.0 0x00b0: 2f30 2022 4745 5420 2f20 4854 5450 2f31 /0."GET./.HTTP/1 0x00c0: 2e31 220a 0a00 .1"... 10:12:05.793157 IP 10.0.0.55.39079> 10.0.0.3.514: SYSLOG syslog.info, length: 170 0x0000: 4500 00c6 0000 4000 4011 25ee 0a00 0037 E.....@.@.%....7 0x0010: 0a00 0003 98a7 0202 00b2 14fd 3c34 363e ............<46> 0x0020: 4175 6720 3232 2031 303a 3132 3a30 3520 Aug.22.10:12:05. 0x0030: 6861 7072 6f78 795b 3135 3034 315d 3a20 haproxy[15041]:. 0x0040: 3a3a 6666 6666 3a31 302e 302e 302e 333a ::ffff:10.0.0.3: 0x0050: 3536 3437 3320 5b32 322f 4175 672f 3230 56473.[22/Aug/20 0x0060: 3133 3a31 303a 3132 3a30 352e 3738 385d 13:10:12:05.788] 0x0070: 206d 7966 726f 6e74 656e 6420 6d79 6261 .myfrontend.myba 0x0080: 636b 656e 642f 7333 2030 2f30 2f30 2f34 ckend/s3.0/0/0/4 0x0090: 2f34 2034 3034 2031 3434 3320 2d20 2d20 /4.404.1443.-.-. 0x00a0: 2d2d 2d2d 2031 2f31 2f30 2f30 2f30 2030 ----.1/1/0/0/0.0 0x00b0: 2f30 2022 4745 5420 2f20 4854 5450 2f31 /0."GET./.HTTP/1 0x00c0: 2e31 220a 0a00 .1"... Cheers, Lukas
