RE: syslogd dropping requests from haproxy
Found some more info, when haproxy configured to send logs to remote host instead of local syslogd, it works fine. Definately something to do with the local syslogd under heavy load. Thanks Sachin From: Sachin Shetty [mailto:sshe...@egnyte.com] Sent: Saturday, October 15, 2011 12:57 AM To: 'haproxy@formilux.org' Subject: syslogd dropping requests from haproxy Hi, We have a pretty heavily loaded haproxy server - more than one are running on a single machine. I am seeing not all requests are being logged to syslogd - I am sure this is not related to the httpclose parameter since the same conf file works fine on another machine where the load is pretty low Anybody faced any such problem? any configs in haproxy to write to a different log file instead of syslogd? Thanks Sachin
RE: syslogd dropping requests from haproxy
Hi, On 14.10.2011 22:05, Sachin Shetty wrote: Found some more info, when haproxy configured to send logs to remote host instead of local syslogd, it works fine. Definately something to do with the local syslogd under heavy load. Which syslogd do you use? Remote: Local: Which Distribution do you use? Which haproxy version do you use? Please can you show us your haproxy config. BR Aleks
Re: syslogd dropping requests from haproxy
On Sat, Oct 15, 2011 at 01:35:46AM +0530, Sachin Shetty wrote: > Found some more info, when haproxy configured to send logs to remote host > instead of local syslogd, it works fine. Definately something to do with the > local syslogd under heavy load. Check that your syslog doesn't log synchronously. For the basic sysklogd, it means you need to have a "-" in front of your file names. And even with this, sysklogd's limits are quickly reached (between 300 and 1000 logs/s depending on the machine). For high loads, I strongly recommend syslog-ng. It's the only one I found which managed to log more than 1 logs/s without dropping any : http://www.balabit.com/network-security/syslog-ng Regards, Willy
RE: syslogd dropping requests from haproxy
Thanks Willy - I will these and let you know. -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Saturday, October 15, 2011 11:32 AM To: Sachin Shetty Cc: haproxy@formilux.org Subject: Re: syslogd dropping requests from haproxy On Sat, Oct 15, 2011 at 01:35:46AM +0530, Sachin Shetty wrote: > Found some more info, when haproxy configured to send logs to remote host > instead of local syslogd, it works fine. Definately something to do with the > local syslogd under heavy load. Check that your syslog doesn't log synchronously. For the basic sysklogd, it means you need to have a "-" in front of your file names. And even with this, sysklogd's limits are quickly reached (between 300 and 1000 logs/s depending on the machine). For high loads, I strongly recommend syslog-ng. It's the only one I found which managed to log more than 1 logs/s without dropping any : http://www.balabit.com/network-security/syslog-ng Regards, Willy
Re: syslogd dropping requests from haproxy
Willy, For high loads, I strongly recommend syslog-ng. It's the only one I found which managed to log more than 1 logs/s without dropping any : http://www.balabit.com/network-security/syslog-ng Have you tested rsyslog? I've had bad experiences with syslog-ng dropping messages in the past, though the logging configuration was fairly complex. It was at least able to report how many messages had been dropped. This was with an older version of syslog-ng as well. -JohnF
Re: syslogd dropping requests from haproxy
On Sat, Oct 15, 2011 at 09:33:56AM -0400, John Marrett wrote: > Willy, > > >For high loads, I strongly recommend syslog-ng. It's the only one I > >found which managed to log more than 1 logs/s without dropping > >any : > > > > http://www.balabit.com/network-security/syslog-ng > > > Have you tested rsyslog? I've had bad experiences with syslog-ng > dropping messages in the past, though the logging configuration was > fairly complex. Yes I've already tested rsyslog and was not convinced at all, although several people reported good results. It offers nice features but its config is quite cryptic to me so I tend not to use it. > It was at least able to report how many messages had been dropped. This is quite important. That's why I want to add a log ID to haproy, so that it becomes obvious what log is missing. Regards, Willy
RE: syslogd dropping requests from haproxy
We switched to rsyslog and since then seeing a huge increase in the log volume. Thanks for all the help! Thanks Sachin -Original Message- From: Sachin Shetty [mailto:sshe...@egnyte.com] Sent: Saturday, October 15, 2011 11:35 AM To: 'Willy Tarreau' Cc: 'haproxy@formilux.org' Subject: RE: syslogd dropping requests from haproxy Thanks Willy - I will these and let you know. -Original Message- From: Willy Tarreau [mailto:w...@1wt.eu] Sent: Saturday, October 15, 2011 11:32 AM To: Sachin Shetty Cc: haproxy@formilux.org Subject: Re: syslogd dropping requests from haproxy On Sat, Oct 15, 2011 at 01:35:46AM +0530, Sachin Shetty wrote: > Found some more info, when haproxy configured to send logs to remote host > instead of local syslogd, it works fine. Definately something to do with the > local syslogd under heavy load. Check that your syslog doesn't log synchronously. For the basic sysklogd, it means you need to have a "-" in front of your file names. And even with this, sysklogd's limits are quickly reached (between 300 and 1000 logs/s depending on the machine). For high loads, I strongly recommend syslog-ng. It's the only one I found which managed to log more than 1 logs/s without dropping any : http://www.balabit.com/network-security/syslog-ng Regards, Willy
