Hi Joe, On Thu, Jan 12, 2012 at 08:40:01PM -0500, Joseph Hardeman wrote: > Hey Chris, > > What flavor of linux will you be putting syslog-ng on? Be sure the > syslog-ng you install can handle multi-threading of its processes, so > version 3.0 or newer I believe, otherwise it will eat up all of 1 CPU and > could most certainly lose logs then if you have a lot of traffic going > through haproxy.
My experience with syslog-ng has already been extremely good since version 1.4 around 10 years ago. I remember reaching 20000 logs per second with zero losses on a pentium-3 933 MHz. You need to tune it to use large buffers to cover disk latency, and that's all. Syslog-ng is an excellent piece of software, which is why I always recommend it to everyone who needs high logging rates. > We have it setup for one of our customers now, actually I just finished > setting it up four days ago and I have syslog-ng splitting out logs per > hour. I don't really see much in the way of missing logs, if anything they > now have more information than they were getting for the visits to their > site from Google Analytics. > > But just as an idea, using "option httplog clf" in the listen section for > mode http, yesterday I receiving around 12G of logs from a single haproxy > box while today they are at 4.9G and the day isn't over yet. So today may > end up around 10G as the west coast is now getting off of work. And the > clf option sends through less data than the normal option httplog so the > amount of data is a bit lower than if you log normal logs from haproxy. This point surprizes me a little bit because CLF logs contain the same info with more delimiters. Maybe they compress better but I'm surprized you find them smaller. For instance : normal: Jan 13 07:52:50 pcw haproxy[839]: 127.0.0.1:56837 [13/Jan/2012:07:52:46.258] echo echo/<NOSRV> 0/0/0/3325/3789 200 14 - - ---- 0/0/0/0/0 0/0 "GET / HTTP/1.1" clf: Jan 13 07:52:34 pcw haproxy[834]: 127.0.0.1 - - [13/Jan/2012:06:52:31 +0000] "GET / HTTP/1.1" 200 14 "-" "-" 56835 759 "echo" "echo" "<NOSRV>" 0 0 0 2285 2845 "----" 0 0 0 0 0 0 0 "-" "-" Regards, Willy
