Hello,
We are working on a way to collect access haproxy on hadoop hbase[1] via
flume[2].
- haproxy logs on 127.0.0.1:5140
- flumeagent runs on the same host with rules : syslogUdp(5140) |
agentBEChain( "collector1", "collector2" )
=> run locally to prevent maximum of UDP paquet loss, listen on syslog
port 5140 and send event with failover on host collector1 or collector2
- collector1 and collector2 runs rules : collectorSource | digest("MD5",
"digest",base64=true) haproxyLogExtractor() insistentOpen hbase("logs",
"_f:%{haproxy.frontend}_t:%Y%m%d%H%M%S_b:%{haproxy.backend}_s:%{haproxy.statusCode}-%{digest}",
"body", "body", "%{body}")
=> listen as Flume collector, create md5sum of the log as attribute
"digest" extract haproxy data and store the event in hbase table "logs",
with rowkey as _f:front_t:date_b:backend_s:status--digest
Row key format is important to use the native index of hbase on it. (filter
by front+time is easy so). Digest goal is to prevent from doublons.
haproxyLogExtractor() is a special plugins we created [3] to "augment" the
syslog flume event with haproxy meta data
Maybe bigdata is overkill for you :) but the technology is amazing and
you'll be prepared for success ;)
[1] http://hbase.apache.org/
[2] http://www.cloudera.com/blog/category/flume/ (note that the Flume-NG in
on its way as apache incubator project https://cwiki.apache.org/FLUME/
[3] https://github.com/figarocms/flume-haproxy-extractor
Best regards,
--
Damien
2012/1/12 Chris Miller <ct...@scratchspace.com>
>
> We're looking to utilize access logs from HAProxy rather than from
> the backend application servers. It appears we can set logging
> directives to one syslog host per "listen" directive, this doesn't
> really help us split into separate logs per host. One thought it to
> use syslog-ng which has filters that would allow this, but at an
> unknown overhead for a high traffic load balancer. Before we
> reinvent the wheel, I just wanted to see if anyone has a recommended
> way of addressing this. I was unable to find anything on Google...
>
> Regards,
> Chris
>
> Chris Miller
> President - Rocket Scientist
> ScratchSpace Inc.
> (831) 621-7928
> http://www.scratchspace.com
>
>
>
