Re: Virtual Hosting and logs
Hello,
We are working on a way to collect access haproxy on hadoop hbase[1] via
flume[2].
- haproxy logs on 127.0.0.1:5140
- flumeagent runs on the same host with rules : syslogUdp(5140) |
agentBEChain( "collector1", "collector2" )
=> run locally to prevent maximum of UDP paquet loss, listen on syslog
port 5140 and send event with failover on host collector1 or collector2
- collector1 and collector2 runs rules : collectorSource | digest("MD5",
"digest",base64=true) haproxyLogExtractor() insistentOpen hbase("logs",
"_f:%{haproxy.frontend}_t:%Y%m%d%H%M%S_b:%{haproxy.backend}_s:%{haproxy.statusCode}-%{digest}",
"body", "body", "%{body}")
=> listen as Flume collector, create md5sum of the log as attribute
"digest" extract haproxy data and store the event in hbase table "logs",
with rowkey as _f:front_t:date_b:backend_s:status--digest
Row key format is important to use the native index of hbase on it. (filter
by front+time is easy so). Digest goal is to prevent from doublons.
haproxyLogExtractor() is a special plugins we created [3] to "augment" the
syslog flume event with haproxy meta data
Maybe bigdata is overkill for you :) but the technology is amazing and
you'll be prepared for success ;)
[1] http://hbase.apache.org/
[2] http://www.cloudera.com/blog/category/flume/ (note that the Flume-NG in
on its way as apache incubator project https://cwiki.apache.org/FLUME/
[3] https://github.com/figarocms/flume-haproxy-extractor
Best regards,
--
Damien
2012/1/12 Chris Miller
>
> We're looking to utilize access logs from HAProxy rather than from
> the backend application servers. It appears we can set logging
> directives to one syslog host per "listen" directive, this doesn't
> really help us split into separate logs per host. One thought it to
> use syslog-ng which has filters that would allow this, but at an
> unknown overhead for a high traffic load balancer. Before we
> reinvent the wheel, I just wanted to see if anyone has a recommended
> way of addressing this. I was unable to find anything on Google...
>
> Regards,
>Chris
>
> Chris Miller
> President - Rocket Scientist
> ScratchSpace Inc.
> (831) 621-7928
> http://www.scratchspace.com
>
>
>
Re: Flume plugin to extract haproxy log data
Grumbl !!! s/geoip/haproxy/ of course. Source of my confusion : we have released another decorator plugin for Flume to geotag IP from a flume event attribute (like haproxy.clientIP for example :) based on geoip ( http://www.maxmind.com/app/ip-location ) Could be found here https://github.com/figarocms/flume-geoip-extractor. Cheers, -- Damien 2011/12/5 Damien Hardy > Hello, > > Following my mail about log format a couple of days ago (thank you for > answers) , we have released a plugin for Apache Flume ( > https://cwiki.apache.org/FLUME/ ) a former Cloudera project ( > https://ccp.cloudera.com/display/CDHDOC/Flume+Installation ). > > Here it is : https://github.com/figarocms/flume-haproxy-extractor > > jar archives could be found on https://oss.sonatype.org/ (searching for > "geoip" for example) you need also org.kohsuke.geoip too. > > The aim of this plugin is to extend the Flume event generated by the > syslog of haproxy with new attributes for each fields of the haproxy HTTP > log. > > Those fields could be used after in sink (hbase rowid or collumn) > elasticsearch document whatever. > > Feel free to review it and comment. > > Cheers, > > -- > Damien >
Flume plugin to extract haproxy log data
Hello, Following my mail about log format a couple of days ago (thank you for answers) , we have released a plugin for Apache Flume ( https://cwiki.apache.org/FLUME/ ) a former Cloudera project ( https://ccp.cloudera.com/display/CDHDOC/Flume+Installation ). Here it is : https://github.com/figarocms/flume-haproxy-extractor jar archives could be found on https://oss.sonatype.org/ (searching for "geoip" for example) you need also org.kohsuke.geoip too. The aim of this plugin is to extend the Flume event generated by the syslog of haproxy with new attributes for each fields of the haproxy HTTP log. Those fields could be used after in sink (hbase rowid or collumn) elasticsearch document whatever. Feel free to review it and comment. Cheers, -- Damien
Re: HTTP Log format unconstant
Hello Cyril,
Hmm, I read the sources again searching on clf format and seeing the
timezone is forced to "+" in this particular format bring me some
doubts ...
I suppose that "get_gmtime(s->logs.accept_date.tv_sec, &tm);" (GMT ?) is
the same every where on the planet :)
So no need to get the Timezone anymore :)
Second problem solved today ... You rocks !
Thank you,
--
Damien
2011/11/30 Cyril Bonté
> Hi Damien,
>
> Le mercredi 30 novembre 2011 10:50:04, Damien Hardy a écrit :
> > Hello,
> >
> > An other point that could be enhanced in the log format is the date
> format:
> > we miss the timezone in IT
>
> Did you give a try to "option httplog clf" ?
>
> >
> > Best regards,
> >
> > > Hello haproxy-list,
> > >
> > > We are working on near real time web statistics architecture based on
> the
> > > haproxy access logs.
> > > (a first shoot of a flume plugin is available here :
> > > https://github.com/figarocms/flume-haproxy-extractor it is now
> designed
> > > to feet or needs especially but we are working on making it available
> for
> > > more general pupose)
> > >
> > > But HTTP logs are quite difficult to parse especially if there is
> captured
> > > headers:
> > > * Request and response headers could be captured enclosed both by {},
> but
> > > if you capture only one of them in a frontend there is no way to now
> if it
> > > is the request or the response header. because if the capture is not
> set
> > > the field is not set either in teh log.
> > > My suggestion is maybe to add a letter before "{" to precise if it's
> query
> > > ( ex : "Q{" ) or response ( "R{" ) set of captured headers. or always
> make
> > > them appear.
> > >
> > > * Olso the header captured is not precised in it, only values appear
> so
> > > you have olso to know the order of capture in the configuration to
> parse it
> > > well.
> > > Maybe the whole line of the header could be set in the field.
> > >
> > > * The headers are separated by a "|" but the eventuelly "|" allready
> > > present in the headers are not escaped, that could occure some
> probleme of
> > > parsing I suppose.
> > > No really answer to address this probleme but maybe enclosing headers
> by "
> > > (and escaped this char it like many other String method could do)
> could be
> > > a good envolvment.
> > >
> > > And As I precised it before syslog default size of 1024 is very short
> for
> > > today statistical need. I had to increase the constant to 4096
> (capture of
> > > full UA, and Referer are costly).
> > >
> > > Best regards,
> > >
> > > --
> > > Damien
>
> --
> Cyril Bonté
>
Re: HTTP Log format unconstant
[...] > >> * The headers are separated by a "|" but the eventuelly "|" allready >> present in the headers are not escaped, that could occure some probleme of >> parsing I suppose. >> No really answer to address this probleme but maybe enclosing headers by >> " (and escaped this char it like many other String method could do) could >> be a good envolvment. >> > My mistake ! I read the source with more attention and see that special characters are escaped for the capture header fields in "void init_proto_http()" This problem is [solved] \o/ that was fast ! Best regards, -- Damien
Re: HTTP Log format unconstant
Hello,
An other point that could be enhanced in the log format is the date format:
we miss the timezone in IT
Best regards,
--
Damien
2011/11/29 Damien Hardy
> Hello haproxy-list,
>
> We are working on near real time web statistics architecture based on the
> haproxy access logs.
> (a first shoot of a flume plugin is available here :
> https://github.com/figarocms/flume-haproxy-extractor it is now designed
> to feet or needs especially but we are working on making it available for
> more general pupose)
>
> But HTTP logs are quite difficult to parse especially if there is captured
> headers:
> * Request and response headers could be captured enclosed both by {}, but
> if you capture only one of them in a frontend there is no way to now if it
> is the request or the response header. because if the capture is not set
> the field is not set either in teh log.
> My suggestion is maybe to add a letter before "{" to precise if it's query
> ( ex : "Q{" ) or response ( "R{" ) set of captured headers. or always make
> them appear.
>
> * Olso the header captured is not precised in it, only values appear so
> you have olso to know the order of capture in the configuration to parse it
> well.
> Maybe the whole line of the header could be set in the field.
>
> * The headers are separated by a "|" but the eventuelly "|" allready
> present in the headers are not escaped, that could occure some probleme of
> parsing I suppose.
> No really answer to address this probleme but maybe enclosing headers by "
> (and escaped this char it like many other String method could do) could be
> a good envolvment.
>
> And As I precised it before syslog default size of 1024 is very short for
> today statistical need. I had to increase the constant to 4096 (capture of
> full UA, and Referer are costly).
>
> Best regards,
>
> --
> Damien
>
>
>
>
>
>
>
>
>
>
HTTP Log format unconstant
Hello haproxy-list,
We are working on near real time web statistics architecture based on the
haproxy access logs.
(a first shoot of a flume plugin is available here :
https://github.com/figarocms/flume-haproxy-extractor it is now designed to
feet or needs especially but we are working on making it available for more
general pupose)
But HTTP logs are quite difficult to parse especially if there is captured
headers:
* Request and response headers could be captured enclosed both by {}, but
if you capture only one of them in a frontend there is no way to now if it
is the request or the response header. because if the capture is not set
the field is not set either in teh log.
My suggestion is maybe to add a letter before "{" to precise if it's query
( ex : "Q{" ) or response ( "R{" ) set of captured headers. or always make
them appear.
* Olso the header captured is not precised in it, only values appear so
you have olso to know the order of capture in the configuration to parse it
well.
Maybe the whole line of the header could be set in the field.
* The headers are separated by a "|" but the eventuelly "|" allready
present in the headers are not escaped, that could occure some probleme of
parsing I suppose.
No really answer to address this probleme but maybe enclosing headers by "
(and escaped this char it like many other String method could do) could be
a good envolvment.
And As I precised it before syslog default size of 1024 is very short for
today statistical need. I had to increase the constant to 4096 (capture of
full UA, and Referer are costly).
Best regards,
--
Damien
Re: Increase log size in config.
By the way, haproxy could be configured with default max size at 1024 and if we want to increase it by config it may be available (as long as we are not using a real syslog server). My concern is about network device with UDP frame bigger than 1024 if anybody is aware of potential probleme that could occure 2011/9/8 Graeme Donaldson > Hi Damien > > I may be wrong, but as far as I can gather the 1024-byte limitation is > imposed by the syslog protocol RFC (http://www.faqs.org/rfcs/rfc3164.html) > and not HAproxy itself. > > Regards, > Graeme. > > On 8 September 2011 16:09, Damien Hardy wrote: > >> Hello there. >> >> We are working on our realtime statistics of consultation platform based >> on haproxy logs. >> Our probleme is that logs can capture HTTP headers but are limited to a >> 1024 length string witch is very short if Host: Referer: and User-Agent: are >> captured as we are doing. >> >> Is it planned to set the max log size by config parameter. >> >> Do you now if there is any side effects increasing this value and rebuild >> a package ? >> The syslog server is a cloudera flume node located on the same server with >> haproxy (flume is not able to read in a Unix socket as far as I know). >> >> Thank you. >> >> Best regards, >> >> -- >> Damien >> >> >> >
Re: Increase log size in config.
Hi Graeme, You are rigth so maybe it could be great that haproxy could log the full entry by other way than syslog Using the Unix socket allready existing could be tailed by any external program I suppose, and shouldn't be limited by a 1024 string maybe. Regards, -- Damien 2011/9/8 Graeme Donaldson > Hi Damien > > I may be wrong, but as far as I can gather the 1024-byte limitation is > imposed by the syslog protocol RFC (http://www.faqs.org/rfcs/rfc3164.html) > and not HAproxy itself. > > Regards, > Graeme. > > On 8 September 2011 16:09, Damien Hardy wrote: > >> Hello there. >> >> We are working on our realtime statistics of consultation platform based >> on haproxy logs. >> Our probleme is that logs can capture HTTP headers but are limited to a >> 1024 length string witch is very short if Host: Referer: and User-Agent: are >> captured as we are doing. >> >> Is it planned to set the max log size by config parameter. >> >> Do you now if there is any side effects increasing this value and rebuild >> a package ? >> The syslog server is a cloudera flume node located on the same server with >> haproxy (flume is not able to read in a Unix socket as far as I know). >> >> Thank you. >> >> Best regards, >> >> -- >> Damien >> >> >> >
Increase log size in config.
Hello there. We are working on our realtime statistics of consultation platform based on haproxy logs. Our probleme is that logs can capture HTTP headers but are limited to a 1024 length string witch is very short if Host: Referer: and User-Agent: are captured as we are doing. Is it planned to set the max log size by config parameter. Do you now if there is any side effects increasing this value and rebuild a package ? The syslog server is a cloudera flume node located on the same server with haproxy (flume is not able to read in a Unix socket as far as I know). Thank you. Best regards, -- Damien
Fwd: erratic X-Forwarded-For patch for stunnel
(this ML need a reply-to header :)
For the conclusion :
-- Forwarded message --
From: Damien Hardy
Date: 2011/8/5
Subject: Re: erratic X-Forwarded-For patch for stunnel
To: Guillaume Bourque
Good point for you.
I was running with option http-server-close as global configuration.
Now with option httpclose it get the X-Forwarded-For for every request.
Thank you a lot.
--
Damien
2011/8/5 Guillaume Bourque
> **
> Hi,
>
> are you using httpclose in haproxy in the frontend for the ssl portion of
> haproxy ? Willy has talk about other ways to solve this yesterday but just
> to do a test you could put option httpclose in this frontend.
>
> "most of the time there is only 192.168.134.222 the IP of haproxy)" It's
> the ip of stunnel to I imagine ?
>
> You can see more then 1 X-Forwarded-For in the log it's cumulative... But
> you can tell haproxy to not include X-Forwarded-For when stunnel already put
> the client ip with a option like this:
>
> option forwardfor except 10.222.0.0/27
>
> for me this is the subnet of the ssl offloader 10.222.0.0/27.
>
> SO the way I understand it
>
>
> Client - stunnel (add the client ip to X-For) ---
> haproxy (will not add X-For) - apache1
>
> - apache2
>
>
> Client -- ---
> haproxy (will add X-For) - apache1
>
> - apache2
>
> Then you need to decide if you will be using option httpclose or what was
> discuss yesterday
>
> From Willy;
>
> So if you need stunnel to provide the IP to haproxy, you have two
> solutions :
> - either disable keep-alive using "option httpclose" on haproxy so that
> it forces stunnel to reopen a new connection for each request and to
> add the header to each of them ;
>
> - or make use of the "send-proxy" patch for stunnel, which is compatible
> with the "accept-proxy" feature of haproxy. This is the preferred solution
> because instead of mangling the beginning of the HTTP request, stunnel
> then informs haproxy about the source address in an out-of-band fashion,
> which makes it compatible with keep-alive.
>
>
> Bye.
>
> e 2011-08-05 05:45, Damien Hardy a écrit :
>
> Hello,
>
> I patched the debian stunnel4 package for squeeze
>
> # aptitude install devscripts build-essential fakeroot
> # apt-get build-dep stunnel4
> # apt-get source stunnel4
> # wget
> http://haproxy.1wt.eu/download/patches/stunnel-4.29-xforwarded-for.diff
> # cd stunnel4-4.29/
> # patch -p1 -i ../stunnel-4.29-xforwarded-for.diff
> # debuild -us -uc
> # dpkg -i ../stunnel4_4.29-1_amd64.deb
>
> change my conf /etc/stunnel/stunnel.conf as :
> [...]
> [https]
> accept = 192.168.134.222:443
> connect = 192.168.134.222:4430
> TIMEOUTclose = 0
> xforwardedfor = yes
>
> change my conf /etc/haproxy/haproxy.conf as :
> listen sslsite
> bind 192.168.134.222:4430
> balance roundrobin
> cookie SRV insert indirect nocache
> capture request header X-Forwarded-For len 256
> rspirep ^Location:\ http://(.*)Location:\ https://\1
> server vexft04 192.168.16.55:80 cookie ahD2Fiel check inter 5000 fall
> 3
> server vexft05 192.168.16.50:80 cookie ifaop7Ge check inter 5000 fall
> 3
> server vexft06 192.168.128.52:80 cookie aina1oRo check inter 5000
> fall 3
> server vexft07 192.168.128.53:80 cookie ohQuai5g check inter 5000
> fall 3
>
> But X-Forwarded-For header is inconstantly set in logs as :
>
> Aug 5 11:23:54 haproxy[8423]:
> 192.168.134.222:43889[05/Aug/2011:11:23:54.218] sslsite sslsite/vexft04
> 0/0/0/250/250 200 3865 -
> - --NI 1/1/0/1/0 0/0 {10.147.28.20} "GET /admin/AdmInscriptionPro.shtml
> HTTP/1.1"
> Aug 5 11:23:54 haproxy[8423]:
> 192.168.134.222:43889[05/Aug/2011:11:23:54.468] sslsite sslsite/vexft04
> 31/0/1/1/33 200 471 - -
> --VN 1/1/0/1/0 0/0 {} "GET /css/admin/master.css HTTP/1.1"
> Aug 5 11:23:54 haproxy[8423]:
> 192.168.134.222:43889[05/Aug/2011:11:23:54.502] sslsite sslsite/vexft04
> 173/0/0/5/178 200 2018 -
> - --VN 1/1/0/1/0 0/0 {} "GET /css/lightwindow.css HTTP/1.1"
> Aug 5 11:23:54 haproxy[8423]:
> 192.168.134.222:43889[05/Aug/2011:11:23:54.680] sslsite sslsite/vexft04
> 56/0/1/1/58 200 573 - -
> --VN 1/1/0/1/0 0/0 {} "GET /css/sIFR-screen.css HTTP/1.1"
> Aug 5 11:23:54 haproxy[8423]:
> 192.168.134.222:43889[05/Aug/2011:11:23:54.739] sslsite sslsite/vexft04
> 64/0/1/1/66 200 722 - -
> --VN 1/1/0/1/0 0/0 {} "GET /css/niftyCorners.css HTTP/1.1"
> Aug 5 11:23:54 haproxy[8423]:
> 192.168.134.222:
erratic X-Forwarded-For patch for stunnel
Hello,
I patched the debian stunnel4 package for squeeze
# aptitude install devscripts build-essential fakeroot
# apt-get build-dep stunnel4
# apt-get source stunnel4
# wget
http://haproxy.1wt.eu/download/patches/stunnel-4.29-xforwarded-for.diff
# cd stunnel4-4.29/
# patch -p1 -i ../stunnel-4.29-xforwarded-for.diff
# debuild -us -uc
# dpkg -i ../stunnel4_4.29-1_amd64.deb
change my conf /etc/stunnel/stunnel.conf as :
[...]
[https]
accept = 192.168.134.222:443
connect = 192.168.134.222:4430
TIMEOUTclose = 0
xforwardedfor = yes
change my conf /etc/haproxy/haproxy.conf as :
listen sslsite
bind 192.168.134.222:4430
balance roundrobin
cookie SRV insert indirect nocache
capture request header X-Forwarded-For len 256
rspirep ^Location:\ http://(.*)Location:\ https://\1
server vexft04 192.168.16.55:80 cookie ahD2Fiel check inter 5000 fall 3
server vexft05 192.168.16.50:80 cookie ifaop7Ge check inter 5000 fall 3
server vexft06 192.168.128.52:80 cookie aina1oRo check inter 5000 fall
3
server vexft07 192.168.128.53:80 cookie ohQuai5g check inter 5000 fall
3
But X-Forwarded-For header is inconstantly set in logs as :
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.218] sslsite
sslsite/vexft04 0/0/0/250/250 200 3865 -
- --NI 1/1/0/1/0 0/0 {10.147.28.20} "GET /admin/AdmInscriptionPro.shtml
HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.468] sslsite
sslsite/vexft04 31/0/1/1/33 200 471 - -
--VN 1/1/0/1/0 0/0 {} "GET /css/admin/master.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.502] sslsite
sslsite/vexft04 173/0/0/5/178 200 2018 -
- --VN 1/1/0/1/0 0/0 {} "GET /css/lightwindow.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.680] sslsite
sslsite/vexft04 56/0/1/1/58 200 573 - -
--VN 1/1/0/1/0 0/0 {} "GET /css/sIFR-screen.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.739] sslsite
sslsite/vexft04 64/0/1/1/66 200 722 - -
--VN 1/1/0/1/0 0/0 {} "GET /css/niftyCorners.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.805] sslsite
sslsite/vexft04 3/0/1/11/16 200 28961 - -
--VN 1/1/0/1/0 0/0 {} "GET /script/aculous/prototype.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43922[05/Aug/2011:11:23:54.832] sslsite
sslsite/vexft04 0/0/0/1/1 200 2071 - -
--VN 4/4/3/4/0 0/0 {10.147.28.20} "GET /script/espace-pro.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43920[05/Aug/2011:11:23:54.831] sslsite
sslsite/vexft04 0/0/0/2/2 200 1811 - -
--VN 4/4/2/3/0 0/0 {10.147.28.20} "GET /script/niftyCorners.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43924[05/Aug/2011:11:23:54.832] sslsite
sslsite/vexft04 0/0/0/2/2 200 739 - -
--VN 6/6/3/4/0 0/0 {10.147.28.20} "GET /script/niftyDeclare.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43928[05/Aug/2011:11:23:54.834] sslsite
sslsite/vexft04 0/0/0/1/1 200 604 - -
--VN 6/6/2/3/0 0/0 {10.147.28.20} "GET /script/admin/menu_admin.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.821] sslsite
sslsite/vexft04 7/0/0/7/14 200 13798 - -
--VN 6/6/2/3/0 0/0 {} "GET /script/aculous/lightwindow.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43926[05/Aug/2011:11:23:54.833] sslsite
sslsite/vexft04 0/0/0/3/3 200 2640 - -
--VN 6/6/1/2/0 0/0 {10.147.28.20} "GET /script/espace-admin.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43922[05/Aug/2011:11:23:54.833] sslsite
sslsite/vexft04 2/0/0/1/3 200 945 - -
--VN 6/6/2/3/0 0/0 {} "GET /script/recherche/SearchLightWindow.js HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43924[05/Aug/2011:11:23:54.835] sslsite
sslsite/vexft04 2/0/1/1/4 200 810 - -
--VN 6/6/2/3/0 0/0 {} "GET /css/admin/typo.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43928[05/Aug/2011:11:23:54.835] sslsite
sslsite/vexft04 2/0/0/1/3 200 1138 - -
--VN 6/6/2/3/0 0/0 {} "GET /css/admin/lists.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43920[05/Aug/2011:11:23:54.833] sslsite
sslsite/vexft04 3/0/1/1/5 200 1617 - -
--VN 6/6/2/3/0 0/0 {} "GET /css/admin/layout.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43926[05/Aug/2011:11:23:54.837] sslsite
sslsite/vexft04 2/0/0/1/3 200 2914 - -
--VN 6/6/2/3/0 0/0 {} "GET /css/admin/navbar.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43922[05/Aug/2011:11:23:54.837] sslsite
sslsiteo/vexft04 2/0/0/1/3 200 1726 - -
--VN 6/6/1/2/0 0/0 {} "GET /css/admin/forms.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43924[05/Aug/2011:11:23:54.839] sslsite
sslsite/vexft04 2/0/0/1/3 200 669 - -
--VN 6/6/3/4/0 0/0 {} "GET /css/niftyDeclare.css HTTP/1.1"
Aug 5 11:23:54 haproxy[8423]:
192.168.134.222:43889[05/Aug/2011:11:23:54.836] sslsite
sslsite/vexft04 4/0/1/1/6 200 1740 - -
--VN 6/6/3/4/0 0/0 {} "GET /css/admin/ventre_general.
Re: auto reply 200 on stats URL just for logs
2011/7/12 Willy Tarreau
> On Tue, Jul 12, 2011 at 10:42:31AM +0200, Damien Hardy wrote:
> > 2011/7/12 Willy Tarreau
> >
> > >
> > > Then you can use the errorfile method : you send your request to a
> > > backend without any server (which will produce an error 503) and you
> > > use "errorfile 503 /path/to/file" to return this file instead of the
> > > error 503. Some people use this to serve a few static files.
> > >
> >
> > If it reply 200 it could be the good thing
>
> The file is HTTP and not HTML, so you put the status code you want in
> it with whatever header you like.
>
> > > > Maybe haproxy could send the well know tiny "transparent pixel" on
> > > > demand without even go through caching servers. #featurerequest :)
> > >
> > > I'm not sure what well known method you're speaking about. However,
> > > the "return" feature is planned in the roadmap in order to be able
> > > to return static contents or a specific file. It's just not implemented
> > > yet.
> > >
> >
> > This is exactly the feature I was thinking (transparent pixel is a
> classic
> > small resources used to make design fit well or getting some stats via
> logs
> > like google analitics or xity tags)
> > called with an html img tag it works even when javascript is disabled :)
>
> OK I did not understand you were talking about that.
>
> Regards,
> Willy
>
>
I have done some tests.
The errorfile 503 directive on empty backend do the job from the client
point of view but to be very perfect I would like there is no 503 error code
in my haproxy logs.
Jul 12 15:11:41 ip6-localhost haproxy[16185]:
10.147.28.20:49423[12/Jul/2011:15:11:41.362] varnish tag/
0/-1/-1/-1/0 503 190 - - SC--
0/0/0/0/0 0/0 {vhost||Mozilla/5.0 (X11; Linux x86_64; rv:5.0) Gecko/20100101
Firefox/5.0} "GET
/stat?action=view&resources=123+456+789(&session=423)(&user=p...@gmail.com)
HTTP/1.1"
Thank you,
Waiting for the "return" feature.
Regards,
--
Damien
Re: auto reply 200 on stats URL just for logs
2011/7/12 Willy Tarreau > > Then you can use the errorfile method : you send your request to a > backend without any server (which will produce an error 503) and you > use "errorfile 503 /path/to/file" to return this file instead of the > error 503. Some people use this to serve a few static files. > If it reply 200 it could be the good thing > > > Maybe haproxy could send the well know tiny "transparent pixel" on > > demand without even go through caching servers. #featurerequest :) > > I'm not sure what well known method you're speaking about. However, > the "return" feature is planned in the roadmap in order to be able > to return static contents or a specific file. It's just not implemented > yet. > This is exactly the feature I was thinking (transparent pixel is a classic small resources used to make design fit well or getting some stats via logs like google analitics or xity tags) called with an html img tag it works even when javascript is disabled :) Cdt, -- Damien
Re: auto reply 200 on stats URL just for logs
Le 11/07/2011 22:07, Willy Tarreau a écrit : > On Mon, Jul 11, 2011 at 02:20:37PM +0200, Damien Hardy wrote: >> Whatever method is used, I want that haproxy never ask to any backend an >> answer directly to client and log the request like others > > Then define a "monitor-uri", it's done for that. > > Example : > >frontend XXX >mode http >monitor-uri /test > > All checks to /test will return a 200 OK. Its used so that haproxy can be > tested by other components (eg: keepalived). > > Regards, > Willy > Yes but if I refer to the doc there is no log entries when someone request monitor uri and this is what à want for my stats platform. Maybe haproxy could send the well know tiny "transparent pixel" on demand without even go through caching servers. #featurerequest :) Regards, -- Damien
Re: auto reply 200 on stats URL just for logs
Whatever method is used, I want that haproxy never ask to any backend an answer directly to client and log the request like others best regards, -- Damien 2011/7/11 Baptiste > hi, > > There is a HTTP method for that: HEAD. > > cheers > > On Mon, Jul 11, 2011 at 11:21 AM, Damien Hardy > wrote: > > I have to precise that this is not related to the "stats" delivered by > > haproxy but a static resource used on our pages to get counters based on > > access logs provide by haproxy. > > > > 2011/7/11 Damien Hardy > >> > >> Hello, > >> > >> I want haproxy to reply 200 with no data on a predefined URL without > going > >> through backends just for getting logs with referrer for our stats > platform. > >> > >> Is there a way to do that with haproxy directives (redirect with code > 200 > >> maybe) ? > >> > >> Best regards, > >> > >> -- > >> Damien > > > > >
Re: auto reply 200 on stats URL just for logs
I have to precise that this is not related to the "stats" delivered by haproxy but a static resource used on our pages to get counters based on access logs provide by haproxy. 2011/7/11 Damien Hardy > Hello, > > I want haproxy to reply 200 with no data on a predefined URL without going > through backends just for getting logs with referrer for our stats platform. > > Is there a way to do that with haproxy directives (redirect with code 200 > maybe) ? > > Best regards, > > -- > Damien >
auto reply 200 on stats URL just for logs
Hello, I want haproxy to reply 200 with no data on a predefined URL without going through backends just for getting logs with referrer for our stats platform. Is there a way to do that with haproxy directives (redirect with code 200 maybe) ? Best regards, -- Damien
Re: Parsing httplog with java
Yes it could help us. I precised Java because there is no way in that language to tag regex captures like you have done in it (according to my deskmate). I transmit it now to the dev :) hope that could help us. many thanks, -- Damien 2011/7/4 Holger Just > Hi Damien, > > On 2011-07-04 14:34, Damien Hardy wrote: > > Does anyone have ever done the regex to parse the haproxy apachelog. > > (we want to inject logs in hbase via flume :) > > although it's not directly targeted for Java, but written in Python, but > I have already posted my approach of parsing the HAProxy HTTP logs to > this list some time ago. See > http://permalink.gmane.org/gmane.comp.web.haproxy/5320 for the > boilerplate script including the regex. > > Hope that helps, > Holger > >
Re: Parsing httplog with java
Not really, it's "common" log format not "combined", I need to capture referrer and user agent that is added in other place in the raw of haproxy log (with a specific format). And spécific data of haproxy could be interresting too. And by the way awstat cannot permit to store logs forever to requets new stats like Hbase. The idea is to do something like http://blog.mozilla.com/data/2011/03/22/how-glow-mozilla-org-gets-its-data/or http://hadoopblog.blogspot.com/2011/07/realtime-hadoop-usage-at-facebook.html Thanks you for helping. -- Damien 2011/7/4 Chris Sarginson > Can you pull it from another app such as AWstats? It's just a standard log > format? > > Chris > > Sent from my iPhone > > On 4 Jul 2011, at 13:34, Damien Hardy wrote: > > > Hi there, > > > > Does anyone have ever done the regex to parse the haproxy apachelog. > > (we want to inject logs in hbase via flume :) > > > > thank you. > > > > Cdt > > > > -- > > Damien > >
Parsing httplog with java
Hi there, Does anyone have ever done the regex to parse the haproxy apachelog. (we want to inject logs in hbase via flume :) thank you. Cdt -- Damien
Log parts of POST data ?
Hi there, Is there any chance that I could log a part of the data in a POST request. I want to do some asynchronous stats (with mongodb) against Http log files. getting GET parameters from uri is easy. But it's quite hard to get POST parameter (even the N first char for example). Haproxy is able to search for stickyness (balance uri) in the POST data so maybe it is possible to log it ? Thank you for your answer. Regards, -- Damien
Re: Redirect 1 time per day
> > Hi all, > > I have a request from my users: > > We want iphones (hdr_sub(user-agent) -i iphone) to be redirected to Apple store the first time thez come on our site. but only 1 time per days. > > So I want to user the "redirect" directive > > acl iphone hdr_sub(User-Agent) -i iphone > acl seen hdr_sub(cookie) SEEN=1 > redirect location http://itunes.apple.com/us/app/myapp/ set-cookie SEEN=1 if iphone !seen > > But how to set the cookie : expires=/TOMOROW/ ? > > I use haproxy 1.3 packaged by debian lenny. > > Thanks a lot. > > Best regards, > Hmm I have much problems : here is my version of haproxy : ii haproxy 1.3.15.2-2+lenny2 fast and reliable load balancing reverse proxy when I try to load my new conf : ha03:~# /etc/init.d/haproxy reload Reloading haproxy: haproxy[ALERT] 137/165103 (27677) : parsing [/etc/haproxy/haproxy.cfg:117] : unknown keyword 'redirect' in 'listen' section [ALERT] 137/165103 (27677) : Error reading configuration file : /etc/haproxy/haproxy.cfg failed! But whene I read the documentation for 1.3 there is a redirect directive available in listen section ( http://haproxy.1wt.eu/download/1.3/doc/configuration.txt#redirectlocation for those how use my script ;) What's is wrong ? ( BTW I notice a mistake in the explanaition of the option of this "redirect" directive : 301 and 302 are both "Moved permanently" in the doc) best regards, -- Damien
Redirect 1 time per day
Hi all, I have a request from my users: We want iphones (hdr_sub(user-agent) -i iphone) to be redirected to Apple store the first time thez come on our site. but only 1 time per days. So I want to user the "redirect" directive acl iphone hdr_sub(User-Agent) -i iphone acl seen hdr_sub(cookie) SEEN=1 redirect location http://itunes.apple.com/us/app/myapp/ set-cookie SEEN=1 if iphone !seen But how to set the cookie : expires=/TOMOROW/ ? I use haproxy 1.3 packaged by debian lenny. Thanks a lot. Best regards, -- Damien
Re: Script greasmonkey to navigate in Haproxy docs ...
Le 27/04/2010 10:33, Damien Hardy a écrit : > Hello all, > > haproxy docs are quite difficult to manipulate. > > I had begun a Greasemonkey > (https://addons.mozilla.org/fr/firefox/addon/748) script to create a > "clickable" ToC in the haproxy documentation ... > > You can DL it at http://userscripts.org/scripts/show/75315 > > Next step : > search the options, add anchor. Search "See also", add links to > conresponding option/anchor... add a search engine ... > > Feel free free to use/modify it. > > Best regards, > Hi all, I've done some improuvment of the pattern detection in the script (the regex for options is writeonly /o\ now unreadable). And add some features like the options search engine (textbox top/right with google sugguest like display in a green div) and "See also" options reference clickable. I keep the same design spirit of the original doc ... it's austere. (some could say ugly) i concentrate on features. Feel free to add your own usercss ^^ -- Damien
Script greasmonkey to navigate in Haproxy docs ...
Hello all, haproxy docs are quite difficult to manipulate. I had begun a Greasemonkey (https://addons.mozilla.org/fr/firefox/addon/748) script to create a "clickable" ToC in the haproxy documentation ... You can DL it at http://userscripts.org/scripts/show/75315 Next step : search the options, add anchor. Search "See also", add links to conresponding option/anchor... add a search engine ... Feel free free to use/modify it. Best regards, -- Damien
