bytes_sent?
The HTTP Log Format tracks bytes_read, which appears to be the number of bytes in the HTTP response send from the backend to the requesting client. https://code.google.com/p/haproxy-docs/wiki/HTTPLogFormat I feel also that bytes_sent may be an interesting field to track. For example: An Image upload site may care a lot more about image uploads to their servers than downloads. (esp for an upload backend pool) Checking the ratio of bytes_read vs bytes_sent also provides interesting insight in to how an application is performing. Is there anyway I can lobby to get bytes_sent added to future revisions of the HTTP Log format? Is there any other method to collect this info? The stats page is already collecting Bytes in and Bytes out, but I'm missing it in the log format, so I can't look at it on a per-request level. Thanks, Joel
Re: Simple health check return... -- without backend needed?
On Fri, Jan 18, 2013 at 1:44 AM, Jonathan Matthews cont...@jpluscplusm.comwrote: On 18 January 2013 02:26, Joel Krauska jkrau...@gmail.com wrote: monitor-uri looks like it would only work as a path match. I need to match a Host header. Could you elaborate a bit on why you're trying to do this? Is it for a human or machine audience? Etc etc ... It's for an external check that unfortunately I cannot change. It looks like the custom 503-200 rewrite will do the trick. But it would be simpler to have a more generic 'monitor' method that could be ACL triggered. --Joel
Simple health check return... -- without backend needed?
Is there a simple way to have haproxy send a simple response for a given URL? Matching somewhat the functionality of Nginx's empty_gif module? http://wiki.nginx.org/HttpEmptyGifModule I don't need to send an actual empty gif, but just a simple 200 and OK would be fine. Use case would be to have an ACL that when matched always responds with a simple 200. (external check to make sure HAProxy is functional without needing to hit any backends.) Can an acl respond with errorcode 200? Would creating a backend with no members and changing the 500 response file to look like a 200 response file work? (seems hacky) Any advice? Thanks, Joel
Re: Simple health check return... -- without backend needed?
On Thu, Jan 17, 2013 at 3:27 PM, Jonathan Matthews cont...@jpluscplusm.comwrote: On 17 January 2013 23:02, Joel Krauska jkrau...@gmail.com wrote: Is there a simple way to have haproxy send a simple response for a given URL? Matching somewhat the functionality of Nginx's empty_gif module? http://wiki.nginx.org/HttpEmptyGifModule I don't need to send an actual empty gif, but just a simple 200 and OK would be fine. Use case would be to have an ACL that when matched always responds with a simple 200. (external check to make sure HAProxy is functional without needing to hit any backends.) Sounds like you want to configure monitor-uri: http://cbonte.github.com/haproxy-dconv/configuration-1.4.html#monitor-uri. monitor-uri looks like it would only work as a path match. I need to match a Host header. eg. ping.example.com (always return 200 when haproxy is running) vs www.example.com (proxy to backend pool)
Re: Backends Referencing other Backends?
The reasons I would want them are in the original email, but here are some more details. 1. -- Gathering of unique stats without having to define a pool twice This would be stellar for adhoc debug. (how much traffic matches this ACL??), but also useful for general traffic classification. 2. -- Allowing a pool to be served by a backup pool if all of the original pool's servers are down. Our app tier machines are technically capable of serving /ANY/ content. (we have a common code deploy) However, we use HAproxy and header or URL matching to group certain sub-groups of traffic to certain pools. eg. Send image renders (which take longer and can block other traffic) to a dedicated pool of image render boxes. If somehow the entire image render pool died, it would be /ok/ to briefly allow that traffic to hit other servers in our general pool. I find that the longer a config file gets, the more error prone it becomes. Having some 'macros' or 'includes' and some of the techniques I was asking for avoids having to repeat configuration and reduce errors. On Wed, Oct 24, 2012 at 12:41 AM, Baptiste bed...@gmail.com wrote: Hi Joel, Unfortunately, this kind of configuration is not doable. Could you tell us why you want to do such thing, what is the real need for this (even if I have some ideas about it ;) ) cheers
Backends Referencing other Backends?
Hello fellow Haproxy users.
Here are some thoughts and example ideas about backend definitions and
seeing if anyone has thought of these features or if they're even
possible today.
Cheers,
Joel Krauska
1. 'Stats only backend'
I'd like to keep stats for a certain matching condition, but don't
want to define the same backend twice.
eg.
use_backend one if { hdr_dom(host) -i one }
use_backend two if { hdr_dom(host) -i two }
backend one
cookie SID insert indirect
server app-01:8080 app-01:8080 cookie app1-8080 check inter 3s
rise 2 fall 2 maxconn 255
server app-02:8080 app-02:8080 cookie app2-8080 check inter 3s
rise 2 fall 2 maxconn 255
option httpchk GET /healthcheck.html
stats admin if TRUE
backend two
# MADE UP COMMAND IS HERE
use_backend one
Not sure if stats for one should == stats for one + stats for two, or
if it would be better for hits to two to only count on two...
I think my preference would be that stats would only count against backend two.
Obviously you need to be careful not to allow for backend looping.
2. 'backend with backup backend'
Fall through to a different backend if primary servers are unavailable
(similar to backup server -- but not requiring defining groups twice
if there's some overlap in pool definitions)
use_backend one if { hdr_dom(host) -i one }
use_backend two if { hdr_dom(host) -i two }
backend one
cookie SID insert indirect
server app-01:8080 app-01:8080 cookie app1-8080 check inter 3s
rise 2 fall 2 maxconn 255
server app-02:8080 app-02:8080 cookie app2-8080 check inter 3s
rise 2 fall 2 maxconn 255
option httpchk GET /healthcheck.html
stats admin if TRUE
backend two
server app-03:8080 app-03:8080 cookie app1-8080 check inter 3s
rise 2 fall 2 maxconn 255
server app-04:8080 app-04:8080 cookie app2-8080 check inter 3s
rise 2 fall 2 maxconn 255
# MADE UP COMMAND IS HERE
backup_backend one
3. 'backend macro includes'
Similar in concept to #1, but maybe more flexible.
use_backend one if { hdr_dom(host) -i one }
use_backend two if { hdr_dom(host) -i two }
backend_macro macro-one
cookie SID insert indirect
server app-01:8080 app-01:8080 cookie app1-8080 check inter 3s
rise 2 fall 2 maxconn 255
server app-02:8080 app-02:8080 cookie app2-8080 check inter 3s
rise 2 fall 2 maxconn 255
option httpchk GET /healthcheck.html
stats admin if TRUE
backend one
# MADE UP COMMAND IS HERE
include_backend_macro macro-one
backend two
# MADE UP COMMAND IS HERE
include_backend_macro macro-one
Re: request per second statistics
I use something like this:
(it has a few things that are specific to my implementation)
#!/usr/bin/python
import os
import csv
import urllib2
import pprint
import time
datadir = '/home/rrdcollect/haproxy-stats/rrds'
hostlist = ['a.network.com']
haproxycreds = ['url','user','pass']
key_values = ['scur', 'bin', 'bout']
for host in hostlist:
print host
url = 'http://%s/%s;csv;norefresh' % (host, haproxycreds[0])
print url
pp = pprint.PrettyPrinter(indent=4)
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, haproxycreds[1], haproxycreds[2])
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
f = urllib2.urlopen(url)
firstrow = True
e = dict()
for row_ind,row in enumerate(csv.reader(f)):
if firstrow:
mapping = row
firstrow = False
if FRONTEND in row or BACKEND in row:
d = {}
for ind,col_val in enumerate(row):
col_name = mapping[ind]
d[col_name]=col_val
row_key = row[0],row_ind
e[row_key]=d
for keys in key_values:
for key_tuple,row_dict in e.iteritems():
row_val = row_dict.get(keys,'')
if keys == scur:
pitem = CurrentSessionRate
elif keys == bin:
pitem = bpsIn
row_val = int(row_val)*8
row_val = str(row_val)
elif keys == bout:
pitem = bpsOut
row_val = int(row_val)*8
row_val = str(row_val)
rrd = %s/haproxyStat_%s_%s_%s.rrd % (datadir,
pitem, key_tuple[0], host)
print rrd
cmd = rrdtool update %s N:%s % (rrd, row_val)
print cmd
if not os.path.isfile(rrd):
print 'RRD does not exist!'
from time import time
now = int(time())
print now
now-=300
print now
maxvalue = 10
if bps in pitem:
cmd = 'rrdtool create %s --start %s
--step 300 DS:data:DERIVE:600:0:%s RRA:AVERAGE:0.5:1:9000
RRA:AVERAGE:0.5:12:4000 RRA:AVERAGE:0.5:72:5000
RRA:AVERAGE:0.5:288:2500' % (rrd, now, maxvalue)
else:
cmd = 'rrdtool create %s --start %s
--step 300 DS:data:COUNTER:600:0:%s RRA:AVERAGE:0.5:1:9000
RRA:AVERAGE:0.5:12:4000 RRA:AVERAGE:0.5:72:5000
RRA:AVERAGE:0.5:288:2500' % (rrd, now, maxvalue)
print cmd
os.system(cmd)
os.system(cmd)
On Tue, Oct 2, 2012 at 11:17 AM, Saul Waizer saulwai...@gmail.com wrote:
Hello List,
This is a general question, has anyone developed a method of polling
statistics from HAproxy to determine the current requests per second rate? I
am building a custom nagios plugin that will also work with graphite and
cacti, i am trying to find the best approach to gather RPS statistics. Any
suggestions?
Thank You,
Saul
Delay Frontend Binding?
Is it possible to delay the FrontEnd binding until the back ends have had time to do health checks and rise? Scenario: I use multiple HAProxy nodes with multiple A records. eg: foo.example.com resolves to a.b.c.d and a.b.c.z If the first machine goes down, most browsers happily move along to a different IP if the first IP fails to respond to port 80. However, I think I am seeing when I recover the original host, it will briefly send 503s when it has bound to port 80 on the frontend, but hasn't connected all the back ends. Thus I'm looking for a mechanism to delay the binding on the FrontEnd for a small timewindow. (ideally I would set this window to just above my 'RISE' thresholds for backend health checks) I was looking for this in the config, but couldn't find it. Do other people see this problem? I imagine I could hack together a gatekeeper using iptables, but I feel like it might be a nice feature for HAProxy. --J
Expire Cookies ??
It would be nice to be able to set an expiration timer on server persistence cookies set by HAProxy. A'la: http://en.wikipedia.org/wiki/HTTP_cookie#Expires_and_Max-Age From what I can tell HAProxy is not setting an expires window for cookies. Is this a feature that I just can't find in the documentation? Cheers, Joel
Apache Compatible HAProxy Log Formatting?
Has anyone worked out a way to get HAProxy to output logging in an Apache Combined Log Format? Most log analysis tools out there already speak Apache log format and I was hoping to avoid writing a parser for what is essentially mostly the same information. Anyone got a work around for this situation? Best, Joel
Re: Configuration API?
On 2/7/11 1:01 PM, Willy Tarreau wrote: On Mon, Feb 07, 2011 at 09:45:21AM +0100, Bedis 9 wrote: Do you have an example of what purpose it would serve ? I'm asking because it's not very easy to implement with table-based algorithms, since the size of the table is determined by the GCD of all active servers' weights. Thus adding a new server will change the size of the table. It's also a feature I've never seen on other products either, which makes be doubt about its usefulness. Hey Willy, It's really useful in big organizations, when you have to manage tens and tens of LBs and you want to ease the management. An API available remotely allows you to write your own centralized management tools: - configuration backup - configuration edit and push - collection of statistics etc... I'm well aware of the usefulness of the API, I was meaning that switching a server's role between active and backup did not seem useful to me ;-) What was your original intent of the Backup Server feature? Our organization uses backup servers to assist with new code rollout and easy rollback. (I'm not fond of it, just looking to automate it with APIs) example use case: Four Servers A,B,C,D All Running The Same Code Rev A B are primary C D are in backup The Upgrade: C D upgrade to new code rev. The Flip: (as quickly as possible to try to stay atomic) C D taken out of Backup State A B put in to Backup State Sanity Checking Phase: Make sure new live roll is performing as expected. Hold on to older rev A B boxes until you feel C D are solid. (typically 15-30 minutes) If the new push is terrible, you can revert The Flip above. Steady State: Upgrade A B to match C D's code revs. Now A B can be thought of as emergency backup for if/when C/D fall over. Lather, rise repeat (swapping AB/CD above) Does the above use case make sense to you? Known limitations: Ideally it would be much easier to downgrade a single system's code. (working on that) Also this clearly doesn't well scale to larger deployments -- 50 active and 50 standby. (working on that too.) --Joel
Re: Configuration API?
Awesome - This is a great start. Putting a server in backup or primary mode would be a nice addition. Adding new servers would also be nice, but that's likely to be a harder project. --Joel On 2/4/11 12:46 PM, John Marrett wrote: Check out section 9.2. Unix Socket commands http://haproxy.1wt.eu/download/1.4/doc/configuration.txt -JohnF On 11-02-04 03:40 PM, Joel Krauska wrote: Has there ever been effort in building out a configuration API for HAProxy? It would be nice to be able to tweak configs externally to do things. (simpler things now, but obviously being able to tweak /ANY/ config parameter might be nice down the road) My wish list: - put a server in backup mode or primary node as needed - raise or lower a server weight - add a new server to an existing pool How do other folks do this in a programmatic fashion? (just adjust config files and HUP?) --Joel
Re: Load Balancing with haproxy makes my application slower?
Sean, I think it would be helpful to further explain your testing scenario. How do you simulate concurrent users? What is RSTav? Usersps is sessions per second?? I think most folks use Apache Bench http://httpd.apache.org/docs/2.0/programs/ab.html as a fairly common industry standard for HTTP server performance. Would you consider rerunning your test using ab as well? Equivalently, you might look at httpperf (see the haproxy web page for some notes) One tuning thing you might try is dropping down your timeouts. You have: timeout connect 1 timeout client 30 timeout server 30 I typically use an order of magnitude smaller. 5000 5 5 (these are exaple defaults listed in an example in 2.3 of the HA proxy docs) http://haproxy.1wt.eu/download/1.4/doc/configuration.txt Best of luck, Joel On 1/29/11 10:53 AM, Sean Hess wrote: I'm performing real-world load tests for the first time, and my results aren't making a lot of sense. Just to make sure I have the test harness working, I'm not testing real application code yet, I'm just hitting a web page that simulates an IO delay (500 ms), and then serializes out some json (about 85 bytes of content). It's not accessing the database, or doing anything other than printing out that data. My application servers are written in node.js, on 512MB VPSes on rackspace (centos55). Here are the results that don't make sense: https://gist.github.com/802082 When I run this test against a single application server (bottom one), You can see that it stays pretty flat (about 550ms response time) until it gets to 1500 simultaneous users, when it starts to error out and get slow. When I run it against an haproxy instance in front of 4 of the same nodes (top one), my performance is worse. It doesn't drop any connections, but the response time edges up much earlier than against a single node. Does this make any sense to you? Does haproxy need more RAM? I was watching the box while the test was running and the haproxy process didn't get higher than 20% CPU and 10% RAM. Please help, thanks!
Re: Custom field addition to Statistics Report webpage?
On 1/25/11 10:48 PM, Willy Tarreau wrote: Hi Joel, On Tue, Jan 25, 2011 at 01:29:11PM -0800, Joel Krauska wrote: I'm not sure if people have asked for this, but I would love a custom field/column in the Stats Report web page. Specifically, I would like to be able to see my backend server version. This is a custom text string which I could provide a health-check-like URL to populate. Then the HAProxy stats page could be a one-stop page to watch during rolling upgrades of backends. Probably for easy customization, an external call to third party script would be ideal. Has anyone considered this before? Is there any way to hack existing features to do this? You should take a look at stats show-desc. You can already use it to report a section-specific string. You can then have your upgrade script fill it from a config template. For instance : backend foo stats show-desc @@foo.version@@ Then replace @@foo.version@@ with whatever you like. Assuming you're in a secure enough environment to consider such things, you should also take a look at stats show-legends which reports much more information (eg: mode, IPs, cookie names, etc...). This is different from what you asked for but will surely help troubleshoot issues if you already feel concerned with the config version. Cheers, Willy Willy, I think you've described a per-backend description. I enabled stats show-desc, but that's a per backend feature, right? I'm looking for a per-server (subset of backend) field. Use case: backend pool foo has four servers: a,b,c,d initial state, all four servers are running code rev 22. I initiate a rolling upgrade, which includes a graceful shut, upgrade of code and graceful start of each server. The new code rev is 23. As I upgrade the code on each server from version 22 to version 23... It would be nice if HAProxy was able to poll a given URL to identify the unique version of each server to monitor the progress of the upgrade and confirm quickly/visually that all members of the same backend are in sync at the end of an upgrade mid upgrade a - version 23 b - version 23 c - down (currently be upgraded) d - version 22 It's sort of like a special case of a healthcheck, but for a given URI and displaying the output, not just if the check succeeded. Something you'd consider adding? In the meantime I've got my own little per server version table page that has this info, but it's very similar to the haproxy page (pools and member servers) Cheers, Joel
Re: Custom field addition to Statistics Report webpage?
On 1/26/11 2:06 AM, Willy Tarreau wrote: On Wed, Jan 26, 2011 at 12:56:54AM -0800, Joel Krauska wrote: On 1/25/11 10:48 PM, Willy Tarreau wrote: Hi Joel, On Tue, Jan 25, 2011 at 01:29:11PM -0800, Joel Krauska wrote: I'm not sure if people have asked for this, but I would love a custom field/column in the Stats Report web page. Specifically, I would like to be able to see my backend server version. This is a custom text string which I could provide a health-check-like URL to populate. Then the HAProxy stats page could be a one-stop page to watch during rolling upgrades of backends. Probably for easy customization, an external call to third party script would be ideal. Has anyone considered this before? Is there any way to hack existing features to do this? You should take a look at stats show-desc. You can already use it to report a section-specific string. You can then have your upgrade script fill it from a config template. For instance : backend foo stats show-desc @@foo.version@@ Then replace @@foo.version@@ with whatever you like. Assuming you're in a secure enough environment to consider such things, you should also take a look at stats show-legends which reports much more information (eg: mode, IPs, cookie names, etc...). This is different from what you asked for but will surely help troubleshoot issues if you already feel concerned with the config version. Cheers, Willy Willy, I think you've described a per-backend description. I enabled stats show-desc, but that's a per backend feature, right? I'm looking for a per-server (subset of backend) field. Use case: backend pool foo has four servers: a,b,c,d initial state, all four servers are running code rev 22. I initiate a rolling upgrade, which includes a graceful shut, upgrade of code and graceful start of each server. The new code rev is 23. As I upgrade the code on each server from version 22 to version 23... It would be nice if HAProxy was able to poll a given URL to identify the unique version of each server to monitor the progress of the upgrade and confirm quickly/visually that all members of the same backend are in sync at the end of an upgrade mid upgrade a - version 23 b - version 23 c - down (currently be upgraded) d - version 22 It's sort of like a special case of a healthcheck, but for a given URI and displaying the output, not just if the check succeeded. Something you'd consider adding? In the meantime I've got my own little per server version table page that has this info, but it's very similar to the haproxy page (pools and member servers) OK I see better what you're looking for. Then you could already do something like this using the health checks. Since the stats page reports the health check response (eg: 200 OK), the code which handles the health check on the server could simply adjust the response to report the version (200 OK: version 1.12) and you'll find it in the stats page. Willy Ok, so I've modified our HTTP responses to include the version. I've noticed that it's only displayed as a tooltip (pop up box when you mouse-over the TD). td class=ac title=Layer7 check passed: OK: version v201 u L7OK/200 in 3ms/u/td I'd really like it to show up in the text portion of the page. td class=ac title=Layer7 check passed: OK: version v201 u L7OK/200 in 3ms : version v201/u/td Configurable option? Cheers, Joel
Re: Custom field addition to Statistics Report webpage?
On 1/26/11 4:30 PM, Willy Tarreau wrote: On Wed, Jan 26, 2011 at 03:00:00PM -0800, Joel Krauska wrote: Ok, so I've modified our HTTP responses to include the version. OK. I've noticed that it's only displayed as a tooltip (pop up box when you mouse-over the TD). td class=ac title=Layer7 check passed: OK: version v201 u L7OK/200 in 3ms/u/td yes indeed. I'd really like it to show up in the text portion of the page. td class=ac title=Layer7 check passed: OK: version v201 u L7OK/200 in 3ms : version v201/u/td We should not do that simply because that enlarges columns. It's hard to have the whole table fit on 1600x1200 screens when numbers are big, so we cannot enlarge colums with varying response sizes. Configurable option? Not until the stats are fully reworked to become more configurable at least. Regards, Willy Understood. How about exporting that extra information in the csv? --Joel
Custom field addition to Statistics Report webpage?
I'm not sure if people have asked for this, but I would love a custom field/column in the Stats Report web page. Specifically, I would like to be able to see my backend server version. This is a custom text string which I could provide a health-check-like URL to populate. Then the HAProxy stats page could be a one-stop page to watch during rolling upgrades of backends. Probably for easy customization, an external call to third party script would be ideal. Has anyone considered this before? Is there any way to hack existing features to do this? Cheers, --Joel
