Re: Configuration API?
Hi Joel, On Mon, Feb 07, 2011 at 01:10:53PM -0800, Joel Krauska wrote: > On 2/7/11 1:01 PM, Willy Tarreau wrote: > >On Mon, Feb 07, 2011 at 09:45:21AM +0100, Bedis 9 wrote: > >>>Do you have an example of what purpose it would serve ? I'm asking > >>>because it's not very easy to implement with table-based algorithms, > >>>since the size of the table is determined by the GCD of all active > >>>servers' weights. Thus adding a new server will change the size of > >>>the table. > >>> > >>>It's also a feature I've never seen on other products either, which > >>>makes be doubt about its usefulness. > >> > >> > >>Hey Willy, > >> > >>It's really useful in big organizations, when you have to manage tens > >>and tens of LBs and you want to ease the management. > >>An API available remotely allows you to write your own centralized > >>management tools: > >>- configuration backup > >>- configuration edit and push > >>- collection of statistics > >>etc... > > > >I'm well aware of the usefulness of the API, I was meaning that switching > >a server's role between active and backup did not seem useful to me ;-) > > > What was your original intent of the "Backup Server" feature? The backup server is the server that must be used when everything else dies. In two-server setups, it happens that an application does not support load balancing at all and requires a single active/backup cluster so this is perfect for this. Other common usages are sorry servers to serve excuse pages when a site is down. Sometimes it even happens that backup servers are the old servers being upgraded, and they're left there for a short validation period (eg: one week) in case something bad happens to the new servers (bug in a network driver causing them to panic twice a day, ...). That's why in this usage it does not make much sense to switch them. > Our organization uses backup servers to assist with new code rollout and > easy rollback. (I'm not fond of it, just looking to automate it with APIs) I see what you mean, two of my customers are doing the same. > example use case: > > Four Servers A,B,C,D > All Running The Same Code Rev > A & B are primary > C & D are in backup > > The Upgrade: > C & D upgrade to new code rev. > > The Flip: (as quickly as possible to try to stay atomic) > C & D taken out of Backup State > A & B put in to Backup State > > Sanity Checking Phase: > Make sure new live roll is performing as expected. > Hold on to older rev A & B boxes until you feel C & D are solid. > (typically 15-30 minutes) > If the new push is terrible, you can revert "The Flip" above. > > Steady State: > Upgrade A & B to match C & D's code revs. > Now A & B can be thought of as emergency backup for if/when C/D fall over. > > Lather, rise repeat (swapping AB/CD above) > > > Does the above use case make sense to you? Yes except that in my experience, once the code has been switched to C/D, A/B will never be put in production anymore because the application is backwards compatible but not upwards compatible, or simply because it's not acceptable to present the old site to visitors after the new version has been advertised and published. What I'm seeing in field is a variant of this. A/B will have their nominal weights while C/D will have a zero weight. That means they'll never ever be used. C/D get installed with the new code. They're tested using cookies and/or force-persist. Once the application seems to run OK on C/D, they're either slowly added to the farm (if the application supports it) or switched at once simply by switching weights, either in the config or on the stats CLI. It does not break existing sessions and leaves much more flexibility for the switchover. Also there is no risk that the undesired servers gets accidentely used in case some hiccups happens on the new servers. > Known limitations: > > Ideally it would be much easier to downgrade a single system's code. > (working on that) > > Also this clearly doesn't well scale to larger deployments -- 50 active > and 50 standby. (working on that too.) The principle with switching weights allows that too. I know some people who run a new version of one server with a small weight for some time before the main switch. You can't do that with backup servers. Sometimes they can even test the application's robustness by exagerating the weight on one server and measure the effects. I must say I really like this way of doing it because it's just as if they were having a mixing table full of potentiometers when others just have on/off switches. Seeing that in action is impressive when you deal with hundreds of servers in a single instance ! Willy
Re: Configuration API?
On 2/7/11 1:01 PM, Willy Tarreau wrote: On Mon, Feb 07, 2011 at 09:45:21AM +0100, Bedis 9 wrote: Do you have an example of what purpose it would serve ? I'm asking because it's not very easy to implement with table-based algorithms, since the size of the table is determined by the GCD of all active servers' weights. Thus adding a new server will change the size of the table. It's also a feature I've never seen on other products either, which makes be doubt about its usefulness. Hey Willy, It's really useful in big organizations, when you have to manage tens and tens of LBs and you want to ease the management. An API available remotely allows you to write your own centralized management tools: - configuration backup - configuration edit and push - collection of statistics etc... I'm well aware of the usefulness of the API, I was meaning that switching a server's role between active and backup did not seem useful to me ;-) What was your original intent of the "Backup Server" feature? Our organization uses backup servers to assist with new code rollout and easy rollback. (I'm not fond of it, just looking to automate it with APIs) example use case: Four Servers A,B,C,D All Running The Same Code Rev A & B are primary C & D are in backup The Upgrade: C & D upgrade to new code rev. The Flip: (as quickly as possible to try to stay atomic) C & D taken out of Backup State A & B put in to Backup State Sanity Checking Phase: Make sure new live roll is performing as expected. Hold on to older rev A & B boxes until you feel C & D are solid. (typically 15-30 minutes) If the new push is terrible, you can revert "The Flip" above. Steady State: Upgrade A & B to match C & D's code revs. Now A & B can be thought of as emergency backup for if/when C/D fall over. Lather, rise repeat (swapping AB/CD above) Does the above use case make sense to you? Known limitations: Ideally it would be much easier to downgrade a single system's code. (working on that) Also this clearly doesn't well scale to larger deployments -- 50 active and 50 standby. (working on that too.) --Joel
Re: Configuration API?
On Mon, Feb 07, 2011 at 09:45:21AM +0100, Bedis 9 wrote: > > Do you have an example of what purpose it would serve ? I'm asking > > because it's not very easy to implement with table-based algorithms, > > since the size of the table is determined by the GCD of all active > > servers' weights. Thus adding a new server will change the size of > > the table. > > > > It's also a feature I've never seen on other products either, which > > makes be doubt about its usefulness. > > > Hey Willy, > > It's really useful in big organizations, when you have to manage tens > and tens of LBs and you want to ease the management. > An API available remotely allows you to write your own centralized > management tools: > - configuration backup > - configuration edit and push > - collection of statistics > etc... I'm well aware of the usefulness of the API, I was meaning that switching a server's role between active and backup did not seem useful to me ;-) Willy
Re: Configuration API?
> Do you have an example of what purpose it would serve ? I'm asking > because it's not very easy to implement with table-based algorithms, > since the size of the table is determined by the GCD of all active > servers' weights. Thus adding a new server will change the size of > the table. > > It's also a feature I've never seen on other products either, which > makes be doubt about its usefulness. Hey Willy, It's really useful in big organizations, when you have to manage tens and tens of LBs and you want to ease the management. An API available remotely allows you to write your own centralized management tools: - configuration backup - configuration edit and push - collection of statistics etc... cheers
Re: Configuration API?
On Sun, Feb 06, 2011 at 10:30:03PM -0800, carlo flores wrote: > Your last suggestion is what are suits call auto-scaling, and the idea of > doing that with HAProxy is really appealing and would be a big plus for us. > > > As a test tomorrow with 1.4.10/stable idea, let's say a pool of 10 servers > had two servers with zero weight, and a script watching the unix socket set > their weight proportionate to the rest if a) the leastconn queue size was > past some threshold or b) too many servers have gone unavailable. What do > you think? That's the idea. Though instead of counting the number of missing servers, I'd really suggest checking for an available cumulated weight. After all, servers weight indicate a processing capacity. So let's say that you add X servers until the total weight is at least equal to a predefined threshold. Willy
Re: Configuration API?
Your last suggestion is what are suits call auto-scaling, and the idea of doing that with HAProxy is really appealing and would be a big plus for us. As a test tomorrow with 1.4.10/stable idea, let's say a pool of 10 servers had two servers with zero weight, and a script watching the unix socket set their weight proportionate to the rest if a) the leastconn queue size was past some threshold or b) too many servers have gone unavailable. What do you think? On Sun, Feb 6, 2011 at 10:18 PM, Willy Tarreau wrote: > Hi Joel, > > On Fri, Feb 04, 2011 at 03:04:40PM -0800, Joel Krauska wrote: > > Awesome - This is a great start. > > > > Putting a server in backup or primary mode would be a nice addition. > > Do you have an example of what purpose it would serve ? I'm asking > because it's not very easy to implement with table-based algorithms, > since the size of the table is determined by the GCD of all active > servers' weights. Thus adding a new server will change the size of > the table. > > It's also a feature I've never seen on other products either, which > makes be doubt about its usefulness. > > What I'd like to implement however is the notion of spare servers, > servers which are activated only above a certain load or when too > many servers were lost. Maybe this is in fact what you're looking > for ? > > > Adding new servers would also be nice, but that's likely to be a harder > > project. > > Yes, it was already proposed but there are quite a number of structural > issues. This as simple as file descriptors to check the servers, and > table size for load balancing come to mind. I'd really prefer that we > spend our time working on a hot reconfiguration (which will bring this > benefit) than on border-line features that are supposed to work around > the lack of hot reconf. > > Regards, > Willy > > >
Re: Configuration API?
Hi Joel, On Fri, Feb 04, 2011 at 03:04:40PM -0800, Joel Krauska wrote: > Awesome - This is a great start. > > Putting a server in backup or primary mode would be a nice addition. Do you have an example of what purpose it would serve ? I'm asking because it's not very easy to implement with table-based algorithms, since the size of the table is determined by the GCD of all active servers' weights. Thus adding a new server will change the size of the table. It's also a feature I've never seen on other products either, which makes be doubt about its usefulness. What I'd like to implement however is the notion of spare servers, servers which are activated only above a certain load or when too many servers were lost. Maybe this is in fact what you're looking for ? > Adding new servers would also be nice, but that's likely to be a harder > project. Yes, it was already proposed but there are quite a number of structural issues. This as simple as file descriptors to check the servers, and table size for load balancing come to mind. I'd really prefer that we spend our time working on a hot reconfiguration (which will bring this benefit) than on border-line features that are supposed to work around the lack of hot reconf. Regards, Willy
Re: Configuration API?
Try using Hatop...this will do all the things from your list http://feurix.org/projects/hatop/ I am using it, and learning more about it now.. --- On Fri, 2/4/11, Joel Krauska wrote: From: Joel Krauska Subject: Re: Configuration API? To: "John Marrett" Cc: haproxy@formilux.org Date: Friday, February 4, 2011, 6:04 PM Awesome - This is a great start. Putting a server in backup or primary mode would be a nice addition. Adding new servers would also be nice, but that's likely to be a harder project. --Joel On 2/4/11 12:46 PM, John Marrett wrote: > Check out section "9.2. Unix Socket commands" > > http://haproxy.1wt.eu/download/1.4/doc/configuration.txt > > -JohnF > > On 11-02-04 03:40 PM, Joel Krauska wrote: >> Has there ever been effort in building out a configuration API for >> HAProxy? >> >> It would be nice to be able to tweak configs externally to do things. >> (simpler things now, but obviously being able to tweak /ANY/ config >> parameter might be nice down the road) >> >> My wish list: >> - put a server in backup mode or primary node as needed >> - raise or lower a server weight >> - add a new server to an existing pool >> >> How do other folks do this in a programmatic fashion? >> (just adjust config files and HUP?) >> >> --Joel >> >
Re: Configuration API?
Awesome - This is a great start. Putting a server in backup or primary mode would be a nice addition. Adding new servers would also be nice, but that's likely to be a harder project. --Joel On 2/4/11 12:46 PM, John Marrett wrote: Check out section "9.2. Unix Socket commands" http://haproxy.1wt.eu/download/1.4/doc/configuration.txt -JohnF On 11-02-04 03:40 PM, Joel Krauska wrote: Has there ever been effort in building out a configuration API for HAProxy? It would be nice to be able to tweak configs externally to do things. (simpler things now, but obviously being able to tweak /ANY/ config parameter might be nice down the road) My wish list: - put a server in backup mode or primary node as needed - raise or lower a server weight - add a new server to an existing pool How do other folks do this in a programmatic fashion? (just adjust config files and HUP?) --Joel
Re: Configuration API?
As an example, our shop extends your wish list with the ability to pull stats from the UNIX socket and trigger alerts via Nagios or Cloudkick ass we need. If you want it, easy to add whatever you need to a fork of this. We'll commit it back in: https://github.com/flores/haproxyctl I believe you can only enable or disable already configured servers in your pool. A way around this could be to set disabled servers to some floating ip address/dns you assign to new servers. Still limited by the number of preconfigured ips/dns records of course. Maybe there's a better way... On Fri, Feb 4, 2011 at 12:46 PM, John Marrett wrote: > Check out section "9.2. Unix Socket commands" > > http://haproxy.1wt.eu/download/1.4/doc/configuration.txt > > -JohnF > > > On 11-02-04 03:40 PM, Joel Krauska wrote: > >> Has there ever been effort in building out a configuration API for >> HAProxy? >> >> It would be nice to be able to tweak configs externally to do things. >> (simpler things now, but obviously being able to tweak /ANY/ config >> parameter might be nice down the road) >> >> My wish list: >> - put a server in backup mode or primary node as needed >> - raise or lower a server weight >> - add a new server to an existing pool >> >> How do other folks do this in a programmatic fashion? >> (just adjust config files and HUP?) >> >> --Joel >> >> > >
Re: Configuration API?
Check out section "9.2. Unix Socket commands" http://haproxy.1wt.eu/download/1.4/doc/configuration.txt -JohnF On 11-02-04 03:40 PM, Joel Krauska wrote: Has there ever been effort in building out a configuration API for HAProxy? It would be nice to be able to tweak configs externally to do things. (simpler things now, but obviously being able to tweak /ANY/ config parameter might be nice down the road) My wish list: - put a server in backup mode or primary node as needed - raise or lower a server weight - add a new server to an existing pool How do other folks do this in a programmatic fashion? (just adjust config files and HUP?) --Joel
Configuration API?
Has there ever been effort in building out a configuration API for HAProxy? It would be nice to be able to tweak configs externally to do things. (simpler things now, but obviously being able to tweak /ANY/ config parameter might be nice down the road) My wish list: - put a server in backup mode or primary node as needed - raise or lower a server weight - add a new server to an existing pool How do other folks do this in a programmatic fashion? (just adjust config files and HUP?) --Joel
