VM Power Control/Elasticity
I was wondering if it is possible or there's a recommended way to deal with dynamic capacity expansion for a given back-end. I searched through the documentation some and didn't see anything obvious, so figured I would ask here. Basically, I would like a way to trigger a script when the number of active connections gets within a certain range of the total available connections for the available backend servers. I would like this script to be able to do something like trigger a script or command that creates or powers on or off a virtual machine that is already added to or could be dynamically added to or removed from the back-end. The basic scenario is this: - Back-end starts with 10 configured systems or 10 connections each. - 5 of the 10 systems are powered on by default, with the other 5 down. - Users connect, and close to 40/50 available connections. - HAProxy detects the connection limit and triggers a script that starts up the 6th VM. - Once HAProxy detects that the 6th VM is running, the number of available connections moves to 60. - Users continue to connect and close to 50/60, triggering another power event. I'd also like the reverse of that to happen: - Users begin to disconnect and connections drop to 40/60. - HAProxy triggers another script to stop one of the the configured back-end systems that has zero connections. Is this possible? Or on the roadmap? Or something that, while not implemented directly in the HAProxy configuration could be done some other way - some periodic polling of HAProxy some other way? Thanks, Nick
Re: VM Power Control/Elasticity
Nick, HAProxy provides statistics via socket or HTTP interface. You can easily monitor these stats and run scripts. Some cron jobs and regex should suffice. Specific cases like this are usually not something I would imagine belongs in HAProxy core, since it is not directly related to load balancing but more of a specific requirement. You could simply configure all 10 servers, 5 of which would mostly be in the down state until your script brought them UP. HAProxy will happily balance traffic to all UP servers, and shift it away from DOWN servers. You can also use the stats socket to mark machines as up or down, so that traffic can be gracefully shifted before after VM power up/down. However, this may or may not be necessary depending on the services you are load balancing. https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2
Re: VM Power Control/Elasticity
Nick, Here is some information on using socat to interact with the stats socket. This might be useful for shell scripting. http://www.mgoff.in/2010/07/14/haproxy-gathering-stats-using-socat/
Re: VM Power Control/Elasticity
Thanks for the hints, Ben. I'll defer to those who are experts about whether or not something like that should be part of the core functionality; however, it seems that even though this case might not be a great one for adding that to the core, there are a multitude of reasons why you'd want some sort of trigger mechanism on events within HAProxy. I think people have asked here before about e-mail notifications for downed hosts, and I can think of a few other cases outside of mine corner case that would seem to warrant some generic trigger mechanism within HAProxy. Seems if a generic one is implemented the flexibility is there for everyone who needs a trigger of some sort or another to use it for whatever purposes suite their needs. I'm not a very good programmer (barely can be called a programmer at all), so I don't have code to contribute to make that a reality, and I don't want to start any sort of war over where that belongs, just my view as a HAProxy user. Thanks! -Nick From: "Ben Timby" To: "Nick E Couchman" Cc: haproxy@formilux.org Sent: Monday, May 11, 2015 11:24:58 AM Subject: Re: VM Power Control/Elasticity Nick, HAProxy provides statistics via socket or HTTP interface. You can easily monitor these stats and run scripts. Some cron jobs and regex should suffice. Specific cases like this are usually not something I would imagine belongs in HAProxy core, since it is not directly related to load balancing but more of a specific requirement. You could simply configure all 10 servers, 5 of which would mostly be in the down state until your script brought them UP. HAProxy will happily balance traffic to all UP servers, and shift it away from DOWN servers. You can also use the stats socket to mark machines as up or down, so that traffic can be gracefully shifted before after VM power up/down. However, this may or may not be necessary depending on the services you are load balancing. https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2
Re: VM Power Control/Elasticity
Thanks for the site - I will take a look. Sounds like the socket is the way to go - should be easier to parse than the web UI. -Nick From: "Ben Timby" To: "Nick E Couchman" Cc: haproxy@formilux.org Sent: Monday, May 11, 2015 11:35:03 AM Subject: Re: VM Power Control/Elasticity Nick, Here is some information on using socat to interact with the stats socket. This might be useful for shell scripting. http://www.mgoff.in/2010/07/14/haproxy-gathering-stats-using-socat/
Re: VM Power Control/Elasticity
On Mon, May 11, 2015 at 3:47 PM, Nick Couchman wrote: > Thanks for the hints, Ben. I'll defer to those who are experts about > whether or not something like that should be part of the core > functionality; however, it seems that even though this case might not be a > great one for adding that to the core, there are a multitude of reasons why > you'd want some sort of trigger mechanism on events within HAProxy. I > think people have asked here before about e-mail notifications for downed > hosts, and I can think of a few other cases outside of mine corner case > that would seem to warrant some generic trigger mechanism within HAProxy. > Seems if a generic one is implemented the flexibility is there for everyone > who needs a trigger of some sort or another to use it for whatever purposes > suite their needs. > Especially for down host notification, there are tools to do that, like Nagios for one. Even a tool like Nagios can run a command (like a simple shell script to read stats from HAProxy) and then conditionally run another command (like one that starts some VMs). You may find there is another tool even better suited to your needs that already exists. The shell scripting required may be minimal to get the needed metrics into the tool and run hypervisor commands to control your VMs. This may even be a tool related more closely to your hypervisor, the software that manages your VMs. It may be able to run a shell script and use the exit code to provision more/less machines of a certain class. Then all you need a script to read stats and produce an exit code.
Re: VM Power Control/Elasticity
You can't add or remove hosts to a pool without doing a reload - you can change the weights, mark them up and down, but not add or remove. On Mon, May 11, 2015 at 1:00 PM, Nick Couchman wrote: > I was wondering if it is possible or there's a recommended way to deal > with dynamic capacity expansion for a given back-end. I searched through > the documentation some and didn't see anything obvious, so figured I would > ask here. > > Basically, I would like a way to trigger a script when the number of > active connections gets within a certain range of the total available > connections for the available backend servers. I would like this script to > be able to do something like trigger a script or command that creates or > powers on or off a virtual machine that is already added to or could be > dynamically added to or removed from the back-end. The basic scenario is > this: > - Back-end starts with 10 configured systems or 10 connections each. > - 5 of the 10 systems are powered on by default, with the other 5 down. > - Users connect, and close to 40/50 available connections. > - HAProxy detects the connection limit and triggers a script that starts > up the 6th VM. > - Once HAProxy detects that the 6th VM is running, the number of available > connections moves to 60. > - Users continue to connect and close to 50/60, triggering another power > event. > > I'd also like the reverse of that to happen: > - Users begin to disconnect and connections drop to 40/60. > - HAProxy triggers another script to stop one of the the configured > back-end systems that has zero connections. > > Is this possible? Or on the roadmap? Or something that, while not > implemented directly in the HAProxy configuration could be done some other > way - some periodic polling of HAProxy some other way? > > Thanks, > Nick > > >
Re: VM Power Control/Elasticity
Thanks for the info, CJ. Has this feature been considered - the ability to add/remove on-the-fly? Anyway, I think in my situation I would just list all of the hosts in the config file initially and have some of them down, then I would already know which ones to start up to bump up the connection limit. I think the use of the socket will get me most of the way toward where I want to go - particularly since the output comes in easily-parsable CSV. Thanks, everyone! -Nick > From: "CJ Ess" > To: "Nick E Couchman" > Cc: "HAProxy" > Sent: Tuesday, May 12, 2015 9:46:00 AM > Subject: Re: VM Power Control/Elasticity > You can't add or remove hosts to a pool without doing a reload - you can > change > the weights, mark them up and down, but not add or remove. > On Mon, May 11, 2015 at 1:00 PM, Nick Couchman < nick.couch...@seakr.com > > wrote: >> I was wondering if it is possible or there's a recommended way to deal with >> dynamic capacity expansion for a given back-end. I searched through the >> documentation some and didn't see anything obvious, so figured I would ask >> here. >> Basically, I would like a way to trigger a script when the number of active >> connections gets within a certain range of the total available connections >> for >> the available backend servers. I would like this script to be able to do >> something like trigger a script or command that creates or powers on or off a >> virtual machine that is already added to or could be dynamically added to or >> removed from the back-end. The basic scenario is this: >> - Back-end starts with 10 configured systems or 10 connections each. >> - 5 of the 10 systems are powered on by default, with the other 5 down. >> - Users connect, and close to 40/50 available connections. >> - HAProxy detects the connection limit and triggers a script that starts up >> the >> 6th VM. >> - Once HAProxy detects that the 6th VM is running, the number of available >> connections moves to 60. >> - Users continue to connect and close to 50/60, triggering another power >> event. >> I'd also like the reverse of that to happen: >> - Users begin to disconnect and connections drop to 40/60. >> - HAProxy triggers another script to stop one of the the configured back-end >> systems that has zero connections. >> Is this possible? Or on the roadmap? Or something that, while not implemented >> directly in the HAProxy configuration could be done some other way - some >> periodic polling of HAProxy some other way? >> Thanks, >> Nick
Re: VM Power Control/Elasticity
For what it's worth, I've implemented a basic script in PHP (please don't yell at me for using PHP) that sucks in the info from the haproxy socket and then executes commands to the XenServer hosts using the XenAPI (XAPI). It works decently well - I have the script running every 5 minutes, which seems to be okay so far for the demand on these systems. It seems to be an okay balance between boot time of the systems and people connecting to the systems. I will say it would still be better to have something trigger-based rather than polling. I even looked into trying to use iptables on Linux to trigger something upon connection to a certain port, but iptables doesn't quite have the right functionality to do it. In any case, the periodic script will suffice for now, and I'm sure I'll change and improve upon it a bit. For the time being it has been able to manage my overall usage of physical resources a little more efficiently as it can spin down VMs when they aren't being used, which frees those resources for other systems and cuts the overall heat generation and power consumption. -Nick From: "Ben Timby" To: "Nick E Couchman" Cc: haproxy@formilux.org Sent: Monday, May 11, 2015 11:35:03 AM Subject: Re: VM Power Control/Elasticity Nick, Here is some information on using socat to interact with the stats socket. This might be useful for shell scripting. http://www.mgoff.in/2010/07/14/haproxy-gathering-stats-using-socat/
Re: VM Power Control/Elasticity
On Tue, May 12, 2015 at 10:58:35AM -0600, Nick Couchman wrote: > Thanks for the info, CJ. Has this feature been considered - the ability to > add/remove on-the-fly? Anyway, I think in my situation I would just list all > of the hosts in the config file initially and have some of them down, then I > would already know which ones to start up to bump up the connection limit. Yes it's been considered a number of times but it comes with a number of shortcomings which will affect reliability, and for most users it cannot be traded. The first one is that the max number of FDs is automatically determined based on the number of checks etc. This could be worked around by implementing an extra margin to add up to X servers, with X to be configured. The second one is that it will not be possible to use certain LB algorithms (those based on a table) since the tables have a size based on the PGCD of the weights of all the servers in the farm. That might be acceptable for some users. There are a few other annoting points I don't have in mind right now. That doesn't mean it will not be done, just that it will not be as trivial as it could seem and will have some limitations. But we'll work on it due to the fact that the demand is increasing. In the mean time, I would strongly recommend you to test the "balance first" algorithm which packs the load on the first servers. You must configure a maxconn value on your servers. The purpose of this mechanism is to only use the smallest possible set of servers, allowing you to stop the other ones. In the CSV stats there's a lastsess field saying how long ago the last session used this server. Those remaining unused for "long enough" (threshold to set in your monitoring scripts) can safely be turned off. And in order to detect when to start new ones, you simply have to look at the backend queue : once the queue grows to approx 1/2 of a maxconn, it makes sense to start a new server. You will find that if you preconfigure your servers, you won't even have to change your haproxy config nor to reload, the monitoring tools will have a reliable way to take the decision to add/ remove a server and everything will naturally follow. > I think the use of the socket will get me most of the way toward where I want > to go - particularly since the output comes in easily-parsable CSV. Absolutely :-) Regards, Willy
