Fastest response
I am using haproxy to load balance a pool of FTP servers. Since haproxy only handles the command channel and I am using leastconn, it is able to pretty much keep the load balanced between all servers. However, not all users (or command channels) are equal. For example a specific user may open a lot of file transfers, which are opened independently of the command channel, and are not subject to load balancing. Once a user opens a command channel, the way my FTP servers are configured, all data channels will be established with the same backend server. What I have going on is that a particular server will become very busy, which is the reason for load balancing in the first place. It will respond slowly and HAProxy will continue to send it traffic. A portion of users will experience slowdown even while the majority have no issue. Ideally, it would be great if haproxy would try to connect for something like 2 seconds, and then transparently switch to another backend that is responding more quickly. Optionally, I would like to balance based on the backend's response time, rather than the number of connections (since connections are so unequal with FTP). I read the manual a few times looking for inspiration, the best I have been able to come up with is to have an agent that dynamically adjusts weights given some criteria I am yet to define. Any ideas on how to achieve better balancing? Is anyone using a scheme like the above or did I miss a more obvious solution?
Re: Fastest response
Hi Ben, On Mon, Oct 22, 2012 at 04:34:43PM -0400, Ben Timby wrote: > I am using haproxy to load balance a pool of FTP servers. Since > haproxy only handles the command channel and I am using leastconn, it > is able to pretty much keep the load balanced between all servers. > > However, not all users (or command channels) are equal. For example a > specific user may open a lot of file transfers, which are opened > independently of the command channel, and are not subject to load > balancing. Once a user opens a command channel, the way my FTP servers > are configured, all data channels will be established with the same > backend server. > > What I have going on is that a particular server will become very > busy, which is the reason for load balancing in the first place. It > will respond slowly and HAProxy will continue to send it traffic. A > portion of users will experience slowdown even while the majority have > no issue. > > Ideally, it would be great if haproxy would try to connect for > something like 2 seconds, and then transparently switch to another > backend that is responding more quickly. You can already achieve this by playing with the connect timeout and having the redispatch option set. But I recommend against doing so because you could end up with many more connection failures than you expect under load. > Optionally, I would like to > balance based on the backend's response time, rather than the number > of connections (since connections are so unequal with FTP). I don't think it is a good idea. The connect response time it relatively independant on the server load. It will depend more on the NIC settings and the number of hops to the server than on the load. And in FTP, there is really no measurable response time. > I read the manual a few times looking for inspiration, the best I have > been able to come up with is to have an agent that dynamically adjusts > weights given some criteria I am yet to define. Exactly, in my opinion, what you need is to have the servers report their number of data connections. This is an external piece of information for the load balancer so clearly an agent is needed. It's similar to the problem of checking SMTP servers and considering their mail queue length. > Any ideas on how to achieve better balancing? Is anyone using a scheme > like the above or did I miss a more obvious solution? Some of us have already been discussing about the possibility to adapt the HTTP checks to report a header to modulate the server's weight (in fact it was planned for 1.3.14 but skipped because of no use at this time). But we can bring this back on the table. The agent would be very simple in my opinion, it would accept incoming connections from the load balancer on a specific port, would check that the server is available and will adjust a weight between 0 and 100% depending on the number of available connection slots relative to a configured max on the servers. So a server which supports 1000 concurrent connections and runs at 150 would have 850/1000 == 85% weight. Then haproxy will still be able to use leastconn depending on that weight, to distribute the load across all servers. Does that sound good to you ? Willy
Re: Fastest response
Willy, On Tue, Oct 23, 2012 at 2:10 AM, Willy Tarreau wrote: > Some of us have already been discussing about the possibility to adapt the > HTTP checks to report a header to modulate the server's weight (in fact it > was planned for 1.3.14 but skipped because of no use at this time). But we > can bring this back on the table. > > The agent would be very simple in my opinion, it would accept incoming > connections from the load balancer on a specific port, would check that > the server is available and will adjust a weight between 0 and 100% > depending on the number of available connection slots relative to a > configured max on the servers. > > So a server which supports 1000 concurrent connections and runs at 150 > would have 850/1000 == 85% weight. Then haproxy will still be able to > use leastconn depending on that weight, to distribute the load across > all servers. > > Does that sound good to you ? Yes, that sounds exactly like what I am looking for. I suppose if/when this feature hits, I would configure an HTTP check for each FTP backend (in addition to the port 21 tcp check). The HTTP check would connect to a simple agent that emitted a header containing the server's desired weight. HAproxy would handle the rest. I could get started today by writing a simple poller that would sit on the load balancer, poll the FTP servers via HTTP and update weights using the HAProxy control socket. Eventually, HAProxy would handle the polling via the HTTP check and I could discard the interim poller.
Re: Fastest response
On Tue, Oct 23, 2012 at 02:27:54AM -0400, Ben Timby wrote: > Willy, > > On Tue, Oct 23, 2012 at 2:10 AM, Willy Tarreau wrote: > > Some of us have already been discussing about the possibility to adapt the > > HTTP checks to report a header to modulate the server's weight (in fact it > > was planned for 1.3.14 but skipped because of no use at this time). But we > > can bring this back on the table. > > > > The agent would be very simple in my opinion, it would accept incoming > > connections from the load balancer on a specific port, would check that > > the server is available and will adjust a weight between 0 and 100% > > depending on the number of available connection slots relative to a > > configured max on the servers. > > > > So a server which supports 1000 concurrent connections and runs at 150 > > would have 850/1000 == 85% weight. Then haproxy will still be able to > > use leastconn depending on that weight, to distribute the load across > > all servers. > > > > Does that sound good to you ? > > Yes, that sounds exactly like what I am looking for. > > I suppose if/when this feature hits, I would configure an HTTP check > for each FTP backend (in addition to the port 21 tcp check). The HTTP > check would connect to a simple agent that emitted a header containing > the server's desired weight. HAproxy would handle the rest. Exactly. > I could get started today by writing a simple poller that would sit on > the load balancer, poll the FTP servers via HTTP and update weights > using the HAProxy control socket. Eventually, HAProxy would handle the > polling via the HTTP check and I could discard the interim poller. Yes, very good idea indeed. And it will help you tune the weight reporting algorithm BTW. Willy
