Re: maxqueue for a backend?
Hi David, On Wed, Jun 06, 2012 at 01:03:05PM -0700, David Birdsong wrote: > it got me thinking how great it would be if we could bury links in the > manual (or a copy of it) to public urls to these discussions. It's not worth it I think, and links on the net are short-lived nowadays. Better have some application notes explaining how to do X or Y (which basically is the goal of the quite old arch manual). > it > doesn't fit to have the conversational tone in a manual, but to link > out to this discussion would be so awesome. When you reread the doc in 5 years and find all the links are broken, you'll be a bit disgusted I think :-) > maybe this what stackexchange could be used for, but it'd require us > to copy and replay the whole discussion. No, ideally if we had enough time to spare on documentation, we'd spend less words on the list because things would be much more obvious from the beginning. The problem is that the team working on the code is small and it's more or less the same team who tries to maintain the doc up to date. This is fine when we achieve a stable state, but doing so considerably slows down development phases, which is why doc is always late. > > If you would prevent the backend from queueing any connections, you would > > constantly emit 503 to some visitors, because traffic is never perfectly > > smooth (which is observed by your backend queue inflating). > > i my case, i have another backend fully capable of serving the > requests, so if haproxy can route away accurately, the requests could > land in a capable backend. OK but if you need persistence, you'll break it every time you send a request to the wrong backend. Queues are something normal. For instance, when you look at the traffic on your ethernet link, let's say you're seeing 200 Mbps out. In fact, you're having only bursts of 1 Gbps and pauses of 0. It's thanks to the system's queues that you don't drop packets all the time. Here it's the same. When you run at 3kreq/s, you can be sure that you have peaks of more than 10k/s for a few ms, and pause of several tens to hundreds of ms. You clearly need queues to ensure this does not become a random session killer. > is the 503 only because haproxy can't know ahead of time before > choosing the backend what will happen in the future and so enforching > no queue is technically not reasonable? No, it's simply because haproxy tries applies the load balancing algorithm in the farm, it finds no server (since they're all full), so it wants to put the request in the backend's queue. And if we apply the maxqueue method, then once the backend's queue reaches this value, the request has to be rejected because there's nowhere to service it. With the ACLs at least you can take decisions before reaching this situation. If you have a backend which is never full, you can route the requests there once the first one is considered full, but as explained above, if you don't share the same servers and cookies, you break persistence. > > The best way I found to manage queues is the avg_queue ACL. It returns > > the average number of queued connections per active server. This means > > that wherever the connections are queued, they're accounted for, and > > divided by the number of servers. The result is an average queue length > > for the backend, which translates in an average wait time for visitors. > > cool, i'll use it. is it expensive to calc on every request? I have just rechecked (acl_fetch_avg_queue_size()) and it happens that I had already got rid of the computation loop and used a "totpend" variable instead, to store the global number of queued requests per backend, all servers included. So in fact it's not expensive anymore, it's just a divide by the number of enabled servers. > > BTW, what I'd suggest if you use this is to split the traffic depending > > on the persistence cookie : stop sending new visitors to a backend which > > has too many queued connections, but still send existing ones there. You > > will see that the load on the servers will then regulate itself very well > > and become extremely smooth. And that way you can finely control your > > response times. In short this looks like this : > > > > in this case, requests are completely stateless. we have a pool of 40 > api servers capable of handling paths at /v2/*. the api servers are a > java jetty app. OK. > i'm experimenting with carving up the urls space to > send the same requests to the same servers to see if garbage > collection is slightly more efficient if the servers don't see all of > the api namespace and can 'specialize'. It should theorically be better because the overall code and data should remain in warm cache instead of thrashing all the time. Now how much better it should be is a mystery! Cheers, Willy
Re: maxqueue for a backend?
On Tue, Jun 5, 2012 at 10:35 PM, Willy Tarreau wrote: > Hi David, as usual, great response willy. it got me thinking how great it would be if we could bury links in the manual (or a copy of it) to public urls to these discussions. it doesn't fit to have the conversational tone in a manual, but to link out to this discussion would be so awesome. maybe this what stackexchange could be used for, but it'd require us to copy and replay the whole discussion. > > On Tue, Jun 05, 2012 at 06:17:15PM -0700, David Birdsong wrote: >> Is there a way to set the maxqueu for an entire backend? > > No. When I developped the server maxqueue, I remember having done the same > for the backend (it even ended up in the doc, causing later confusion), but > I removed it during tests because I noticed it was totally useless the way > it worked. Initially the goal was to limit the queue in the backend, not > counting the servers, which did not make much sense since it depends whether > the clients have a cookie or not. Then I figured that we'd have to count all > the queues of all servers, and I did not want to do that for each connection > since it's always expensive for just a few use cases. So I removed it before > ever releasing it. > >> I set maxqueu >> on default-server to 1, and on the frontend ACL required that >> connslots be greater than 1. I expected connslots to be maxconn * num >> server + 1 * num servers (connections + queue slots). >> >> I found that the backend still queued requests. My ACL could check the >> current queue size of the backend and route away from that backend if >> it's gt 0, but I figured connslots was the more correct approach. What >> am I doing or understanding wrongly? > > You're not necessarily doing anything wrong, there are a wide number of > reasons for deciding not to queue in a specific way, and that's the first > thing to define. If you have maxconn on servers, you need some queues. > The server queues are only for requests which target a specific server. > If the request has no cookie set, it will remain in the backend queue so > that any server may pick it. So as you can see, you *need* to support > queues both in the backend and servers whenever you use maxconn. > > If you would prevent the backend from queueing any connections, you would > constantly emit 503 to some visitors, because traffic is never perfectly > smooth (which is observed by your backend queue inflating). i my case, i have another backend fully capable of serving the requests, so if haproxy can route away accurately, the requests could land in a capable backend. is the 503 only because haproxy can't know ahead of time before choosing the backend what will happen in the future and so enforching no queue is technically not reasonable? > > The best way I found to manage queues is the avg_queue ACL. It returns > the average number of queued connections per active server. This means > that wherever the connections are queued, they're accounted for, and > divided by the number of servers. The result is an average queue length > for the backend, which translates in an average wait time for visitors. > cool, i'll use it. is it expensive to calc on every request? > I think it's the most efficient way of taking decisions based on queue > length because you don't need to consider any difference between new and > past visitors, and you don't care whether a server has a deeper queue > than another one. This way you can safely remove the maxqueue parameter > from your servers and only decide to switch to another backend based on > the avg_queue. > > BTW, what I'd suggest if you use this is to split the traffic depending > on the persistence cookie : stop sending new visitors to a backend which > has too many queued connections, but still send existing ones there. You > will see that the load on the servers will then regulate itself very well > and become extremely smooth. And that way you can finely control your > response times. In short this looks like this : > in this case, requests are completely stateless. we have a pool of 40 api servers capable of handling paths at /v2/*. the api servers are a java jetty app. i'm experimenting with carving up the urls space to send the same requests to the same servers to see if garbage collection is slightly more efficient if the servers don't see all of the api namespace and can 'specialize'. > frontend pub > acl bk_full avg_queue(bk_www) gt 10 > acl has_cook cook_len(SRV) gt 0 > use_backend bk_sorry if bk_full !has_cook > default_backend bk_www > > backend bk_www > cookie SRV insert indirect nocache > server s1 1.1.1.1:80 cookie 1 maxconn 50 > ... > > backend bk_sorry > server sorry 127.0.0.1:8080 > > Regards, > Willy >
Re: FW: problem with haproxy reload
Its toally dirty, but we have our wrapper check for such exceptions, then
force a listener if an haproxy listener doesn't exist after a
reload/restart to the existing (now dead) haproxy process. I've grown to
not fret about such dirty when running haproxy dev branch, but ymmv.
https://github.com/flores/haproxyctl
On Wednesday, June 6, 2012, Senthil wrote:
>
>
> Hi,
>
>
>
> We faced with haproxy, we have a script which deletes the
>
> frontend and backend entries of haproxy based on name and does a reload
of
>
> haproxy after haproxy file check is done.
>
>
>
>
>
>
>
> In one such scenario after deleting the frontend and backend and
reloading
>
> we found that haproxy was in stop state
>
>
>
>
>
>
>
> Below are the logs which shows the backend was started again during
reload
>
>but the frontends were not started and the same are shown in logs
after we manually restarted
>
> haproxy
>
>
>
>
>
>
>
> Any feedback regarding this will be very useful.
>
>
>
> Regards
>
> Senthil
>
>
>
>
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend ssl_frontend_1
in 0 ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend
ssl_frontend_1BACK in 0 ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend
ssl_frontend_2 in 0 ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend
ssl_frontend_2BACK in 0 ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend Star in 0 ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend StarBACK in 0
ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend Staging in 0
ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend StagingBACK in
0 ms.
>
>
>
> May 18 19:36:10 indya-lb haproxy[13147]: Proxy ssl_frontend_2BACK
started.
>
>
>
> May 18 19:36:10 indya-lb haproxy[13147]: Proxy StarBACK started.
>
>
>
> May 18 19:36:10 indya-lb haproxy[13147]: Proxy StagingBACK started.
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_1 stopped
(FE: 3886 conns, BE: 0 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_1BACK
stopped (FE: 0 conns, BE: 3583 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_2 stopped
(FE: 0 conns, BE: 0 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_2BACK
stopped (FE: 0 conns, BE: 0 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy Star stopped (FE:
60927284 conns, BE: 0 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy StarBACK stopped (FE: 0
conns, BE: 59690087 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy Staging stopped (FE: 0
conns, BE: 0 conns).
>
>
>
> May 18 19:36:10 indya-lb haproxy[7375]: Proxy StagingBACK stopped (FE:
0 conns, BE: 0 conns).
>
>
>
> May 18 20:09:32 indya-lb haproxy[13204]: Proxy ssl_frontend_2 started.
>
>
>
> May 18 20:09:32 indya-lb haproxy[13204]: Proxy ssl_frontend_2BACK
started.
>
>
>
> May 18 20:09:32 indya-lb haproxy[13204]: Proxy Star started.
>
>
>
> May 18 20:09:32 indya-lb haproxy[13204]: Proxy StarBACK started.
>
>
>
> May 18 20:09:32 indya-lb haproxy[13204]: Proxy Staging started.
>
>
>
> May 18 20:09:32 indya-lb haproxy[13204]: Proxy StagingBACK started.
>
>
>
>
>
> We are the using the init script to reload haproxy "service haproxy
reload" in centos and the script is as follows
>
>
>
> #!/bin/sh
>
>
>
> #
>
>
>
> # chkconfig: - 85 15
>
>
>
> # description: HA-Proxy is a TCP/HTTP reverse proxy which is
particularly
>
> suited
>
>
>
>\
>
>
>
> # for high availability environments.
>
>
>
> # processname: haproxy
>
>
>
> # config: /etc/haproxy.cfg
>
>
>
> # pidfile: /var/run/haproxy.pid
>
>
>
>
>
>
>
> # Source function library.
>
>
>
> if [ -f /etc/init.d/functions ]; then
>
>
>
> . /etc/init.d/functions
>
>
>
> elif [ -f /etc/rc.d/init.d/functions ] ; then
>
>
>
> . /etc/rc.d/init.d/functions
>
>
>
> else
>
>
>
> exit 0
>
>
>
> fi
>
>
>
>
>
>
>
> # Source networking configuration.
>
>
>
> . /etc/sysconfig/network
>
>
>
>
>
>
>
> # Check that networking is up.
>
>
>
> [ ${NETWORKING} = "no" ] && exit 0
>
> --
> CAUTION - Disclaimer * This e-mail
contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the
use of the addressee(s). If you are not the intended recipient, please
notify the sender by e-mail and delete the original message. Further, you
are not to copy, disclose, or distribute this e-mail or its contents to any
other person and any such actions are unlawful. This e-mail may contain
viruses. Netmagic Solutions Pvt. Ltd. has taken every reasonable precaution
to minimize this risk, but is not liable for any damage you may sustain as
a result of any virus in this e-mail. You should carry out your own virus
checks before opening the e-mail or attachment. Netmagic Solutions Pvt.
Ltd. reserves the right to mo
FW: problem with haproxy reload
Hi,
We faced with haproxy, we have a script which deletes the
frontend and backend entries of haproxy based on name and does a reload of
haproxy after haproxy file check is done.
In one such scenario after deleting the frontend and backend and reloading
we found that haproxy was in stop state
Below are the logs which shows the backend was started again during reload
but the frontends were not started and the same are shown in logs after
we manually restarted
haproxy
Any feedback regarding this will be very useful.
Regards
Senthil
May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend ssl_frontend_1 in
0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend
ssl_frontend_1BACK in 0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend ssl_frontend_2
in 0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend
ssl_frontend_2BACK in 0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend Star in 0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend StarBACK in 0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping frontend Staging in 0 ms.
May 18 19:36:10 indya-lb haproxy[7375]: Stopping backend StagingBACK in 0
ms.
May 18 19:36:10 indya-lb haproxy[13147]: Proxy ssl_frontend_2BACK started.
May 18 19:36:10 indya-lb haproxy[13147]: Proxy StarBACK started.
May 18 19:36:10 indya-lb haproxy[13147]: Proxy StagingBACK started.
May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_1 stopped (FE:
3886 conns, BE: 0 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_1BACK stopped
(FE: 0 conns, BE: 3583 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_2 stopped (FE:
0 conns, BE: 0 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy ssl_frontend_2BACK stopped
(FE: 0 conns, BE: 0 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy Star stopped (FE: 60927284
conns, BE: 0 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy StarBACK stopped (FE: 0
conns, BE: 59690087 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy Staging stopped (FE: 0
conns, BE: 0 conns).
May 18 19:36:10 indya-lb haproxy[7375]: Proxy StagingBACK stopped (FE: 0
conns, BE: 0 conns).
May 18 20:09:32 indya-lb haproxy[13204]: Proxy ssl_frontend_2 started.
May 18 20:09:32 indya-lb haproxy[13204]: Proxy ssl_frontend_2BACK started.
May 18 20:09:32 indya-lb haproxy[13204]: Proxy Star started.
May 18 20:09:32 indya-lb haproxy[13204]: Proxy StarBACK started.
May 18 20:09:32 indya-lb haproxy[13204]: Proxy Staging started.
May 18 20:09:32 indya-lb haproxy[13204]: Proxy StagingBACK started.
We are the using the init script to reload haproxy "service haproxy reload"
in centos and the script is as follows
#!/bin/sh
#
# chkconfig: - 85 15
# description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly
suited
\
# for high availability environments.
# processname: haproxy
# config: /etc/haproxy.cfg
# pidfile: /var/run/haproxy.pid
# Source function library.
if [ -f /etc/init.d/functions ]; then
. /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
. /etc/rc.d/init.d/functions
else
exit 0
fi
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0
[ -f /etc/haproxy.cfg ] || exit 1
RETVAL=0
start() {
/usr/sbin/haproxy -c -q -f /etc/haproxy.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file."
return 1
fi
echo -n "Starting HAproxy: "
daemon /usr/sbin/haproxy -D -f /etc/haproxy.cfg -p /var/run/haproxy.pid
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && touch /var/lock/subsys/haproxy
return $RETVAL
}
stop() {
echo -n "Shutting down HAproxy: "
killproc haproxy -USR1
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/haproxy
[ $RETVAL -eq 0 ] && rm -f /var/run/haproxy.pid
return $RETVAL
}
restart() {
/usr/sbin/haproxy -c -q -f /etc/haproxy.cfg
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with 'haproxy
check'."
return 1
fi
stop
start
}
check() {
/usr/sbin/haproxy -c -q -V -f /etc/haproxy.cfg
}
rhstatus() {
status haproxy
}
condrestart() {
[ -e /var/lock/subsys/haproxy ] && restart |
