High client request times (Tq) for no apparent reason
Hi everyone,
I’ve been seeing abnormally high client request times, up to anywhere below
8000 ms (that’s when we timeout the client request). To cut the story short I
setup a backend that just replies a short string with the URL back at the
client, and on my own laptop started accessing it:
http:///url1,
http:///url2,
http:///url3…
*All* the responses literally *immediately* showed up on the browser:
helloworld_url1, helloworld_url2, helloworld_3…
But turns out that some of these requests are logged by HAProxy with abnormally
high Tq (and consequently Tt), that just don’t correspond to the experience I’m
seeing in my browser doing this test. An example with 10 requests I’ve done:
Mar 20 11:36:46 ha-2.proxy haproxy[9031]: :3959 2347-532AD2CC-0011
[20/Mar/2014:11:36:44.569] internet clientapp/lighty-3 1709/0/0/1/1710 200 200
- - 1/1/0/1/0 0/0 {} "GET /url10 HTTP/1.1"
Mar 20 11:36:44 ha-2.proxy haproxy[9031]: :3959 2347-532AD2CA-0010
[20/Mar/2014:11:36:42.851] internet clientapp/lighty-2 1717/0/1/1/1719 200 199
- - 1/1/0/1/0 0/0 {} "GET /url9 HTTP/1.1"
Mar 20 11:36:42 ha-2.proxy haproxy[9031]: :3959 2347-532AD2C8-000F
[20/Mar/2014:11:36:40.638] internet clientapp/lighty-3 2210/0/1/1/2212 200 199
- - 1/1/0/1/0 0/0 {} "GET /url8 HTTP/1.1"
Mar 20 11:36:40 ha-2.proxy haproxy[9031]: :3959 2347-532AD2C6-000E
[20/Mar/2014:11:36:38.827] internet clientapp/lighty-2 1810/0/1/1/1812 200 199
- - 1/1/0/1/0 0/0 {} "GET /url7 HTTP/1.1"
Mar 20 11:36:38 ha-2.proxy haproxy[9031]: :3959 2347-532AD2C5-000D
[20/Mar/2014:11:36:37.209] internet clientapp/lighty-3 1615/0/1/1/1617 200 199
- - 1/1/0/1/0 0/0 {} "GET /url6 HTTP/1.1"
Mar 20 11:36:37 ha-2.proxy haproxy[9031]: :3959 2347-532AD2C3-000C
[20/Mar/2014:11:36:35.681] internet clientapp/lighty-2 1527/0/0/2/1529 200 199
- - 1/1/0/1/0 0/0 {} "GET /url5 HTTP/1.1"
Mar 20 11:36:35 ha-2.proxy haproxy[9031]: :3959 2347-532AD2C3-000B
[20/Mar/2014:11:36:35.678] internet clientapp/lighty-3 1/0/0/2/3 200 199 - -
1/1/0/1/0 0/0 {} "GET /url4 HTTP/1.1"
Mar 20 11:34:56 ha-2.proxy haproxy[9031]: :33161 2347-532AD25E-0009
[20/Mar/2014:11:34:54.581] internet clientapp/lighty-2 2349/0/0/2/2351 200 199
- - 1/1/0/1/0 0/0 {} "GET /url3 HTTP/1.1"
Mar 20 11:34:54 ha-2.proxy haproxy[9031]: :33161 2347-532AD25E-0008
[20/Mar/2014:11:34:54.579] internet clientapp/lighty-3 0/0/0/2/2 200 199 - -
1/1/0/1/0 0/0 {} "GET /url2 HTTP/1.1"
Mar 20 11:34:42 ha-2.proxy haproxy[9031]: :48491 2347-532AD252-0006
[20/Mar/2014:11:34:42.784] internet clientapp/lighty-2 0/0/0/2/2 200 199 - -
1/1/0/1/0 0/0 {} "GET /url1 HTTP/1.1”
I can show you another one that logged 5588 ms:
Mar 20 11:41:36 ha-2.proxy haproxy[9031]: :55502 2347-532AD3EA-0014
[20/Mar/2014:11:41:30.974] internet clientapp/lighty-3 5588/0/1/1/5590 200 200
- - 1/1/0/1/0 0/0 {} "GET /url12 HTTP/1.1”
I initially thought this could be something to do with keep alive between
client and HAProxy (wasn't sure how HAProxy would handle logging in this
situation) but I believe I turned off keep alive just before I ran these tests.
I was using "option http-server-close" in the defaults section and I commented
that out and put in "no option http-keep-alive”.
Any thoughts on this?
Thank you,
Pedro.
What makes exploring 40 resorts so unique? Hear our members speak
If you're having trouble viewing this email, please At Club Mahindra, holidays come with benefits for every member. Experience magical moments in our exquisite resorts for 7 days every year for the next 25 years. With over 40 resorts at the most beautiful and diverse locations across the country, you can experience a different kind of holiday every year. You can now pay easy EMIs and enjoy the benefit of inflation-free holidays. Become a member and go on a perfect family holiday every year. You'll definitely come back with memories that last forever. www.clubmahindra.com Follow us on /clubmahindra To unsubscribe from this offer, please click here to unsubscribe.
recent test for dev22 on BSD
Hi,lists, I tested dev22 on FreeBSD 10-stable recently, and found: 1. "ipfw fwd" works well with dev22+tproxy. It's have a nice guide in the /usr/local/share/examples. But pf's divert-to and divert-reply can't work with haproxy. Maybe haproxy does not use "getsockname(2)" and "setsockopt(2)". 2. There are some issue with "option http-server-close", haproxy crashed after a while, whennever set it on frontend or backend. 3. Sometimes stalled with "tcp-smart-connect" and "tcp-smart-accept", when I removed it, it's work normal. But I am not sure about it. 4.The dev22 can compiled on DragonflyBSD, but it's silent stalled. Regards Simon
Re: recent test for dev22 on BSD
Hi Simon,
1- pf divert-reply,
The issue with pf i think is the following text 'yyerror("divert-reply
has no meaning in FreeBSD pf(4)");' in /sbin/pfctl/parse.y
Was surprised to find it. Even though the option is listed on the man pages.
So it sounds like it is not possible with a current native FreeBSD install.
I anyway haven't been successful in writing a firewallrule that contains
divert-reply and can be validated by pfctl. I do think that would be the
best solution reading the text in the man page it seems exactly what is
needed.. Catching reply packets for a nonlocal socket.
Where you able to write such a firewall rule?
I'm now trying to manually merge a old patch i found online..
http://lists.freebsd.org/pipermail/freebsd-net/2009-June/022166.html
Seems some parts are present already while other are 'missing' in the
current sources...
Not sure if that will work out working as some bitflags are used already
for other options and i am not sure if thats resolvable by me.
2- crash on FreeBSD,
Ive not seen it myself, but another user did report on FreeBSD8.3
(pfSense 2.1) that he also experiences crashes with dev20 and dev22 .
https://forum.pfsense.org/index.php?topic=73927.0
About 3 and 4 i have no clue..
Greets
PiBa-NL
k simon schreef op 20-3-2014 16:12:
> Hi,lists,
> I tested dev22 on FreeBSD 10-stable recently, and found:
> 1. "ipfw fwd" works well with dev22+tproxy. It's have a nice guide in
> the /usr/local/share/examples.
> But pf's divert-to and divert-reply can't work with haproxy. Maybe
> haproxy does not use "getsockname(2)" and "setsockopt(2)".
>
> 2. There are some issue with "option http-server-close", haproxy crashed
> after a while, whennever set it on frontend or backend.
>
> 3. Sometimes stalled with "tcp-smart-connect" and "tcp-smart-accept",
> when I removed it, it's work normal. But I am not sure about it.
>
> 4.The dev22 can compiled on DragonflyBSD, but it's silent stalled.
>
>
>
> Regards
> Simon
>
du nouveau pour votre prospection commerciale
Title: XRM Bonjour Vous êtes une petite équipe commerciale et vous n'avez pas d'outil adapté pour gérer votre prospection commerciale ? Je vous invite à découvrir XRM10. C'est une solution CRM simple, sans rien à installer, qui permet de gérer facilement et efficacement le suivi de ses prospects. Vous pouvez de plus alimenter la base en leads (télémarketing ou emailing) grâce à des interconnections avec des outils professionnels de défrichage. Visitez notre site, cela ne vous engage à rien Bien cordialement P.AUGUSTIN
RE: recent test for dev22 on BSD
Hi Simon, > 2. There are some issue with "option http-server-close", haproxy crashed > after a while, whennever set it on frontend or backend. For crashes, please provide gdb backtraces [1]. > 3. Sometimes stalled with "tcp-smart-connect" and "tcp-smart-accept", > when I removed it, it's work normal. But I am not sure about it. What does that exactly mean? Does a single request stall or the whole haproxy service? Does it eat a lot of CPU in this condition (user or kernel)? > 4. The dev22 can compiled on DragonflyBSD, but it's silent stalled. Same as with 3. - we need more informations about this (including the output of ./haproxy -vv, an explanation of what you are trying to achieve and at least a relevant part of your configuration). Regards, Lukas [1] http://www.freebsd.org/doc/en/books/developers-handbook/debugging.html
Goodbye from our Newsletter
Goodbye from our Newsletter, sorry to see you go. You have been unsubscribed from our newsletters. This is the last email you will receive from us. Our newsletter system, phpList, will refuse to send you any further messages, without manual intervention by our administrator. If there is an error in this information, you can re-subscribe: please go to http://tuinforme.com.ar/lists/?p=subscribe and follow the steps. Thank you
