Re: HAProxy 1.7.9 FreeBSD 100% CPU usage

[Willy Tarreau]

Hi Pieter,

On Fri, Nov 10, 2017 at 11:43:41PM +0100, PiBa-NL wrote:
> Okay have been running with haproxy-ss-20171017 for a day now. Sofar it
> sticks to <1% cpu usage.
> 
> Will report if anything changes, cant tell for sure as don't have a clear
> reproduction of the issue, but my issue 'seems' fixed.

Great! If anything changes and you want to report what is wrong, please
don't forget to issue the following :
   - "show sess all" on the cli
   - attach gdb to the running process and issue a few series of "bt full",
 "cont", ctrl-c, "bt full", ... to try to catch the code path commonly
 called.

Cheers,
Willy

Re: HAProxy 1.7.9 FreeBSD 100% CPU usage

[PiBa-NL]

Hi Willy,

Op 9-11-2017 om 5:45 schreef Willy Tarreau:

Hi Pieter,

We had something similar on Linux in relation with TCP splicing and the
fd cache, for which a fix was emitted. But yesterday Christopher explained
me that the fix has an impact on the way applets are scheduled in 1.8, so
actually it could mean that the initial bug might possibly cover a larger
scope than splicing only, and that recv+send might also be affected. If
you're interested in testing, the commit in 1.7 is
c040c1f ("BUG/MAJOR: stream-int: don't re-arm recv if send fails") and
is present in the latest snapshot (we really need to emit 1.7.10 BTW).

I'd be curious to know if it fixes it or not. At least it will tell us
if that's related to this fd cache thing or to something completely
different such as Lua.

I also need to check with Thierry if we could find a way to add some
stats about the time spent in Lua to "show info" to help debugging such
cases where Lua is involved.

By the way, thanks for your dump, we'll check the sessions' statuses.
There are not that many, and maybe it will give us a useful indication!

Cheers,
Willy


Okay have been running with haproxy-ss-20171017 for a day now. Sofar it 
sticks to <1% cpu usage.


Will report if anything changes, cant tell for sure as don't have a 
clear reproduction of the issue, but my issue 'seems' fixed.


Regards,

PiBa-NL / Pieter

Re: core.sleep() ignores param and sleeps for 10 seconds in response action

[Nick Dimov]

thank you for the explanation.


On 10.11.2017 19:28, Willy Tarreau wrote:
> On Fri, Nov 10, 2017 at 07:05:39PM +0200, Nick Dimov wrote:
>> For example, I use a backend like this
>>
>>> backend b_http_hosts
>>>     mode http
>>>
>>>     server s_web1  server:80 maxconn 10 check
>> And I'm testing with wrk -c 20 and checking the established connections
>> from haproxy to apache (with netstat), and that number is always equal
>> to what i specify in -c param for wrk.
>>
>> It's worth mentioning that it respects the number of simultaneous
>> requests (i checked that specifically) but is this behavior normal? Is
>> it ok that it creates more connections than specified in maxconn ?
> Yes, it's expected as soon as you have enabled keep-alive on the backend.
> Counting the number of idle connections in the limit would be counter-
> productive and even prevent new requests from being served while some
> idle connections remain. In the end, since most servers have accept queues
> and dispatch requests to threads, it makes more sense to only count what
> really matters, ie outstanding requests. However if you enable connection
> sharing using "http-reuse", you'll see that the number of outstanding
> requests is much closer to the number of connections.
>
> Regards
> Willy

Re: core.sleep() ignores param and sleeps for 10 seconds in response action

[Willy Tarreau]

On Fri, Nov 10, 2017 at 07:05:39PM +0200, Nick Dimov wrote:
> For example, I use a backend like this
> 
> > backend b_http_hosts
> >     mode http
> >
> >     server s_web1  server:80 maxconn 10 check
> 
> And I'm testing with wrk -c 20 and checking the established connections
> from haproxy to apache (with netstat), and that number is always equal
> to what i specify in -c param for wrk.
> 
> It's worth mentioning that it respects the number of simultaneous
> requests (i checked that specifically) but is this behavior normal? Is
> it ok that it creates more connections than specified in maxconn ?

Yes, it's expected as soon as you have enabled keep-alive on the backend.
Counting the number of idle connections in the limit would be counter-
productive and even prevent new requests from being served while some
idle connections remain. In the end, since most servers have accept queues
and dispatch requests to threads, it makes more sense to only count what
really matters, ie outstanding requests. However if you enable connection
sharing using "http-reuse", you'll see that the number of outstanding
requests is much closer to the number of connections.

Regards
Willy

Re: core.sleep() ignores param and sleeps for 10 seconds in response action

[Nick Dimov]

Hi again,

Since I'm still on this, I'm having another possible issue with haproxy.
The documentation states:

> When a server has a 
> "maxconn" 
> parameter specified, it means that its number
> of concurrent connections *will never go higher*.
But in my experiments, the haproxy creates more connections to the
backend than specified in maxconn.

For example, I use a backend like this

> backend b_http_hosts
>     mode http
>
>     server s_web1  server:80 maxconn 10 check

And I'm testing with wrk -c 20 and checking the established connections
from haproxy to apache (with netstat), and that number is always equal
to what i specify in -c param for wrk.

It's worth mentioning that it respects the number of simultaneous
requests (i checked that specifically) but is this behavior normal? Is
it ok that it creates more connections than specified in maxconn ?

Regards,

Nick.



On 10.11.2017 18:57, Willy Tarreau wrote:
> On Fri, Nov 10, 2017 at 06:43:42PM +0200, Nick Dimov wrote:
>> Hello,
>>
>> The patch works great. I tested on 1.8 and 1.7 and both are working good
>> now.
> thanks for confirming!
>
> Willy

Re: core.sleep() ignores param and sleeps for 10 seconds in response action

[Willy Tarreau]

On Fri, Nov 10, 2017 at 06:43:42PM +0200, Nick Dimov wrote:
> Hello,
> 
> The patch works great. I tested on 1.8 and 1.7 and both are working good
> now.

thanks for confirming!

Willy

Re: core.sleep() ignores param and sleeps for 10 seconds in response action

[Nick Dimov]

Hello,

The patch works great. I tested on 1.8 and 1.7 and both are working good
now.

Thanks a lot!


On 10.11.2017 18:20, Willy Tarreau wrote:
> Hello Nick,
>
> On Fri, Nov 10, 2017 at 04:50:37PM +0200, Nick Dimov wrote:
>> Hello, everyone.
>>
>> I am encountering a problem with LUA in haproxy, I also reported it here
>> https://github.com/sflow/haproxy/issues/2 but the problem is lieke this:
>>
>> When using a response action, this function - sleeps for 10 seconds, no
>> matter what param i pass to it. Also it seems that the wait time always
>> equals timeout connect. The sample config is:
>>
>> global
>> daemon
>> log /dev/log local6
>> lua-load /etc/haproxy/delay.lua
>>
>> defaults
>> mode http
>> timeout connect 1ms
>>
>> frontend fe
>> bind *:80
>> mode http
>> default_backend b_http_hosts
>>
>> backend b_http_hosts
>> mode http
>> http-response lua.delay_response
>> server s_web1 server:80 check
>>
>> and the LUA code:
>>
>> function delay_response(txn)
>> core.msleep(1)
>> end
>>
>> core.register_action("delay_response", {"tcp-res", "http-res" },
>> delay_response);
>>
>> Note that if core.msleep() is commented out - everything works as expected.
>>
>> I tested version 1.6 (it hangs 30 seconds there), 1.7 - matches timeout
>> connect, and 1.8 - same as 1.7.
>>
>> Any idea how to overcome this problem? All i need is to delay the
>> responses based on information from backend header.
> I've checked and in fact it's been like this forever, it's just that Lua
> uncovered it :-)  Basically the response analyse timeout was never handled
> in process_stream().
>
> I've just fixed it upstream now and verified that your example above
> correctly pauses for delays smaller than the connect timeout.
>
> You can apply the attached patch, it should work on 1.7 as well.
>
> Thanks for reporting this!
> Willy

Re: option redispatch clarification

[Willy Tarreau]

Hello Arthur,

On Fri, Nov 10, 2017 at 06:05:06PM +0200, Arthur Titeica wrote:
> Hello,
> 
> I'm trying to understand if the redispatch option may be used to retry a HTTP 
> connection in the following scenario:
> 
> Suppose there are multiple backend web servers and the first one that gets 
> chosen has a problem and replies with a 4xx or 5xx http status code.
> 
> Will the connection be retried to the next backend webserver in the hope that 
> everything will work OK and we get a http 200 reply?

No if any single byte was sent to the server (hence lost). Yes if the error
happened before the request was sent (eg: connection refused, timeout).

> If this doesn't work is there some other mechanism to achieve something like 
> this?

Aside the fact that you MUST never ever do this on non-idempotent requests,
I don't see an easy way to do it as it would require to keep a complete copy
of the request and to try to send it again. We've already thought about ugly
solutions like rebuilding a new request upon certain responses, but for now
there's nothing satisfying.

Regards,
Willy

Re: core.sleep() ignores param and sleeps for 10 seconds in response action

[Willy Tarreau]

Hello Nick,

On Fri, Nov 10, 2017 at 04:50:37PM +0200, Nick Dimov wrote:
> Hello, everyone.
> 
> I am encountering a problem with LUA in haproxy, I also reported it here
> https://github.com/sflow/haproxy/issues/2 but the problem is lieke this:
> 
> When using a response action, this function - sleeps for 10 seconds, no
> matter what param i pass to it. Also it seems that the wait time always
> equals timeout connect. The sample config is:
> 
> global
> daemon
> log /dev/log local6
> lua-load /etc/haproxy/delay.lua
> 
> defaults
> mode http
> timeout connect 1ms
> 
> frontend fe
> bind *:80
> mode http
> default_backend b_http_hosts
> 
> backend b_http_hosts
> mode http
> http-response lua.delay_response
> server s_web1 server:80 check
> 
> and the LUA code:
> 
> function delay_response(txn)
> core.msleep(1)
> end
> 
> core.register_action("delay_response", {"tcp-res", "http-res" },
> delay_response);
> 
> Note that if core.msleep() is commented out - everything works as expected.
> 
> I tested version 1.6 (it hangs 30 seconds there), 1.7 - matches timeout
> connect, and 1.8 - same as 1.7.
> 
> Any idea how to overcome this problem? All i need is to delay the
> responses based on information from backend header.

I've checked and in fact it's been like this forever, it's just that Lua
uncovered it :-)  Basically the response analyse timeout was never handled
in process_stream().

I've just fixed it upstream now and verified that your example above
correctly pauses for delays smaller than the connect timeout.

You can apply the attached patch, it should work on 1.7 as well.

Thanks for reporting this!
Willy
>From 9a398beac321fdda9f6cf0cb7069960d1a29cfd6 Mon Sep 17 00:00:00 2001
From: Willy Tarreau 
Date: Fri, 10 Nov 2017 17:14:23 +0100
Subject: BUG/MEDIUM: stream: don't ignore res.analyse_exp anymore

It happens that no single analyser has ever needed to set res.analyse_exp,
so that process_stream() didn't consider it when computing the next task
expiration date. Since Lua actions were introduced in 1.6, this can be
needed on http-response actions for example, so let's ensure it's properly
handled.

Thanks to Nick Dimov for reporting this bug. The fix needs to be
backported to 1.7 and 1.6.
---
 src/stream.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/stream.c b/src/stream.c
index 48c4ba5..19d94f3 100644
--- a/src/stream.c
+++ b/src/stream.c
@@ -2427,6 +2427,8 @@ struct task *process_stream(struct task *t)
 
t->expire = tick_first(t->expire, req->analyse_exp);
 
+   t->expire = tick_first(t->expire, res->analyse_exp);
+
if (si_f->exp)
t->expire = tick_first(t->expire, si_f->exp);
 
-- 
1.7.12.1

option redispatch clarification

[Arthur Țițeică]

Hello,

I'm trying to understand if the redispatch option may be used to retry a HTTP 
connection in the following scenario:

Suppose there are multiple backend web servers and the first one that gets 
chosen has a problem and replies with a 4xx or 5xx http status code.

Will the connection be retried to the next backend webserver in the hope that 
everything will work OK and we get a http 200 reply?

If this doesn't work is there some other mechanism to achieve something like 
this?

Thanks
Arthur

core.sleep() ignores param and sleeps for 10 seconds in response action

[Nick Dimov]

Hello, everyone.

I am encountering a problem with LUA in haproxy, I also reported it here
https://github.com/sflow/haproxy/issues/2 but the problem is lieke this:

When using a response action, this function - sleeps for 10 seconds, no
matter what param i pass to it. Also it seems that the wait time always
equals timeout connect. The sample config is:

global
daemon
log /dev/log local6
lua-load /etc/haproxy/delay.lua

defaults
mode http
timeout connect 1ms

frontend fe
bind *:80
mode http
default_backend b_http_hosts

backend b_http_hosts
mode http
http-response lua.delay_response
server s_web1 server:80 check

and the LUA code:

function delay_response(txn)
core.msleep(1)
end

core.register_action("delay_response", {"tcp-res", "http-res" },
delay_response);

Note that if core.msleep() is commented out - everything works as expected.

I tested version 1.6 (it hangs 30 seconds there), 1.7 - matches timeout
connect, and 1.8 - same as 1.7.

Any idea how to overcome this problem? All i need is to delay the
responses based on information from backend header.

Reegards!

Amazon Web Services Email Db

[Janet Chua]

 

 

Hi,

Would you be interested in an Email list of Amazon Web Services 2017?

 

Please fill in the blanks__ below, target market criteria you would be
interested in reaching:

Target Criteria:-

 

Job titles _?

Industry markets ___?

Geography (Country / State): __?

 

Looking forward to hearing from you.

 

Thanks,

Janet Chua

Database Coordinator

 

If you don't wish to receive our newsletters, reply back with Leave-Out in
subject line.