Re: haproxy reloads, stale listeners, SIGKILL required

[Vincent Bernat]

 ❦  3 février 2016 00:11 GMT, David Birdsong  :

> I'm not using consul but am using haproxy in a docker container
> and reloading when backend hosts change registrations. I haven't
> seen this issue. I run using haproxy-systemd-wrapper and HUP that
> process to reload.
>
> does that mean the wrapper ensures that an old process exits and
> forces if it doesn't eventually?

No, the wrapper is only here to make HAProxy appears more like a
traditional daemon on the signal handling front. The old processes are
expected to stay to handle existing connections. Once you locate an old
process, check why it is here with lsof: lsof -n -p . You should
see that the process is not listening anymore and is only handling old
connections. If you kill the process, the connections will just break.
-- 
Follow each decision as closely as possible with its associated action.
- The Elements of Programming Style (Kernighan & Plauger)

Re: haproxy reloads, stale listeners, SIGKILL required

[David Birdsong]

On Wed, Feb 3, 2016 at 3:05 AM Vincent Bernat  wrote:

>  ❦  3 février 2016 00:11 GMT, David Birdsong  :
>
> > I'm not using consul but am using haproxy in a docker container
> > and reloading when backend hosts change registrations. I haven't
> > seen this issue. I run using haproxy-systemd-wrapper and HUP that
> > process to reload.
> >
> > does that mean the wrapper ensures that an old process exits and
> > forces if it doesn't eventually?
>
> No, the wrapper is only here to make HAProxy appears more like a
> traditional daemon on the signal handling front. The old processes are
> expected to stay to handle existing connections. Once you locate an old
> process, check why it is here with lsof: lsof -n -p . You should
> see that the process is not listening anymore and is only handling old
> connections. If you kill the process, the connections will just break.
>

right, thanks, but again, I'm familiar w/ haproxy and graceful reloads.
I've used lsof -Pnp , and I find that sometimes an old haproxy process
is still listening for new connections. This is the problem I'm trying to
ask if anyone else is also seeing.



> --
> Follow each decision as closely as possible with its associated action.
> - The Elements of Programming Style (Kernighan & Plauger)
>
>

cookie prefix & option httpclose

[Michał Pasierb]

Hi,

I have a question about cookie prefix mode. I use configuration like this:

backend b1
balance roundrobin
cookie JSESSIONID prefix
server s1 192.168.122.1:8080 cookie c1

Now the docs
https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4-cookie for
prefix say:

Since all requests and responses are subject to being modified, this
mode requires the HTTP close mode.

Behavior for version 1.4 is consistent with docs. The cookie was not
prefixed for second request within one http keepalive session so I needed
to add "option httpclose" to modify all requests.

Now I did the same setup with version 1.5 and found out that even without
"option httpclose" both requests have the cookie prefixed.

My dillema - are the docs inconsistent and this section should be removed
or am I missing something ?

Thanks,
Michal

Re: haproxy reloads, stale listeners, SIGKILL required

[Willy Tarreau]

Hi David,

On Tue, Feb 02, 2016 at 11:56:25PM +, David Birdsong wrote:
> Has nobody else run into this w/ consul? Given the plethora of tools around
> consul and haproxy and templating, I know others are using reloads to keep
> backend current, but the old haproxy PIDs stick around listening w/
> incorrect backends.

No and this behaviour is really weird. Are you sure the list of pids
passed after "-sf" is *always* correct ? Since you're running on a
kernel 3.13, it supports SO_REUSEPORT, so this means that it supports
multiple processes being bound to the same port. Thus, if sometimes
you pass the wrong pid or an incomplete list, it will not prevent the
new process from starting but the old process will stay there listening.

You should probably check using "ps" that remaining processes really
appear in the -sf line on the new process.

There's one similar case I've seen a long time ago, a customer was
using "-sf $(cat /var/run/haproxy.pid)" in the startup script, and
was running two independant instances accidently using the same
pid file. Thus one instance would overwrite the other one's pid
list and the reload would fail half of the times because the old
pids would not be properly signaled.

I don't know if that helps,
Willy

Re: Force client IP with PROXY protocol

[Willy Tarreau]

On Thu, Jan 28, 2016 at 12:25:05PM +0100, Aleksandar Lazic wrote:
> 
> 
> Am 28-01-2016 12:01, schrieb Jonathan Leroy - Inikup:
> >2016-01-28 11:47 GMT+01:00 Lukas Tribus :
> >>Doesn't:
> >>http-request set-src hdr(CF-Connecting-IP)
> >>
> >>in combination with a standard proxy-protocol config
> >>already do that?
> >
> >Yes, but it doesn't work with SPDY or HTTP/2 backends.
> 
> But then it is missleading that this only is possible whith mode http 
> not also for mode tcp.
> 
> To ask a clear question.
> 
> Do set the 'http-request set-src hdr(CF-Connecting-IP)' the IP in the 
> proxy protocol and in the tcp packet also in tcp mode?

No, set-src replaces the client's src as logged by haproxy and as passed
over the proxy protocol. The only issue is that this action was incompletely
implemented, it's only in http-request while it should also have been in
tcp-request. I hoped that we'd get it completed before the release but
apparently nobody was interested in finishing was was begun :-(

If someone is willing to do it for TCP mode and the patch is small enough,
I'm willing to backport it into 1.6 as I consider it almost a bug to only
be able to use it in HTTP mode.

With that said, Jonathan, you need to keep in mind that by doing so you
will pass the IP address presented by CF in the *first* request as the
source of the whole connection, hence all subsequent requests. So before
doing this you need to be absolutely sure that CF doesn't multiplex
incoming connections from various clients over the same connection. Maybe
this can be configured on cloudflare, I have no idea. And quite frankly,
just for a performance reason you should definitely make nginx aware of
the original client's IP address *per request* and not *per connection*
in order to allow CF to multiplex multiple requests over a single
connection.

Willy

Re: http-ignore-probes produces a warning in tcp frontend

[Willy Tarreau]

Hello Dmitry,

On Thu, Jan 28, 2016 at 05:31:58PM +0300, Dmitry Sivachenko wrote:
> Hello,
> 
> I have an option http-ignore-probes in defaults section.
> When I declare frontend in "tcp" mode, I get the following warning:
> 
> [WARNING] 027/172718 (18281) : config : 'option http-ignore-probes' ignored
> for frontend 'MYTEST-front' as it requires HTTP mode.
> 
> In defaults section I have other http-specific options (e.g.
> http-keep-alive), which does not produce a warning in tcp backend.
> Is it intended?  It looks logical to produce such a warning only if
> http-specific option is used directly in tcp backend and silently ignore it
> when used in defaults.

There's no difference between having the option in defaults or explicitly
in the section itself. You should see defaults as templates for next
sections. The error here is that http-keep-alive should also produce a
warning. But I think I know why it doesn't, most options are handled by
a generic parser which checks the proxy mode, and a few other more
complex ones are implemented "by hand" and do not necessarily run such
checks.

It's a very bad practise to mix TCP and HTTP proxies with the same defaults
sections. This probably is something we should document better in the doc.
A good practise is to have one (or several) defaults sections for HTTP mode
and then other defaults sections for TCP mode. And most often you don't even
have the same timeouts, log settings etc.

Regards,
Willy

Re: Stick-table peers expiration time

[Willy Tarreau]

On Wed, Feb 03, 2016 at 11:05:03AM +1100, Igor Cicimov wrote:
> Any comments on this? Shouldn't the expiration time get replicated upon
> restart or just the keys? Obviously the entry in the first server stick
> table will expire much sooner than the one on the restarted server meaning
> the first one might choose different backend for the same session id.

I could be wrong (need to check with Emeric who started documenting the
peers protocol) but from what I remember, the expiration times are *not*
passed over the protocol, and each peer creates the entry with its local
expiration timer. The reason is simple, you push updates when you see
them, so these ones are supposed to be fresh when you receive them.
However maybe an exception should indeed be made during the first learning
phase so that timers could be passed along with contents (and bounded by
the table's expiration value). I don't know if that would be possible
without changing the protocol again though :-/

Willy

Re: Unexpected haproxy 1.6.3 behaviour with -sf

[Willy Tarreau]

Hi,

On Sun, Jan 31, 2016 at 09:16:23PM +, Cain wrote:
> Hi,
> 
> Sometimes I find i am left with zombie haproxy instances that are *still
> listening* and do *not* have any established connections (netstat told me
> so), even though those particular pids have been told to die.
> 
> I.e.
> 
>  141 ?Ss 0:00 /usr/sbin/haproxy -D -f /proxy.conf -p /proxy.pid
> -sf 139
>  143 ?Ss 0:00 /usr/sbin/haproxy -D -f /proxy.conf -p /proxy.pid
> -sf 141
> 
> It is worth mentioning this is running in a Dockerfile with a go
> application as the parent (interlock).

Are you absolutely sure pid 141 above is still listening ?

(...)
> I've tried ensuring that new haproxy instances are only started at most
> once every 5 seconds (or even 10 seconds), in an attempt to rule out a race
> condition, but no dice.
> 
> Below are pastes of two strace outputs. The first is tracing (just
> capturing signal related calls) the parent (pid 1) and all the children,
> note that process 284 gets nothing even when PID 285 kills it.

Indeed, but what is 284 supposed to be here ? We don't see *any* activity
on it in this trace. I'm seeing 283 killing 176, and 285 killing 284.
Maybe the parameter passed to 285 is simply wrong and 284 is not even
haproxy but the shell script that calls it or your go application ?

> Process 1 attached with 9 threads
> Process 283 attached
(...)
> [pid   283] kill(176, SIGUSR1)  = 0
> [pid   283] +++ exited with 0 +++
> [pid66] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=283,
> si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
> [pid66] rt_sigreturn()  = 283
> Process 284 attached
> Process 285 attached
(...)
> [pid   285] kill(284, SIGUSR1)  = 0
> Process 286 attached
> [pid   285] +++ exited with 0 +++
> [pid66] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=285,
> si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
> [pid66] rt_sigreturn()  = 285
(...)

> In the
> second trace, im capturing everything, note that the process gets the
> SIGUSR1 signal, but continues on anyway. Hopefully these shed some light on
> what is occurring?
> 

> epoll_wait(0, 2191e30, 200, 1000)   = -1 EINTR (Interrupted system call)
> --- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_USER, si_pid=1120, si_uid=0} ---
> rt_sigaction(SIGUSR1, {0x46bb90, [USR1], SA_RESTORER|SA_RESTART,
> 0x7f437988c180}, {0x46bb90, [USR1], SA_RESTORER|SA_RESTART,
> 0x7f437988c180}, 8) = 0
> rt_sigreturn()  = -1 EINTR (Interrupted system call)
> rt_sigprocmask(SIG_SETMASK, ~[PROF RTMIN RT_1], [], 8) = 0
> brk(0x21d9000)  = 0x21d9000
> rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> epoll_ctl(0, EPOLL_CTL_DEL, 4, 6be6e0)  = 0
> close(4)= 0
> epoll_ctl(0, EPOLL_CTL_DEL, 5, 6be6e0)  = 0
> close(5)= 0
> epoll_ctl(0, EPOLL_CTL_DEL, 6, 6be6e0)  = 0
> close(6)= 0

As you can see it has immediately closed these 3 listening sockets (FDs
4, 5 and 6) and you don't see any accept anymore, just regular activity
on already established connections, so the behaviour here is totally
normal and expected.

> epoll_wait(0, {}, 200, 1000)= 0
> epoll_wait(0, {}, 200, 1000)= 0
> epoll_wait(0, {{EPOLLIN, {u32=2, u64=2}}}, 200, 386) = 1
> recvfrom(2,
> "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., 16384, 0,
> NULL, NULL) = 44
> sendto(1, "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"...,
> 44, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 44
> epoll_wait(0, {{EPOLLIN, {u32=9, u64=9}}}, 200, 40) = 1
> recvfrom(9,
> "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., 16384, 0,
> NULL, NULL) = 44
> sendto(8, "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"...,
> 44, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 44
> epoll_wait(0, {{EPOLLIN, {u32=7, u64=7}}}, 200, 32) = 1
> recvfrom(7,
> "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., 16384, 0,
> NULL, NULL) = 44
(...)

> 
> Hopefully these traces tell someone something, i'm at a complete loss as to
> how this is occurring / what else to try to debug this.

I suspect that you might have caught a wrong trace of what you thought
was a bug while trying to debug this issue. It is important to find out
why above you see pid 285 send SIGUSR1 to pid 284 while there's absolutely
no activity on pid 284, which doesn't appear to be haproxy at first glance.
Please double-check how the pid list is built before passing it to -sf, I
suspect you'll discover something there.

Regards,
Willy

Re: [PATCH] DOC: remove old tunnel mode assumptions

[Willy Tarreau]

Hi Lukas,

On Wed, Feb 03, 2016 at 06:09:37PM +0100, Lukas Tribus wrote:
> Micha?? Pasierb reported doc inconsistencies regarding the old default
> HTTP tunnel mode.
> 
> This patch fixes a few of those inconsistencies and should be backported
> to both 1.6 and 1.5.

Applied to 1.7-dev, thanks!
Willy

RE: cookie prefix & option httpclose

[Lukas Tribus]

> Now I did the same setup with version 1.5 and found out that even 
> without "option httpclose" both requests have the cookie prefixed. 
> 
> My dillema - are the docs inconsistent and this section should be 
> removed or am I missing something ?

Docs are inconsistens both 1.5 and 1.6 for this.

Thanks for the report, will fix.


Lukas

  

Re: haproxy reloads, stale listeners, SIGKILL required

[Vincent Bernat]

 ❦  3 février 2016 14:37 GMT, David Birdsong  :

> right, thanks, but again, I'm familiar w/ haproxy and graceful
> reloads. I've used lsof -Pnp , and I find that sometimes an old
> haproxy process is still listening for new connections. This is the
> problem I'm trying to ask if anyone else is also seeing.

My bad, I didn't read the whole thread before answering.
-- 
Use statement labels that mean something.
- The Elements of Programming Style (Kernighan & Plauger)

[PATCH] DOC: remove old tunnel mode assumptions

[Lukas Tribus]

Michał Pasierb reported doc inconsistencies regarding the old default
HTTP tunnel mode.

This patch fixes a few of those inconsistencies and should be backported
to both 1.6 and 1.5.
---
 doc/configuration.txt | 28 
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 5040f76..7dd5744 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -2620,11 +2620,11 @@ cookie  [ rewrite | insert | prefix ] [ indirect 
] [ nocache ]
   of complex combinations of "Set-cookie" and "Cache-control"
   headers is left to the application. The application can then
   decide whether or not it is appropriate to emit a persistence
-  cookie. Since all responses should be monitored, this mode only
-  works in HTTP close mode. Unless the application behaviour is
-  very complex and/or broken, it is advised not to start with this
-  mode for new deployments. This keyword is incompatible with
-  "insert" and "prefix".
+  cookie. Since all responses should be monitored, this mode
+  doesn't work in HTTP tunnel mode. Unless the application
+  behaviour is very complex and/or broken, it is advised not to
+  start with this mode for new deployments. This keyword is
+  incompatible with "insert" and "prefix".
 
 insertThis keyword indicates that the persistence cookie will have to
   be inserted by haproxy in server responses if the client did not
@@ -2650,7 +2650,7 @@ cookie  [ rewrite | insert | prefix ] [ indirect ] 
[ nocache ]
   and a delimiter. The prefix will be removed from all client
   requests so that the server still finds the cookie it emitted.
   Since all requests and responses are subject to being modified,
-  this mode requires the HTTP close mode. The "prefix" keyword is
+  this mode doesn't work with tunnel mode. The "prefix" keyword is
   not compatible with "rewrite" and "insert". Note: it is highly
   recommended not to use "indirect" with "prefix", otherwise server
   cookie updates would not be sent to clients.
@@ -5347,11 +5347,9 @@ no option http-use-proxy-header
 
   By setting this option in a frontend, haproxy can automatically switch to use
   that non-standard header if it sees proxied requests. A proxied request is
-  defined here as one where the URI begins with neither a '/' nor a '*'. The
-  choice of header only affects requests passing through proxies making use of
-  one of the "httpclose", "forceclose" and "http-server-close" options. Note
-  that this option can only be specified in a frontend and will affect the
-  request along its whole life.
+  defined here as one where the URI begins with neither a '/' nor a '*'. This
+  is incompatible with the HTTP tunnel mode. Note that this option can only be
+  specified in a frontend and will affect the request along its whole life.
 
   Also, when this option is set, a request which requires authentication will
   automatically switch to use proxy authentication headers if it is itself a
@@ -5500,10 +5498,8 @@ no option http_proxy
 
   No host address resolution is performed, so this only works when pure IP
   addresses are passed. Since this option's usage perimeter is rather limited,
-  it will probably be used only by experts who know they need exactly it. Last,
-  if the clients are susceptible of sending keep-alive requests, it will be
-  needed to add "option httpclose" to ensure that all requests will correctly
-  be analyzed.
+  it will probably be used only by experts who know they need exactly it. This
+  is incompatible with the HTTP tunnel mode.
 
   If this option has been enabled in a "defaults" section, it can be disabled
   in a specific instance by prepending the "no" keyword before it.
@@ -14773,7 +14769,7 @@ Most common cases :
 spent accepting these connections will inevitably slightly delay processing
 of other connections, and it can happen that request times in the order of
 a few tens of milliseconds are measured after a few thousands of new
-connections have been accepted at once. Setting "option http-server-close"
+connections have been accepted at once. Using one of the keep-alive modes
 may display larger request times since "Tq" also measures the time spent
 waiting for additional requests.
 
-- 
1.9.1

RE: [PATCH] DOC: remove old tunnel mode assumptions

[Lukas Tribus]

Hi Willy,


>> This patch fixes a few of those inconsistencies and should be backported
>> to both 1.6 and 1.5.
>
> Applied to 1.7-dev, thanks!

thanks.

A minor cosmetic glitch: seems that something broke non-ascii characters,
I can see it in this commit here and also in Cyril's latest patch, while
Cyril's penultimate patch (b65e033 from December) is fine:

git log --author cyril.bo...@free.fr -2



cheers,
lukas

  

Re: [PATCH] DOC: remove old tunnel mode assumptions

[Willy Tarreau]

Hi Lukas,

On Wed, Feb 03, 2016 at 07:47:37PM +0100, Lukas Tribus wrote:
> Hi Willy,
> 
> 
> >> This patch fixes a few of those inconsistencies and should be backported
> >> to both 1.6 and 1.5.
> >
> > Applied to 1.7-dev, thanks!
> 
> thanks.
> 
> A minor cosmetic glitch: seems that something broke non-ascii characters,
> I can see it in this commit here and also in Cyril's latest patch, while
> Cyril's penultimate patch (b65e033 from December) is fine:
> 
> git log --author cyril.bo...@free.fr -2

Yes, I'm seeing some non-printable unicode chars from time to time, I
don't know how they display on each one's terminal. If I enable utf-8
encoding in my xterm, I'm seeing them fine. But anyway I don't care
that much as long as it's only once in a while, it depends a lot on 
various things including e-mail encoding, heuristics, hand-edited
patches, etc.

Thanks,
Willy