Re: Unexpected haproxy 1.6.3 behaviour with -sf
Hi, On Sun, Jan 31, 2016 at 09:16:23PM +, Cain wrote: > Hi, > > Sometimes I find i am left with zombie haproxy instances that are *still > listening* and do *not* have any established connections (netstat told me > so), even though those particular pids have been told to die. > > I.e. > > 141 ?Ss 0:00 /usr/sbin/haproxy -D -f /proxy.conf -p /proxy.pid > -sf 139 > 143 ?Ss 0:00 /usr/sbin/haproxy -D -f /proxy.conf -p /proxy.pid > -sf 141 > > It is worth mentioning this is running in a Dockerfile with a go > application as the parent (interlock). Are you absolutely sure pid 141 above is still listening ? (...) > I've tried ensuring that new haproxy instances are only started at most > once every 5 seconds (or even 10 seconds), in an attempt to rule out a race > condition, but no dice. > > Below are pastes of two strace outputs. The first is tracing (just > capturing signal related calls) the parent (pid 1) and all the children, > note that process 284 gets nothing even when PID 285 kills it. Indeed, but what is 284 supposed to be here ? We don't see *any* activity on it in this trace. I'm seeing 283 killing 176, and 285 killing 284. Maybe the parameter passed to 285 is simply wrong and 284 is not even haproxy but the shell script that calls it or your go application ? > Process 1 attached with 9 threads > Process 283 attached (...) > [pid 283] kill(176, SIGUSR1) = 0 > [pid 283] +++ exited with 0 +++ > [pid66] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=283, > si_uid=0, si_status=0, si_utime=0, si_stime=0} --- > [pid66] rt_sigreturn() = 283 > Process 284 attached > Process 285 attached (...) > [pid 285] kill(284, SIGUSR1) = 0 > Process 286 attached > [pid 285] +++ exited with 0 +++ > [pid66] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=285, > si_uid=0, si_status=0, si_utime=0, si_stime=0} --- > [pid66] rt_sigreturn() = 285 (...) > In the > second trace, im capturing everything, note that the process gets the > SIGUSR1 signal, but continues on anyway. Hopefully these shed some light on > what is occurring? > > epoll_wait(0, 2191e30, 200, 1000) = -1 EINTR (Interrupted system call) > --- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_USER, si_pid=1120, si_uid=0} --- > rt_sigaction(SIGUSR1, {0x46bb90, [USR1], SA_RESTORER|SA_RESTART, > 0x7f437988c180}, {0x46bb90, [USR1], SA_RESTORER|SA_RESTART, > 0x7f437988c180}, 8) = 0 > rt_sigreturn() = -1 EINTR (Interrupted system call) > rt_sigprocmask(SIG_SETMASK, ~[PROF RTMIN RT_1], [], 8) = 0 > brk(0x21d9000) = 0x21d9000 > rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 > epoll_ctl(0, EPOLL_CTL_DEL, 4, 6be6e0) = 0 > close(4)= 0 > epoll_ctl(0, EPOLL_CTL_DEL, 5, 6be6e0) = 0 > close(5)= 0 > epoll_ctl(0, EPOLL_CTL_DEL, 6, 6be6e0) = 0 > close(6)= 0 As you can see it has immediately closed these 3 listening sockets (FDs 4, 5 and 6) and you don't see any accept anymore, just regular activity on already established connections, so the behaviour here is totally normal and expected. > epoll_wait(0, {}, 200, 1000)= 0 > epoll_wait(0, {}, 200, 1000)= 0 > epoll_wait(0, {{EPOLLIN, {u32=2, u64=2}}}, 200, 386) = 1 > recvfrom(2, > "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., 16384, 0, > NULL, NULL) = 44 > sendto(1, "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., > 44, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 44 > epoll_wait(0, {{EPOLLIN, {u32=9, u64=9}}}, 200, 40) = 1 > recvfrom(9, > "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., 16384, 0, > NULL, NULL) = 44 > sendto(8, "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., > 44, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 44 > epoll_wait(0, {{EPOLLIN, {u32=7, u64=7}}}, 200, 32) = 1 > recvfrom(7, > "\301*\252V*\311/\310LV\262R\312HR\322QJI,IT\2622415\2664507"..., 16384, 0, > NULL, NULL) = 44 (...) > > Hopefully these traces tell someone something, i'm at a complete loss as to > how this is occurring / what else to try to debug this. I suspect that you might have caught a wrong trace of what you thought was a bug while trying to debug this issue. It is important to find out why above you see pid 285 send SIGUSR1 to pid 284 while there's absolutely no activity on pid 284, which doesn't appear to be haproxy at first glance. Please double-check how the pid list is built before passing it to -sf, I suspect you'll discover something there. Regards, Willy
Re: Stick-table peers expiration time
On Wed, Feb 03, 2016 at 11:05:03AM +1100, Igor Cicimov wrote: > Any comments on this? Shouldn't the expiration time get replicated upon > restart or just the keys? Obviously the entry in the first server stick > table will expire much sooner than the one on the restarted server meaning > the first one might choose different backend for the same session id. I could be wrong (need to check with Emeric who started documenting the peers protocol) but from what I remember, the expiration times are *not* passed over the protocol, and each peer creates the entry with its local expiration timer. The reason is simple, you push updates when you see them, so these ones are supposed to be fresh when you receive them. However maybe an exception should indeed be made during the first learning phase so that timers could be passed along with contents (and bounded by the table's expiration value). I don't know if that would be possible without changing the protocol again though :-/ Willy
Re: http-ignore-probes produces a warning in tcp frontend
Hello Dmitry, On Thu, Jan 28, 2016 at 05:31:58PM +0300, Dmitry Sivachenko wrote: > Hello, > > I have an option http-ignore-probes in defaults section. > When I declare frontend in "tcp" mode, I get the following warning: > > [WARNING] 027/172718 (18281) : config : 'option http-ignore-probes' ignored > for frontend 'MYTEST-front' as it requires HTTP mode. > > In defaults section I have other http-specific options (e.g. > http-keep-alive), which does not produce a warning in tcp backend. > Is it intended? It looks logical to produce such a warning only if > http-specific option is used directly in tcp backend and silently ignore it > when used in defaults. There's no difference between having the option in defaults or explicitly in the section itself. You should see defaults as templates for next sections. The error here is that http-keep-alive should also produce a warning. But I think I know why it doesn't, most options are handled by a generic parser which checks the proxy mode, and a few other more complex ones are implemented "by hand" and do not necessarily run such checks. It's a very bad practise to mix TCP and HTTP proxies with the same defaults sections. This probably is something we should document better in the doc. A good practise is to have one (or several) defaults sections for HTTP mode and then other defaults sections for TCP mode. And most often you don't even have the same timeouts, log settings etc. Regards, Willy
Re: Force client IP with PROXY protocol
On Thu, Jan 28, 2016 at 12:25:05PM +0100, Aleksandar Lazic wrote: > > > Am 28-01-2016 12:01, schrieb Jonathan Leroy - Inikup: > >2016-01-28 11:47 GMT+01:00 Lukas Tribus : > >>Doesn't: > >>http-request set-src hdr(CF-Connecting-IP) > >> > >>in combination with a standard proxy-protocol config > >>already do that? > > > >Yes, but it doesn't work with SPDY or HTTP/2 backends. > > But then it is missleading that this only is possible whith mode http > not also for mode tcp. > > To ask a clear question. > > Do set the 'http-request set-src hdr(CF-Connecting-IP)' the IP in the > proxy protocol and in the tcp packet also in tcp mode? No, set-src replaces the client's src as logged by haproxy and as passed over the proxy protocol. The only issue is that this action was incompletely implemented, it's only in http-request while it should also have been in tcp-request. I hoped that we'd get it completed before the release but apparently nobody was interested in finishing was was begun :-( If someone is willing to do it for TCP mode and the patch is small enough, I'm willing to backport it into 1.6 as I consider it almost a bug to only be able to use it in HTTP mode. With that said, Jonathan, you need to keep in mind that by doing so you will pass the IP address presented by CF in the *first* request as the source of the whole connection, hence all subsequent requests. So before doing this you need to be absolutely sure that CF doesn't multiplex incoming connections from various clients over the same connection. Maybe this can be configured on cloudflare, I have no idea. And quite frankly, just for a performance reason you should definitely make nginx aware of the original client's IP address *per request* and not *per connection* in order to allow CF to multiplex multiple requests over a single connection. Willy
Re: haproxy reloads, stale listeners, SIGKILL required
Hi David, On Tue, Feb 02, 2016 at 11:56:25PM +, David Birdsong wrote: > Has nobody else run into this w/ consul? Given the plethora of tools around > consul and haproxy and templating, I know others are using reloads to keep > backend current, but the old haproxy PIDs stick around listening w/ > incorrect backends. No and this behaviour is really weird. Are you sure the list of pids passed after "-sf" is *always* correct ? Since you're running on a kernel 3.13, it supports SO_REUSEPORT, so this means that it supports multiple processes being bound to the same port. Thus, if sometimes you pass the wrong pid or an incomplete list, it will not prevent the new process from starting but the old process will stay there listening. You should probably check using "ps" that remaining processes really appear in the -sf line on the new process. There's one similar case I've seen a long time ago, a customer was using "-sf $(cat /var/run/haproxy.pid)" in the startup script, and was running two independant instances accidently using the same pid file. Thus one instance would overwrite the other one's pid list and the reload would fail half of the times because the old pids would not be properly signaled. I don't know if that helps, Willy
Re: [PATCH] DOC: remove old tunnel mode assumptions
Hi Lukas, On Wed, Feb 03, 2016 at 07:47:37PM +0100, Lukas Tribus wrote: > Hi Willy, > > > >> This patch fixes a few of those inconsistencies and should be backported > >> to both 1.6 and 1.5. > > > > Applied to 1.7-dev, thanks! > > thanks. > > A minor cosmetic glitch: seems that something broke non-ascii characters, > I can see it in this commit here and also in Cyril's latest patch, while > Cyril's penultimate patch (b65e033 from December) is fine: > > git log --author cyril.bo...@free.fr -2 Yes, I'm seeing some non-printable unicode chars from time to time, I don't know how they display on each one's terminal. If I enable utf-8 encoding in my xterm, I'm seeing them fine. But anyway I don't care that much as long as it's only once in a while, it depends a lot on various things including e-mail encoding, heuristics, hand-edited patches, etc. Thanks, Willy
RE: [PATCH] DOC: remove old tunnel mode assumptions
Hi Willy, >> This patch fixes a few of those inconsistencies and should be backported >> to both 1.6 and 1.5. > > Applied to 1.7-dev, thanks! thanks. A minor cosmetic glitch: seems that something broke non-ascii characters, I can see it in this commit here and also in Cyril's latest patch, while Cyril's penultimate patch (b65e033 from December) is fine: git log --author cyril.bo...@free.fr -2 cheers, lukas
Re: [PATCH] DOC: remove old tunnel mode assumptions
Hi Lukas, On Wed, Feb 03, 2016 at 06:09:37PM +0100, Lukas Tribus wrote: > Micha?? Pasierb reported doc inconsistencies regarding the old default > HTTP tunnel mode. > > This patch fixes a few of those inconsistencies and should be backported > to both 1.6 and 1.5. Applied to 1.7-dev, thanks! Willy
Re: haproxy reloads, stale listeners, SIGKILL required
❦ 3 février 2016 14:37 GMT, David Birdsong : > right, thanks, but again, I'm familiar w/ haproxy and graceful > reloads. I've used lsof -Pnp , and I find that sometimes an old > haproxy process is still listening for new connections. This is the > problem I'm trying to ask if anyone else is also seeing. My bad, I didn't read the whole thread before answering. -- Use statement labels that mean something. - The Elements of Programming Style (Kernighan & Plauger)
[PATCH] DOC: remove old tunnel mode assumptions
Michał Pasierb reported doc inconsistencies regarding the old default HTTP tunnel mode. This patch fixes a few of those inconsistencies and should be backported to both 1.6 and 1.5. --- doc/configuration.txt | 28 1 file changed, 12 insertions(+), 16 deletions(-) diff --git a/doc/configuration.txt b/doc/configuration.txt index 5040f76..7dd5744 100644 --- a/doc/configuration.txt +++ b/doc/configuration.txt @@ -2620,11 +2620,11 @@ cookie [ rewrite | insert | prefix ] [ indirect ] [ nocache ] of complex combinations of "Set-cookie" and "Cache-control" headers is left to the application. The application can then decide whether or not it is appropriate to emit a persistence - cookie. Since all responses should be monitored, this mode only - works in HTTP close mode. Unless the application behaviour is - very complex and/or broken, it is advised not to start with this - mode for new deployments. This keyword is incompatible with - "insert" and "prefix". + cookie. Since all responses should be monitored, this mode + doesn't work in HTTP tunnel mode. Unless the application + behaviour is very complex and/or broken, it is advised not to + start with this mode for new deployments. This keyword is + incompatible with "insert" and "prefix". insertThis keyword indicates that the persistence cookie will have to be inserted by haproxy in server responses if the client did not @@ -2650,7 +2650,7 @@ cookie [ rewrite | insert | prefix ] [ indirect ] [ nocache ] and a delimiter. The prefix will be removed from all client requests so that the server still finds the cookie it emitted. Since all requests and responses are subject to being modified, - this mode requires the HTTP close mode. The "prefix" keyword is + this mode doesn't work with tunnel mode. The "prefix" keyword is not compatible with "rewrite" and "insert". Note: it is highly recommended not to use "indirect" with "prefix", otherwise server cookie updates would not be sent to clients. @@ -5347,11 +5347,9 @@ no option http-use-proxy-header By setting this option in a frontend, haproxy can automatically switch to use that non-standard header if it sees proxied requests. A proxied request is - defined here as one where the URI begins with neither a '/' nor a '*'. The - choice of header only affects requests passing through proxies making use of - one of the "httpclose", "forceclose" and "http-server-close" options. Note - that this option can only be specified in a frontend and will affect the - request along its whole life. + defined here as one where the URI begins with neither a '/' nor a '*'. This + is incompatible with the HTTP tunnel mode. Note that this option can only be + specified in a frontend and will affect the request along its whole life. Also, when this option is set, a request which requires authentication will automatically switch to use proxy authentication headers if it is itself a @@ -5500,10 +5498,8 @@ no option http_proxy No host address resolution is performed, so this only works when pure IP addresses are passed. Since this option's usage perimeter is rather limited, - it will probably be used only by experts who know they need exactly it. Last, - if the clients are susceptible of sending keep-alive requests, it will be - needed to add "option httpclose" to ensure that all requests will correctly - be analyzed. + it will probably be used only by experts who know they need exactly it. This + is incompatible with the HTTP tunnel mode. If this option has been enabled in a "defaults" section, it can be disabled in a specific instance by prepending the "no" keyword before it. @@ -14773,7 +14769,7 @@ Most common cases : spent accepting these connections will inevitably slightly delay processing of other connections, and it can happen that request times in the order of a few tens of milliseconds are measured after a few thousands of new -connections have been accepted at once. Setting "option http-server-close" +connections have been accepted at once. Using one of the keep-alive modes may display larger request times since "Tq" also measures the time spent waiting for additional requests. -- 1.9.1
RE: cookie prefix & option httpclose
> Now I did the same setup with version 1.5 and found out that even > without "option httpclose" both requests have the cookie prefixed. > > My dillema - are the docs inconsistent and this section should be > removed or am I missing something ? Docs are inconsistens both 1.5 and 1.6 for this. Thanks for the report, will fix. Lukas
cookie prefix & option httpclose
Hi, I have a question about cookie prefix mode. I use configuration like this: backend b1 balance roundrobin cookie JSESSIONID prefix server s1 192.168.122.1:8080 cookie c1 Now the docs https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4-cookie for prefix say: Since all requests and responses are subject to being modified, this mode requires the HTTP close mode. Behavior for version 1.4 is consistent with docs. The cookie was not prefixed for second request within one http keepalive session so I needed to add "option httpclose" to modify all requests. Now I did the same setup with version 1.5 and found out that even without "option httpclose" both requests have the cookie prefixed. My dillema - are the docs inconsistent and this section should be removed or am I missing something ? Thanks, Michal
Re: haproxy reloads, stale listeners, SIGKILL required
On Wed, Feb 3, 2016 at 3:05 AM Vincent Bernat wrote: > ❦ 3 février 2016 00:11 GMT, David Birdsong : > > > I'm not using consul but am using haproxy in a docker container > > and reloading when backend hosts change registrations. I haven't > > seen this issue. I run using haproxy-systemd-wrapper and HUP that > > process to reload. > > > > does that mean the wrapper ensures that an old process exits and > > forces if it doesn't eventually? > > No, the wrapper is only here to make HAProxy appears more like a > traditional daemon on the signal handling front. The old processes are > expected to stay to handle existing connections. Once you locate an old > process, check why it is here with lsof: lsof -n -p . You should > see that the process is not listening anymore and is only handling old > connections. If you kill the process, the connections will just break. > right, thanks, but again, I'm familiar w/ haproxy and graceful reloads. I've used lsof -Pnp , and I find that sometimes an old haproxy process is still listening for new connections. This is the problem I'm trying to ask if anyone else is also seeing. > -- > Follow each decision as closely as possible with its associated action. > - The Elements of Programming Style (Kernighan & Plauger) > >
Re: haproxy reloads, stale listeners, SIGKILL required
❦ 3 février 2016 00:11 GMT, David Birdsong : > I'm not using consul but am using haproxy in a docker container > and reloading when backend hosts change registrations. I haven't > seen this issue. I run using haproxy-systemd-wrapper and HUP that > process to reload. > > does that mean the wrapper ensures that an old process exits and > forces if it doesn't eventually? No, the wrapper is only here to make HAProxy appears more like a traditional daemon on the signal handling front. The old processes are expected to stay to handle existing connections. Once you locate an old process, check why it is here with lsof: lsof -n -p . You should see that the process is not listening anymore and is only handling old connections. If you kill the process, the connections will just break. -- Follow each decision as closely as possible with its associated action. - The Elements of Programming Style (Kernighan & Plauger)