date:20170112

On Thu, Jan 12, 2017 at 05:42:35PM +, Carl Pettersson (BN) wrote:
> The interesting thing here is that we could look at the external volume of
> data (it passes through Akamai, without any caching, which lets us see how
> many connections etc they handle before passing it on to us), and it did not
> increase very much at all. So we're a bit interesting in how a 2-5% increase
> of traffic volume could cause this drastic effect. But the suggestion would
> be that it just pushed us over the edge where suddenly the backend did not
> return fast enough and causing a buildup?

It's possible. Also, *some* SSL handshakes can really be slow so you
don't need a huge increase. Here if it's 2-5% only when you were running
around 450/s it doesn't seem much, though it's possible that some attack
traffic has replaced the valid traffic, given that valid clients will
often give up (and on the opposite some will loop faster). It's also
possible that the limit you reached simply is your server's ability to
deliver 2048-bit RSA keys. And 450/s, if running on openssl 1.0.1,
could translate into a 1.8 to 2 GHz CPU if my memory is correct, which
seems realistic.

By the way on your graph we see that the number of in-use TCP connection
immediately jumped and that the CPU usage started when the sockets were
around 8k. 8k looks exactly like the total of what your SSL and clear
frontends can accept + what can be forwarded to the server (2k per frontend
and per side). The CPU usage shows a hole before the peak. This could
indicate that the frontends limits were reached. But then why the CPU
spike ? To me that's a mystery at the moment. Or it could be explained
by first saturating the HTTP frontend (ie less valid traffic can enter,
drop in CPU usage), then switching to the HTTPS one, causing a lot of
CPU to be burnt doing handshakes.

> > OK. Did the situation quickly recover when you stopped it ? Is it what you 
> > did around 00:30 on the graph where everything immediately stopped ? If so 
> > it looks very much like SSL handshakes given how quick the load stopped 
> > after the network!
> 
> Yes, we pushed the stop command 00:28, and by 00:30 the incoming traffic was
> entirely gone. As soon as those stopped, the cpu dropped to 4-5% and
> throughput for the other traffic returned to normal. 

So at least it was not a busy loop bug or something like we have experienced
a long time ago.

Best regards,
Willy

SV: SV: Sudden peak of orphaned sockets -> 100% cpu

2017-01-12 Thread Carl Pettersson (BN)

> 
> On Thu, Jan 12, 2017 at 05:15:41PM +, Carl Pettersson (BN) wrote:
> > > > So, some questions:
> > > > 1. Does it seem reasonable that the orphaned socket could cause this
> > > >behaviour, or are they just a symptom?
> > > 
> > > No, orphans have no such effect and your number was so low that it cannot 
> > > be related. Usually you see them in tends to hundreds of thousands. In 
> > > your case the two events were clearly unrelated when you compare the log 
> > > dates (27 days apart).
> > 
> > Ah, this is due to poor information on my part. The "Too many orphaned 
> > sockets" log was taken after the reboot I mentioned, while the "SYN flood"
> > message was just before the reboot. So in reality they are just ~5 
> > minutes apart (the dangers of relative timestamps).
> 
> OK.
> 
> > Here is a graph showing sockets in use/orphaned and cpu usage:
> > https://cdn.pbrd.co/images/lfCUluWzj.png. We can see that the number 
> > of sockets in use sharply increases at 21:30, and the cpu load follows 
> > just a short while afterwards. The orphaned sockets fluctuate a bit, 
> > but from what you say, those are probably not relevant.
> 
> Indeed, and as you can see they always represent a fraction of the active 
> connections, which experience peaks as well during this high CPU period, very 
> likely indicating short floods.

The interesting thing here is that we could look at the external volume of data 
(it passes through Akamai, without any caching, which lets us see how many 
connections etc they handle before passing it on to us), and it did not 
increase very much at all. So we're a bit interesting in how a 2-5% increase of 
traffic volume could cause this drastic effect. But the suggestion would be 
that it just pushed us over the edge where suddenly the backend did not return 
fast enough and causing a buildup?

> 
> > > > 2. What causes the orphaned sockets? Could haproxy start 
> > > > misbehaving when it is starved for resources?
> > > 
> > > No, simply a connection which was closed and for which the client has not 
> > > yet ACKed the final data. It's a perfectly normal case. However if you 
> > > see the message often, it means your number of orphans is too low, and 
> > > you can increase it in /proc/sys/net/ipv4/tcp_max_orphans.
> > > 
> > 
> > Are there any guidelines on scaling this number? As I understand it, 
> > the reason that this is limited in the first place is to avoid some 
> > simple attacks, but those may be set for different cases than a public 
> > facing load balancer?
> 
> It's never easy but you mostly need it when haproxy has to close the 
> connection (ie it forwards some contents whose size is unknown, or the client 
> explicitly requires connection: close). A good rule of thumb is to consider 
> that you don't want to keep your orphans more than a few seconds (to let the 
> traffic leave). Add to this about 3 seconds to allow a retransmit in case of 
> lost packets. 10s is reasonable to start with, and you may increase it if you 
> experience issues with some clients having very poor connectivity. Now you 
> multiply this 10s by your connection rate (in connections per second) and it 
> will give you the number. If you have
> 500 conn/s and expect at most 1000, that will be around 1. On highly 
> loaded servers, I've already had to push it to about 1 million. But it's 
> important to be careful because these connexions still carry some data and 
> you can eat a lot of memory due to them.
> 
> > > If the issue was only SSL handshakes, extra cores might have helped, till 
> > > the point the attacker was able to reach your capacity. If it was 
> > > something else, there's little point adding extra cores if the machine is 
> > > never loaded.
> > > 
> > > By the way, how did the problem stop, and how long after it started ?
> > 
> > We had the luxury of being able to stop the problem by disabling the 
> > incoming calls. The majority of the inbound traffic on this system at 
> > the moment is diagnostic data, which we can live without for a few 
> > days if it affects the stability of the platform as a whole (We did 
> > want to re-enable it today to see if the problems would resume, but 
> > due to business reasons the risk of destabilizing could not be taken until 
> > later this night).
> 
> OK. Did the situation quickly recover when you stopped it ? Is it what you 
> did around 00:30 on the graph where everything immediately stopped ? If so it 
> looks very much like SSL handshakes given how quick the load stopped after 
> the network!

Yes, we pushed the stop command 00:28, and by 00:30 the incoming traffic was 
entirely gone. As soon as those stopped, the cpu dropped to 4-5% and throughput 
for the other traffic returned to normal. 

> 
> > > I'm not seeing anything wrong in it. One thing however, the default 
> > > maxconn for a frontend is 2000. If you have 400 conn/s, it means 
> > > that you will saturate this once this connections last 5 seconds or 
> >

Re: SV: Sudden peak of orphaned sockets -> 100% cpu

On Thu, Jan 12, 2017 at 05:15:41PM +, Carl Pettersson (BN) wrote:
> > > So, some questions:
> > > 1. Does it seem reasonable that the orphaned socket could cause this
> > >behaviour, or are they just a symptom?
> > 
> > No, orphans have no such effect and your number was so low that it cannot 
> > be related. Usually you see them in tends to hundreds of thousands. In your 
> > case the two events were clearly unrelated when you compare the log dates 
> > (27 days apart).
> 
> Ah, this is due to poor information on my part. The "Too many orphaned
> sockets" log was taken after the reboot I mentioned, while the "SYN flood"
> message was just before the reboot. So in reality they are just ~5 minutes
> apart (the dangers of relative timestamps). 

OK.

> Here is a graph showing sockets in use/orphaned and cpu usage:
> https://cdn.pbrd.co/images/lfCUluWzj.png. We can see that the number of
> sockets in use sharply increases at 21:30, and the cpu load follows just a
> short while afterwards. The orphaned sockets fluctuate a bit, but from what
> you say, those are probably not relevant.

Indeed, and as you can see they always represent a fraction of the active
connections, which experience peaks as well during this high CPU period,
very likely indicating short floods.

> > > 2. What causes the orphaned sockets? Could haproxy start misbehaving 
> > > when it is starved for resources?
> > 
> > No, simply a connection which was closed and for which the client has not 
> > yet ACKed the final data. It's a perfectly normal case. However if you see 
> > the message often, it means your number of orphans is too low, and you can 
> > increase it in /proc/sys/net/ipv4/tcp_max_orphans.
> > 
> 
> Are there any guidelines on scaling this number? As I understand it, the
> reason that this is limited in the first place is to avoid some simple
> attacks, but those may be set for different cases than a public facing load
> balancer?

It's never easy but you mostly need it when haproxy has to close the
connection (ie it forwards some contents whose size is unknown, or the
client explicitly requires connection: close). A good rule of thumb is
to consider that you don't want to keep your orphans more than a few
seconds (to let the traffic leave). Add to this about 3 seconds to allow
a retransmit in case of lost packets. 10s is reasonable to start with,
and you may increase it if you experience issues with some clients having
very poor connectivity. Now you multiply this 10s by your connection rate
(in connections per second) and it will give you the number. If you have
500 conn/s and expect at most 1000, that will be around 1. On highly
loaded servers, I've already had to push it to about 1 million. But it's
important to be careful because these connexions still carry some data
and you can eat a lot of memory due to them.

> > If the issue was only SSL handshakes, extra cores might have helped, till 
> > the point the attacker was able to reach your capacity. If it was something 
> > else, there's little point adding extra cores if the machine is never 
> > loaded.
> > 
> > By the way, how did the problem stop, and how long after it started ?
> 
> We had the luxury of being able to stop the problem by disabling the incoming
> calls. The majority of the inbound traffic on this system at the moment is
> diagnostic data, which we can live without for a few days if it affects the
> stability of the platform as a whole (We did want to re-enable it today to
> see if the problems would resume, but due to business reasons the risk of
> destabilizing could not be taken until later this night). 

OK. Did the situation quickly recover when you stopped it ? Is it what you
did around 00:30 on the graph where everything immediately stopped ? If so
it looks very much like SSL handshakes given how quick the load stopped
after the network!

> > I'm not seeing anything wrong in it. One thing however, the default maxconn 
> > for a frontend is 2000. If you have 400 conn/s, it means that you will 
> > saturate this once this connections last 5 seconds or more. It's possible 
> > you had a hiccup on the server side slowing down connections an making you 
> > reach the limit on the front. You should increase this (in the defaults 
> > section for
> > example) as well as the global.maxconn value.
> 
> Yes, maxconn is one of the settings we realized was much too low as part of
> the investigations. We've seen quite a spectrum of advice on the internet on
> what values to use - from "somewhere around 4096" to "just put it to 55
> million"... We will also be scaling up to 2 or 4 cores which I guess means
> we'll have to also look into setting nbproc and possibly cpu mappings?

Sure. But here it's the same as above for orphans, use you connection rate
as a base and multiply it by the average session time. At least here you can
estimate the memory impact (roughly 34kB per connection with buffers in use,
or 2kB with idle connections but logs enable

SV: Sudden peak of orphaned sockets -> 100% cpu

2017-01-12 Thread Carl Pettersson (BN)

Hi Willy,
Thanks for having a look at this!

> On Thu, Jan 12, 2017 at 11:34:44AM +, Carl Pettersson (BN) wrote:
> > Hi,
> > We have a haproxy setup consisting of a pair of nodes with keepalived, 
> > which then utilize the proxy protocol to pass requests (roundrobin) to 
> > a second pair of haproxy nodes. The first pair mainly terminates SSL 
> > and serves as a highly available entrypoint, while second pair does 
> > all the logic around routing to the correct application.
> > Yesterday at 21:30:35 CET the active node of the first pair suddenly 
> > accumulated thousands of orphaned sockets. Historically there have 
> > been around 300 orphans at any time on this machine. A few seconds 
> > later, the cpu shot up to 100%, up from 30%. At this point, most 
> > requests started timing out. At the moments just before, we were 
> > handling 400-450 req/s. The second pair saw no increase of load during 
> > these problems.
> > As far as we can tell so far (investigations are ongoing), there was 
> > no change anywhere in the environment for several hours preceding this 
> > sudden activity. We are lucky enough that a large part of the traffic 
> > coming in is diagnostic data, which we can live without for a while, 
> > and when we shut down that application, the situation returned to 
> > normal. Before doing that, we tried both failing over to the other 
> > node, as well as both restarting haproxy and a full reboot of the 
> > node. Neither of these helped, the situation returned in less than a minute.
> > 
> > We had a lot of "kernel: [ 1342.944691] TCP: too many orphaned sockets" in 
> > our logs, as well as a few of these:
> > kernel: [2352025.865855] TCP: request_sock_TCP: Possible SYN flooding on 
> > port 443. Sending cookies.  Check SNMP counters.
> > kernel: [2352068.014861] TCP: request_sock_TCP: Possible SYN flooding on 
> > port 80. Sending cookies.  Check SNMP counters.
> > 
> > According to the Arbor our hosting provider has, there was no SYN 
> > attack, however. The servers are fronted by Akamai, which by default 
> > does two retries on connection failures or timeouts, so this may have 
> > amplified our problems.
> > This feature will be turned off.
> > 
> > So, some questions:
> > 1. Does it seem reasonable that the orphaned socket could cause this
> >behaviour, or are they just a symptom?
> 
> No, orphans have no such effect and your number was so low that it cannot be 
> related. Usually you see them in tends to hundreds of thousands. In your case 
> the two events were clearly unrelated when you compare the log dates (27 days 
> apart).

Ah, this is due to poor information on my part. The "Too many orphaned sockets" 
log was taken after the reboot I mentioned, while the "SYN flood" message was 
just before the reboot. So in reality they are just ~5 minutes apart (the 
dangers of relative timestamps). 
Here is a graph showing sockets in use/orphaned and cpu usage: 
https://cdn.pbrd.co/images/lfCUluWzj.png. We can see that the number of sockets 
in use sharply increases at 21:30, and the cpu load follows just a short while 
afterwards. The orphaned sockets fluctuate a bit, but from what you say, those 
are probably not relevant.

> 
> 
> > 2. What causes the orphaned sockets? Could haproxy start misbehaving 
> > when it is starved for resources?
> 
> No, simply a connection which was closed and for which the client has not yet 
> ACKed the final data. It's a perfectly normal case. However if you see the 
> message often, it means your number of orphans is too low, and you can 
> increase it in /proc/sys/net/ipv4/tcp_max_orphans.
> 

Are there any guidelines on scaling this number? As I understand it, the reason 
that this is limited in the first place is to avoid some simple attacks, but 
those may be set for different cases than a public facing load balancer?

> > 3. We were speculating that it could somehow be related to keepalives 
> > not being terminated properly, any merit to that thought?
> 
> No, I think based on your log mentionning a SYN flood that in fact you might 
> have got a "massive" number of SSL handshakes causing the CPU saturation for 
> a while and the impossibility to process incoming connections for a short 
> time. Once the somaxconn limit is reached, the system is forced to drop new 
> SYNs and will log that it was under something looking like a SYN flood. And 
> depending on the sizing of your machine, your permitted ciphers, your cert 
> size etc, the cost of a handshake can be high enough for a moderate 
> connection rate to do this while passing under the radar.
> 
> > The nodes are virtual machines running Centos 7 and haproxy 1.6.7, 
> > single core with 2 GB memory. As I mentioned, they have been handling 
> > this load without any hiccups for several months, but we are still 
> > considering increasing the specs. Would a few more cores have helped, 
> > or would it just have taken a few more seconds to chew up?
> 
> The danger wiht virtual machin

Re: Sudden peak of orphaned sockets -> 100% cpu

Hi Carl,

On Thu, Jan 12, 2017 at 11:34:44AM +, Carl Pettersson (BN) wrote:
> Hi,
> We have a haproxy setup consisting of a pair of nodes with keepalived, which
> then utilize the proxy protocol to pass requests (roundrobin) to a second
> pair of haproxy nodes. The first pair mainly terminates SSL and serves as a
> highly available entrypoint, while second pair does all the logic around
> routing to the correct application.
> Yesterday at 21:30:35 CET the active node of the first pair suddenly
> accumulated thousands of orphaned sockets. Historically there have been
> around 300 orphans at any time on this machine. A few seconds later, the cpu
> shot up to 100%, up from 30%. At this point, most requests started timing
> out. At the moments just before, we were handling 400-450 req/s. The second
> pair saw no increase of load during these problems.
> As far as we can tell so far (investigations are ongoing), there was no
> change anywhere in the environment for several hours preceding this sudden
> activity. We are lucky enough that a large part of the traffic coming in is
> diagnostic data, which we can live without for a while, and when we shut down
> that application, the situation returned to normal. Before doing that, we
> tried both failing over to the other node, as well as both restarting haproxy
> and a full reboot of the node. Neither of these helped, the situation
> returned in less than a minute.
> 
> We had a lot of "kernel: [ 1342.944691] TCP: too many orphaned sockets" in 
> our logs, as well as a few of these:
> kernel: [2352025.865855] TCP: request_sock_TCP: Possible SYN flooding on port 
> 443. Sending cookies.  Check SNMP counters.
> kernel: [2352068.014861] TCP: request_sock_TCP: Possible SYN flooding on port 
> 80. Sending cookies.  Check SNMP counters.
> 
> According to the Arbor our hosting provider has, there was no SYN attack,
> however. The servers are fronted by Akamai, which by default does two retries
> on connection failures or timeouts, so this may have amplified our problems.
> This feature will be turned off.
> 
> So, some questions:
> 1. Does it seem reasonable that the orphaned socket could cause this
>behaviour, or are they just a symptom?

No, orphans have no such effect and your number was so low that it cannot
be related. Usually you see them in tends to hundreds of thousands. In your
case the two events were clearly unrelated when you compare the log dates
(27 days apart).

> 2. What causes the orphaned sockets? Could haproxy start misbehaving when it
> is starved for resources?

No, simply a connection which was closed and for which the client has not
yet ACKed the final data. It's a perfectly normal case. However if you see
the message often, it means your number of orphans is too low, and you can
increase it in /proc/sys/net/ipv4/tcp_max_orphans.

> 3. We were speculating that it could somehow be related to keepalives not
> being terminated properly, any merit to that thought?

No, I think based on your log mentionning a SYN flood that in fact you
might have got a "massive" number of SSL handshakes causing the CPU
saturation for a while and the impossibility to process incoming
connections for a short time. Once the somaxconn limit is reached,
the system is forced to drop new SYNs and will log that it was under
something looking like a SYN flood. And depending on the sizing of
your machine, your permitted ciphers, your cert size etc, the cost of
a handshake can be high enough for a moderate connection rate to do
this while passing under the radar.

> The nodes are virtual machines running Centos 7 and haproxy 1.6.7, single
> core with 2 GB memory. As I mentioned, they have been handling this load
> without any hiccups for several months, but we are still considering
> increasing the specs. Would a few more cores have helped, or would it just
> have taken a few more seconds to chew up?

The danger wiht virtual machines is that their CPU can be stolen by other
machines or by the hypervisor without them noticing, resulting in any
pending traffic to be suddenly delivered as a burst and making this problem
possibly worse. Are you sure you don't have memory ballooning, sub-CPU
allocation nor any such horrors ?

If the issue was only SSL handshakes, extra cores might have helped, till
the point the attacker was able to reach your capacity. If it was something
else, there's little point adding extra cores if the machine is never loaded.

By the way, how did the problem stop, and how long after it started ?

> Below is the haproxy configuration.

(...)

I'm not seeing anything wrong in it. One thing however, the default maxconn
for a frontend is 2000. If you have 400 conn/s, it means that you will saturate
this once this connections last 5 seconds or more. It's possible you had a
hiccup on the server side slowing down connections an making you reach the
limit on the front. You should increase this (in the defaults section for
example) as well as the global.maxconn v

Re: [PATCH] MINOR: systemd unit works with cfgdir and cfgfile

On Thu, Jan 12, 2017 at 01:28:06PM +, Patrick Hemmer wrote:
> This change is rather dangerous. It's not unlikely that people will have
> multiple config files in their `/etc/haproxy` directory. Such might
> happen if users keep backups of previous versions when they make a
> change, or if they have alternate configs for different purposes.

Confirmed, that's exactly my case, and I know places where there are
configs suffixed with the date and trigram of the people having touched
them. We must definitely not do what is proposed here!

> In
> general when applications support loading multiple config files, the
> configs are typically put in a `.d` directory, such as
> `/etc/haproxy/conf.d`.

Yes and people are used to be careful about "conf.d" directories.

Thanks,
Willy

Re: [PATCH] BUG/MINOR: stream: Fix how backend-specific analyzers are set, on a stream

Christopher,

I suspect that below you spotted a deeper bug which probably also affects
older versions :

On Thu, Jan 12, 2017 at 04:45:44PM +0100, Christopher Faulet wrote:
> >From 73b8871a5e31004ec305a3eb2cd4747c4f569d5e Mon Sep 17 00:00:00 2001
> From: Christopher Faulet 
> Date: Mon, 9 Jan 2017 16:33:19 +0100
> Subject: [PATCH] BUG/MINOR: stream: Fix how backend-specific analyzers are set
>  on a stream
> X-Bogosity: Ham, tests=bogofilter, spamicity=0.00, version=1.2.4
> 
> When the stream's backend was defined, the request's analyzers flag was always
> set to 0 if the stream had no listener. This bug was introduced with the 
> filter
> API but never triggered (I think so).
> 
> Because of the commit 5820a366, it is now possible to encountered it. For
> example, this happens when the trace filter is enabled on a SPOE backend. The
> fix is pretty trivial.
> 
> This fix must be backported to 1.7.
> ---
>  src/proxy.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/proxy.c b/src/proxy.c
> index 2d9f06c..d85f6a9 100644
> --- a/src/proxy.c
> +++ b/src/proxy.c
> @@ -1156,7 +1156,7 @@ int stream_set_backend(struct stream *s, struct proxy 
> *be)
>* be more reliable to store the list of analysers that have been run,
>* but what we do here is OK for now.
>*/
> - s->req.analysers |= be->be_req_ana & (strm_li(s) ? 
> ~strm_li(s)->analysers : 0);
> + s->req.analysers |=  (strm_li(s) ? be->be_req_ana & 
> ~strm_li(s)->analysers : be->be_req_ana);

In fact another way to write it is :

s->req.analysers |= be->be_req_ana & (strm_li(s) ? 
~strm_li(s)->analysers : ~0);

Which is equivalent to the more readable :

s->req.analysers |= be->be_req_ana & ~(strm_li(s) ? 
strm_li(s)->analysers : 0);

and I strongly suspect it was the initial intent otherwise the way the
expression is written makes no sense. I'll go review older versions to
see which ones are possibly affected. BTW I'd rather use the latest form
above in the fix if that's fine for you as well.

Thanks,
Willy

[PATCH] BUG/MINOR: stream: Fix how backend-specific analyzers are set, on a stream

2017-01-12 Thread Christopher Faulet



--
Christopher
>From 73b8871a5e31004ec305a3eb2cd4747c4f569d5e Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Mon, 9 Jan 2017 16:33:19 +0100
Subject: [PATCH] BUG/MINOR: stream: Fix how backend-specific analyzers are set
 on a stream
X-Bogosity: Ham, tests=bogofilter, spamicity=0.00, version=1.2.4

When the stream's backend was defined, the request's analyzers flag was always
set to 0 if the stream had no listener. This bug was introduced with the filter
API but never triggered (I think so).

Because of the commit 5820a366, it is now possible to encountered it. For
example, this happens when the trace filter is enabled on a SPOE backend. The
fix is pretty trivial.

This fix must be backported to 1.7.
---
 src/proxy.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/proxy.c b/src/proxy.c
index 2d9f06c..d85f6a9 100644
--- a/src/proxy.c
+++ b/src/proxy.c
@@ -1156,7 +1156,7 @@ int stream_set_backend(struct stream *s, struct proxy *be)
 	 * be more reliable to store the list of analysers that have been run,
 	 * but what we do here is OK for now.
 	 */
-	s->req.analysers |= be->be_req_ana & (strm_li(s) ? ~strm_li(s)->analysers : 0);
+	s->req.analysers |=  (strm_li(s) ? be->be_req_ana & ~strm_li(s)->analysers : be->be_req_ana);
 
 	/* If the target backend requires HTTP processing, we have to allocate
 	 * the HTTP transaction and hdr_idx if we did not have one.
-- 
2.9.3

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

2017-01-12 Thread Simon Horman

On Thu, Jan 12, 2017 at 03:14:27PM +, Scott McKeown wrote:
> Doh, that would do it.
> Sorry wrong git branch.
> 
> # haproxy -v
> HA-Proxy version 1.8-dev0 2016/11/25
> Copyright 2000-2016 Willy Tarreau 
> 
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> [
> {
> "field": {
> "name": "Name",
> "pos": 0
> },
> "processNum": 1,
> "tags": {
> "nature": "Output",
> "origin": "Product",
> "scope": "Service"
> },
> "value": {
> "type": "str",
> "value": "HAProxy"
> }
> },
> {
> "field": {
> "name": "Version",
> "pos": 1
> },
> "processNum": 1,
> "tags": {
> "nature": "Output",
> "origin": "Product",
> "scope": "Service"
> },
> "value": {
> 
> etc. etc. etc.

Thanks, that looks like what I would expect.

-- 
Simon Horman  si...@horms.nl
Horms Solutions BV  www.horms.nl
Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
Tel: +31 (0)20 800 6155Skype: horms7

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

Doh, that would do it.
Sorry wrong git branch.

# haproxy -v
HA-Proxy version 1.8-dev0 2016/11/25
Copyright 2000-2016 Willy Tarreau 

# echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
json.tool
[
{
"field": {
"name": "Name",
"pos": 0
},
"processNum": 1,
"tags": {
"nature": "Output",
"origin": "Product",
"scope": "Service"
},
"value": {
"type": "str",
"value": "HAProxy"
}
},
{
"field": {
"name": "Version",
"pos": 1
},
"processNum": 1,
"tags": {
"nature": "Output",
"origin": "Product",
"scope": "Service"
},
"value": {

etc. etc. etc.



On 12 January 2017 at 13:34, Simon Horman  wrote:

> On Thu, Jan 12, 2017 at 01:27:37PM +, Scott McKeown wrote:
> > Hi Simon,
> >
> > Output below:
>
> ...
>
> Thanks
>
> > On 12 January 2017 at 13:23, Simon Horman  wrote:
> >
> > > Hi Scott,
> > >
> > > could you send the output of the following?
> > >
> > > echo "show info json" | socat /tmp/haproxy.stat stdio
> > >
> > > On Thu, Jan 12, 2017 at 01:18:54PM +, Scott McKeown wrote:
> > > > Hi Guys,
> > > > Sorry for the delay I got tied up with some other issues yesterday
> but
> > > I've
> > > > just finished with Simons git repo pull.
> > > > Simon are you sure this is correct as I thought this was for a
> 1.8-dev
> > > > build which could be why I had problems on Tuesday.
> > > >
> > > > # haproxy -v
> > > > HA-Proxy version 1.6-dev1 2015/03/11
> > > > Copyright 2000-2015 Willy Tarreau 
>
> ...
>
> The above seems wrong. I see:
>
> ./haproxy -v
> HA-Proxy version 1.8-dev0-b738a8-135 2017/01/04
> Copyright 2000-2016 Willy Tarreau 
>
> Could you verify that the head commit you have checked out is as follows?
>
> git log --oneline -1
> b738a8b596d8 MEDIUM: stats: Add show json schema
>



-- 
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)

Re: HAproxy / Reverse proxy Debian

Title: Re: HAproxy / Reverse proxy Debian


Bonjour Daniel,

From my first post, you can see the config with ssl.
To switch to TCP mode, I have removed:

- All ciphers
- In defaults, I have switch from "mode http" to "mode tcp"
- In frontend email-https, I have remove "reqadd X-Forwarded-Proto:\ https" and "ssl crt /etc/ssl/private/full_certs.crt"
- In backend email-https I have removed "redirect scheme https if !{ ssl_fc }"

new config:

frontend email-https
        bind *:888
        default_backend https-email

backend https-email
        server zobe ip_email_server:888

But when testing, I have now this message: this page cannot be displayed ...

logs from HAproxy:

Jan 12 15:46:47 proxy haproxy[17514]: ip_number:52188 [12/Jan/2017:15:45:57.032] email-https https-email/zobe 1/0/50293 6325 cD 0/0/0/0/0 0/0
Jan 12 15:59:33 proxy haproxy[17514]: ip_number:13246 [12/Jan/2017:15:58:43.626] email-https https-email/zobe 1/0/50275 5807 cD 0/0/0/0/0 0/0




Le jeudi 12 janvier 2017 à 15:35:21, vous écriviez :









This email server do have ssl/TLS activated.



As I expected. Apparently that iRedMail server uses nginx. 
Right now, if you talk to haproxy, it decrypts the traffic and then sends it on to nginx in plain text. However, on that port nginx expects encrypted traffic — hence your 400 error message.

If you want to configure TLS on the mail server / web server itself, there is no need to configure haproxy for TLS at all. 
Switch it to TCP mode and remove the TLS configuration. That way it will just hand the still encrypted traffic over to nginx.




-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH                  | Hochstraße 11
                                   | 42697 Solingen
tel: +49 1754155711                | Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431






On 12. Jan. 2017, at 14:30, Thierry  wrote:

Re: HAproxy / Reverse proxy Debian 
Bonjour Daniel,

I am not sure to understand.
I am using iRedMail as email server.
This email server do have ssl/TLS activated.

**

listen 888 http2;    
   ssl on;
   ssl_certificate /etc/ssl/certs/cert.chained.crt;
   ssl_certificate_key /etc/ssl/private/cert.key;
   ssl_trusted_certificate /etc/ssl/certs/GandiStandardSSLCA2.pem;
   ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
   include /etc/nginx/sslciphers.conf;
   add_header Strict-Transport-Security "max-age=15768000";
   ssl_prefer_server_ciphers on;
   ssl_dhparam /etc/ssl/dhparams.pem;
   ssl_stapling on;
   ssl_stapling_verify on;
   resolver 8.8.8.8 8.8.4.4 valid=300s;
   resolver_timeout 10s;

*

My email client do work well with these certificates and if I change the NAT of my router, I can reach the email web interface (Sogo) through HTTPS request.
Why is not possible to pass HTTPS trafic from the HAproxy to my email server ? Will be the same pb with my web server ..

Thx


Le jeudi 12 janvier 2017 à 15:16:57, vous écriviez :





Sounds as if you have nginx set up for TLS termination, too.
This does not make sense, because haproxy will already have decrypted the traffic.
Make sure nginx does not expect https on what in your config would be ip_email_server:888.



-- 
Daniel Schneller
Principal Cloud Engineer

CenterDevice GmbH                  | Hochstraße 11
                                  | 42697 Solingen
tel: +49 1754155711                | Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431






On 12. Jan. 2017, at 14:14, Thierry  wrote:

Re: HAproxy / Reverse proxy Debian 
Bonjour Daniel,

I have resolved my problem, HAproxy do start now (ssl ok).
But when trying to reach my email server, I now do have a:

400 Bad gateway - The plain HTTP request was sent to HTTPS port - Nginx

It should not be the case because 'reqadd x-forwarded-proto:\ https' suppose to correct this ?? And with 'redirect scheme https if !{ ssl_fc }' it should be 100% full HTTPS.

frontend email-https
      bind *:444 ssl crt /etc/ssl/private/full_certs.crt
      reqadd X-Forwarded-Proto:\ https
      default_backend https-email

backend https-email
      redirect scheme https if !{ ssl_fc }
      server email_hostname ip_email_server:888

Thx





Le jeudi 12 janvier 2017 à 14:44:19, vous écriviez :





Re-adding the list.

And:





Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??



Yes. Though I am not sure what file.crt and file.pem are :)










Cheers,
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer

CenterDevice GmbH                  | Hochstraße 11
                                 | 42697 Solingen
tel: +49 1754155711                | Deutschland
daniel.schnel...@centerdevice.de

Re: HAproxy / Reverse proxy Debian

> This email server do have ssl/TLS activated.

As I expected. Apparently that iRedMail server uses nginx. 
Right now, if you talk to haproxy, it decrypts the traffic and then sends it on 
to nginx in plain text. However, on that port nginx expects encrypted traffic — 
hence your 400 error message.

If you want to configure TLS on the mail server / web server itself, there is 
no need to configure haproxy for TLS at all. 
Switch it to TCP mode and remove the TLS configuration. That way it will just 
hand the still encrypted traffic over to nginx.




-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431


> On 12. Jan. 2017, at 14:30, Thierry  wrote:
> 
> Bonjour Daniel,
> 
> I am not sure to understand.
> I am using iRedMail as email server.
> This email server do have ssl/TLS activated.
> 
> **
> 
> listen 888 http2;
>ssl on;
>ssl_certificate /etc/ssl/certs/cert.chained.crt;
>ssl_certificate_key /etc/ssl/private/cert.key;
>ssl_trusted_certificate /etc/ssl/certs/GandiStandardSSLCA2.pem;
>ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
>include /etc/nginx/sslciphers.conf;
>add_header Strict-Transport-Security "max-age=15768000";
>ssl_prefer_server_ciphers on;
>ssl_dhparam /etc/ssl/dhparams.pem;
>ssl_stapling on;
>ssl_stapling_verify on;
>resolver 8.8.8.8 8.8.4.4 valid=300s;
>resolver_timeout 10s;
> 
> *
> 
> My email client do work well with these certificates and if I change the NAT 
> of my router, I can reach the email web interface (Sogo) through HTTPS 
> request.
> Why is not possible to pass HTTPS trafic from the HAproxy to my email server 
> ? Will be the same pb with my web server ..
> 
> Thx
> 
> 
> Le jeudi 12 janvier 2017 à 15:16:57, vous écriviez :
> 
> 
> Sounds as if you have nginx set up for TLS termination, too.
> This does not make sense, because haproxy will already have decrypted the 
> traffic.
> Make sure nginx does not expect https on what in your config would be 
> ip_email_server:888.
> 
> 
> 
> -- 
> Daniel Schneller
> Principal Cloud Engineer
> 
> CenterDevice GmbH  | Hochstraße 11
>   | 42697 Solingen
> tel: +49 1754155711| Deutschland
> daniel.schnel...@centerdevice.de    
> | www.centerdevice.de 
> 
> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
> Michael Rosbach, Handelsregister-Nr.: HRB 18655,
> HR-Gericht: Bonn, USt-IdNr.: DE-815299431
> 
> 
> 
> On 12. Jan. 2017, at 14:14, Thierry  > wrote:
> 
> Re: HAproxy / Reverse proxy Debian 
> Bonjour Daniel,
> 
> I have resolved my problem, HAproxy do start now (ssl ok).
> But when trying to reach my email server, I now do have a:
> 
> 400 Bad gateway - The plain HTTP request was sent to HTTPS port - Nginx
> 
> It should not be the case because 'reqadd x-forwarded-proto:\ https' suppose 
> to correct this ?? And with 'redirect scheme https if !{ ssl_fc }' it should 
> be 100% full HTTPS.
> 
> frontend email-https
>   bind *:444 ssl crt /etc/ssl/private/full_certs.crt
>   reqadd X-Forwarded-Proto:\ https
>   default_backend https-email
> 
> backend https-email
>   redirect scheme https if !{ ssl_fc }
>   server email_hostname ip_email_server:888
> 
> Thx
> 
> 
> 
> 
> 
> Le jeudi 12 janvier 2017 à 14:44:19, vous écriviez :
> 
> 
> Re-adding the list.
> 
> And:
> 
> 
> Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??
> 
> Yes. Though I am not sure what file.crt and file.pem are :)
> 
> 
> 
> 
> 
> Cheers,
> Daniel
> 
> 
> -- 
> Daniel Schneller
> Principal Cloud Engineer
> 
> CenterDevice GmbH  | Hochstraße 11
>  | 42697 Solingen
> tel: +49 1754155711| Deutschland
> daniel.schnel...@centerdevice.de    
> | www.centerdevice.de 
> 
> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
> Michael Rosbach, Handelsregister-Nr.: HRB 18655,
> HR-Gericht: Bonn, USt-IdNr.: DE-815299431
> 
> 
> 
> On 12. Jan. 2017, at 13:27, Thierry  > wrote:
> 
> Hi,
> 
> You are right, I am using the v1.7.1-1 on Debian.
> I do have paid ssl certificate (.key, .crt, .pem). They all are in non 
> world-readable folder.
> Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??
> 
> Thx
> 
> 
> Thierry,
> 
> 
> 
> always helps to know the haproxy version you use.
> As for your error message, do you have private key, your site’s
> certificate and all necessary chain certific

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

2017-01-12 Thread Simon Horman

On Thu, Jan 12, 2017 at 01:27:37PM +, Scott McKeown wrote:
> Hi Simon,
> 
> Output below:

...

Thanks

> On 12 January 2017 at 13:23, Simon Horman  wrote:
> 
> > Hi Scott,
> >
> > could you send the output of the following?
> >
> > echo "show info json" | socat /tmp/haproxy.stat stdio
> >
> > On Thu, Jan 12, 2017 at 01:18:54PM +, Scott McKeown wrote:
> > > Hi Guys,
> > > Sorry for the delay I got tied up with some other issues yesterday but
> > I've
> > > just finished with Simons git repo pull.
> > > Simon are you sure this is correct as I thought this was for a 1.8-dev
> > > build which could be why I had problems on Tuesday.
> > >
> > > # haproxy -v
> > > HA-Proxy version 1.6-dev1 2015/03/11
> > > Copyright 2000-2015 Willy Tarreau 

...

The above seems wrong. I see:

./haproxy -v
HA-Proxy version 1.8-dev0-b738a8-135 2017/01/04
Copyright 2000-2016 Willy Tarreau 

Could you verify that the head commit you have checked out is as follows?

git log --oneline -1
b738a8b596d8 MEDIUM: stats: Add show json schema

Re: HAproxy / Reverse proxy Debian

Title: Re: HAproxy / Reverse proxy Debian


Bonjour Daniel,

I am not sure to understand.
I am using iRedMail as email server.
This email server do have ssl/TLS activated.

**

listen 888 http2;    
    ssl on;
    ssl_certificate /etc/ssl/certs/cert.chained.crt;
    ssl_certificate_key /etc/ssl/private/cert.key;
    ssl_trusted_certificate /etc/ssl/certs/GandiStandardSSLCA2.pem;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    include /etc/nginx/sslciphers.conf;
    add_header Strict-Transport-Security "max-age=15768000";
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/ssl/dhparams.pem;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 10s;

*

My email client do work well with these certificates and if I change the NAT of my router, I can reach the email web interface (Sogo) through HTTPS request.
Why is not possible to pass HTTPS trafic from the HAproxy to my email server ? Will be the same pb with my web server ..

Thx


Le jeudi 12 janvier 2017 à 15:16:57, vous écriviez :





Sounds as if you have nginx set up for TLS termination, too.
This does not make sense, because haproxy will already have decrypted the traffic.
Make sure nginx does not expect https on what in your config would be ip_email_server:888.



-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH                  | Hochstraße 11
                                   | 42697 Solingen
tel: +49 1754155711                | Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431






On 12. Jan. 2017, at 14:14, Thierry  wrote:

Re: HAproxy / Reverse proxy Debian 
Bonjour Daniel,

I have resolved my problem, HAproxy do start now (ssl ok).
But when trying to reach my email server, I now do have a:

400 Bad gateway - The plain HTTP request was sent to HTTPS port - Nginx

It should not be the case because 'reqadd x-forwarded-proto:\ https' suppose to correct this ?? And with 'redirect scheme https if !{ ssl_fc }' it should be 100% full HTTPS.

frontend email-https
       bind *:444 ssl crt /etc/ssl/private/full_certs.crt
       reqadd X-Forwarded-Proto:\ https
       default_backend https-email

backend https-email
       redirect scheme https if !{ ssl_fc }
       server email_hostname ip_email_server:888

Thx





Le jeudi 12 janvier 2017 à 14:44:19, vous écriviez :





Re-adding the list.

And:





Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??



Yes. Though I am not sure what file.crt and file.pem are :)









Cheers,
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer

CenterDevice GmbH                  | Hochstraße 11
                                  | 42697 Solingen
tel: +49 1754155711                | Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431






On 12. Jan. 2017, at 13:27, Thierry  wrote:

Hi,

You are right, I am using the v1.7.1-1 on Debian.
I do have paid ssl certificate (.key, .crt, .pem). They all are in non world-readable folder.
Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??

Thx





Thierry,








always helps to know the haproxy version you use.
As for your error message, do you have private key, your site’s
certificate and all necessary chain certificates in the crt files you reference in your config?








IIRC they need to be in the order 








1. key
2. site cert (“leaf”)
3. intermediates








Make sure to have these files not world-readable as they contain secret crypto material.








HTH,
Daniel















-- 
Cordialement,
Thierry                            e-mail : lenai...@maelenn.org








-- 
Cordialement,
 Thierry                            e-mail : lenai...@maelenn.org

Re: [PATCH] MINOR: systemd unit works with cfgdir and cfgfile

2017-01-12 Thread Patrick Hemmer



On 2017/1/12 06:42, Ricardo Fraile wrote:
> Hello,
>
>
> As 1.7 release allow to load multiple files from a directory:
>
>
> https://cbonte.github.io/haproxy-dconv/1.7/management.html
>
>  -f  : adds  to the list of configuration files
> to be loaded. If  is a directory, all the files (and only files)
> it contains are added in lexical order (using LC_COLLATE=C) to the list
> of configuration files to be loaded ; only files with ".cfg" extension
> are added, only non hidden files (not prefixed with ".") are added.
>
>
> I think that the systemd unit can have the configuration directory
> instead of the path to the file for allow the same behaviour that the
> "-f" option provides.
>
>
> Thanks in advance,
>
>
>
> Regards,
This change is rather dangerous. It's not unlikely that people will have
multiple config files in their `/etc/haproxy` directory. Such might
happen if users keep backups of previous versions when they make a
change, or if they have alternate configs for different purposes. In
general when applications support loading multiple config files, the
configs are typically put in a `.d` directory, such as
`/etc/haproxy/conf.d`.

-Patrick

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

Hi Simon,

Output below:

# echo "show info json" | socat /tmp/haproxy.stat stdio
Name: HAProxy
Version: 1.6-dev1
Release_date: 2015/03/11
Nbproc: 1
Process_num: 1
Pid: 1611
Uptime: 0d 0h14m04s
Uptime_sec: 844
Memmax_MB: 0
Ulimit-n: 4037
Maxsock: 4037
Maxconn: 2000
Hard_maxconn: 2000
CurrConns: 1
CumConns: 177
CumReq: 1005
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 7
SessRate: 0
SessRateLimit: 0
MaxSessRate: 7
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 9
Run_queue: 1
Idle_pct: 100
node: localhost.localdomain
description:



On 12 January 2017 at 13:23, Simon Horman  wrote:

> Hi Scott,
>
> could you send the output of the following?
>
> echo "show info json" | socat /tmp/haproxy.stat stdio
>
> On Thu, Jan 12, 2017 at 01:18:54PM +, Scott McKeown wrote:
> > Hi Guys,
> > Sorry for the delay I got tied up with some other issues yesterday but
> I've
> > just finished with Simons git repo pull.
> > Simon are you sure this is correct as I thought this was for a 1.8-dev
> > build which could be why I had problems on Tuesday.
> >
> > # haproxy -v
> > HA-Proxy version 1.6-dev1 2015/03/11
> > Copyright 2000-2015 Willy Tarreau 
> >
> > I'm still getting the following though:
> > # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> > json.tool
> > No JSON object could be decoded
> >
> > Its a basic build with the following options:
> > # make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
> >
> > Oh and I'm running with your config file now.
> >
> > Anything else that I can try or detail you would like to make sure its
> not
> > me?
> >
> >
> > ~Scott
> >
> >
> >
> > On 9 January 2017 at 16:13, Scott McKeown 
> wrote:
> >
> > > No problem I'll have another look tomorrow morning and I'll let you all
> > > know how I get on.
> > >
> > >
> > > On 9 January 2017 at 15:18, Simon Horman  wrote:
> > >
> > >> Hi Scott,
> > >>
> > >> thanks for testing.
> > >>
> > >> For reference the code I am using is here:
> > >>
> > >> https://github.com/horms/haproxy.git show-json
> > >>
> > >> And my minimal config file is as follows.
> > >> Would it be possible for you to share you config with me (privately) ?
> > >>
> > >> global
> > >> daemon
> > >> stats socket /tmp/haproxy.stat mode 600 level admin
> > >> pidfile /tmp/haproxy.pid
> > >> log /dev/log local4
> > >> #tune.bufsize 2048
> > >> #tune.bufsize 4096
> > >> tune.bufsize 8192
> > >> #tune.bufsize 16384
> > >> tune.maxrewrite 1024
> > >>
> > >> defaults
> > >> mode http
> > >> balance roundrobin
> > >> timeout connect 4000
> > >> timeout client 42000
> > >> timeout server 43000
> > >> log global
> > >>
> > >>
> > >> listen VIP_Name
> > >> bind 127.0.0.1:10080 transparent
> > >> mode http
> > >> balance leastconn
> > >> cookie SERVERID insert nocache indirect
> > >> server backup 127.0.0.1:9081 backup  non-stick
> > >> option http-keep-alive
> > >> option forwardfor
> > >> option redispatch
> > >> option abortonclose
> > >> maxconn 4
> > >> log global
> > >> option httplog
> > >> option log-health-checks
> > >> server RIP1_Name 127.0.0.1  weight 100  cookie RIP_Name
> > >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter
> 2000 rise
> > >> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions
> disabled
> > >> server RIP2_Name 127.0.0.1  weight 100  cookie RIP_Name
> > >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter
> 2000 rise
> > >> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions
> > >>
> > >
> > >
> > >
> > > --
> > > With Kind Regards.
> > >
> > > Scott McKeown
> > > Loadbalancer.org
> > > http://www.loadbalancer.org
> > > Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> > > Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
> > >
> >
> >
> >
> > --
> > With Kind Regards.
> >
> > Scott McKeown
> > Loadbalancer.org
> > http://www.loadbalancer.org
> > Tel (UK) - +44 (0) 3303801064 (24x7)
> > Tel (US) - +1 888.867.9504 (Toll Free)(24x7)
>
> --
> Simon Horman  si...@horms.nl
> Horms Solutions BV  www.horms.nl
> Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
> Tel: +31 (0)20 800 6155Skype: horms7
>



-- 
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)

How can I change the URI when forwarding to a server

2017-01-12 Thread Jürgen Haas

Hi all,

I wonder if I can change the uri that the server receives without doing
a redirect.

Example:
Request from client: https://www.example.com/login/username?p1=something
Request received by server: /login.php?s=username&p1=something

More general:
- if path begins with /login/*[?*]
- add the first * as a query parameter s to the query
- keep other optional query parameters in place

Is anything like that possible?


Thanks
Jürgen



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

2017-01-12 Thread Simon Horman

Hi Scott,

could you send the output of the following?

echo "show info json" | socat /tmp/haproxy.stat stdio

On Thu, Jan 12, 2017 at 01:18:54PM +, Scott McKeown wrote:
> Hi Guys,
> Sorry for the delay I got tied up with some other issues yesterday but I've
> just finished with Simons git repo pull.
> Simon are you sure this is correct as I thought this was for a 1.8-dev
> build which could be why I had problems on Tuesday.
> 
> # haproxy -v
> HA-Proxy version 1.6-dev1 2015/03/11
> Copyright 2000-2015 Willy Tarreau 
> 
> I'm still getting the following though:
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> No JSON object could be decoded
> 
> Its a basic build with the following options:
> # make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
> 
> Oh and I'm running with your config file now.
> 
> Anything else that I can try or detail you would like to make sure its not
> me?
> 
> 
> ~Scott
> 
> 
> 
> On 9 January 2017 at 16:13, Scott McKeown  wrote:
> 
> > No problem I'll have another look tomorrow morning and I'll let you all
> > know how I get on.
> >
> >
> > On 9 January 2017 at 15:18, Simon Horman  wrote:
> >
> >> Hi Scott,
> >>
> >> thanks for testing.
> >>
> >> For reference the code I am using is here:
> >>
> >> https://github.com/horms/haproxy.git show-json
> >>
> >> And my minimal config file is as follows.
> >> Would it be possible for you to share you config with me (privately) ?
> >>
> >> global
> >> daemon
> >> stats socket /tmp/haproxy.stat mode 600 level admin
> >> pidfile /tmp/haproxy.pid
> >> log /dev/log local4
> >> #tune.bufsize 2048
> >> #tune.bufsize 4096
> >> tune.bufsize 8192
> >> #tune.bufsize 16384
> >> tune.maxrewrite 1024
> >>
> >> defaults
> >> mode http
> >> balance roundrobin
> >> timeout connect 4000
> >> timeout client 42000
> >> timeout server 43000
> >> log global
> >>
> >>
> >> listen VIP_Name
> >> bind 127.0.0.1:10080 transparent
> >> mode http
> >> balance leastconn
> >> cookie SERVERID insert nocache indirect
> >> server backup 127.0.0.1:9081 backup  non-stick
> >> option http-keep-alive
> >> option forwardfor
> >> option redispatch
> >> option abortonclose
> >> maxconn 4
> >> log global
> >> option httplog
> >> option log-health-checks
> >> server RIP1_Name 127.0.0.1  weight 100  cookie RIP_Name
> >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
> >> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
> >> server RIP2_Name 127.0.0.1  weight 100  cookie RIP_Name
> >> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
> >> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions
> >>
> >
> >
> >
> > --
> > With Kind Regards.
> >
> > Scott McKeown
> > Loadbalancer.org
> > http://www.loadbalancer.org
> > Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> > Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
> >
> 
> 
> 
> -- 
> With Kind Regards.
> 
> Scott McKeown
> Loadbalancer.org
> http://www.loadbalancer.org
> Tel (UK) - +44 (0) 3303801064 (24x7)
> Tel (US) - +1 888.867.9504 (Toll Free)(24x7)

-- 
Simon Horman  si...@horms.nl
Horms Solutions BV  www.horms.nl
Parnassusweg 819, 1082 LZ Amsterdam, Netherlands
Tel: +31 (0)20 800 6155Skype: horms7

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

Sorry forgot to show the /tmp/ folder

# ls -latrh /tmp/
total 16K
-rw---.  1 root root0 Jan  9 11:00 yum.log
drwxrwxrwt.  2 root root 4.0K Jan  9 11:21 .ICE-unix
dr-xr-xr-x. 22 root root 4.0K Jan  9 11:21 ..
srw---.  1 root root0 Jan  9 14:39 haproxy.stat
-rw-r--r--.  1 root root5 Jan  9 14:39 haproxy.pid
drwxrwxrwt.  3 root root 4.0K Jan  9 14:44 .




On 12 January 2017 at 13:18, Scott McKeown  wrote:

> Hi Guys,
> Sorry for the delay I got tied up with some other issues yesterday but
> I've just finished with Simons git repo pull.
> Simon are you sure this is correct as I thought this was for a 1.8-dev
> build which could be why I had problems on Tuesday.
>
> # haproxy -v
> HA-Proxy version 1.6-dev1 2015/03/11
> Copyright 2000-2015 Willy Tarreau 
>
> I'm still getting the following though:
> # echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
> json.tool
> No JSON object could be decoded
>
> Its a basic build with the following options:
> # make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1
>
> Oh and I'm running with your config file now.
>
> Anything else that I can try or detail you would like to make sure its not
> me?
>
>
> ~Scott
>
>
>
> On 9 January 2017 at 16:13, Scott McKeown  wrote:
>
>> No problem I'll have another look tomorrow morning and I'll let you all
>> know how I get on.
>>
>>
>> On 9 January 2017 at 15:18, Simon Horman  wrote:
>>
>>> Hi Scott,
>>>
>>> thanks for testing.
>>>
>>> For reference the code I am using is here:
>>>
>>> https://github.com/horms/haproxy.git show-json
>>>
>>> And my minimal config file is as follows.
>>> Would it be possible for you to share you config with me (privately) ?
>>>
>>> global
>>> daemon
>>> stats socket /tmp/haproxy.stat mode 600 level admin
>>> pidfile /tmp/haproxy.pid
>>> log /dev/log local4
>>> #tune.bufsize 2048
>>> #tune.bufsize 4096
>>> tune.bufsize 8192
>>> #tune.bufsize 16384
>>> tune.maxrewrite 1024
>>>
>>> defaults
>>> mode http
>>> balance roundrobin
>>> timeout connect 4000
>>> timeout client 42000
>>> timeout server 43000
>>> log global
>>>
>>>
>>> listen VIP_Name
>>> bind 127.0.0.1:10080 transparent
>>> mode http
>>> balance leastconn
>>> cookie SERVERID insert nocache indirect
>>> server backup 127.0.0.1:9081 backup  non-stick
>>> option http-keep-alive
>>> option forwardfor
>>> option redispatch
>>> option abortonclose
>>> maxconn 4
>>> log global
>>> option httplog
>>> option log-health-checks
>>> server RIP1_Name 127.0.0.1  weight 100  cookie RIP_Name
>>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>>> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
>>> server RIP2_Name 127.0.0.1  weight 100  cookie RIP_Name
>>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>>> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions
>>>
>>
>>
>>
>> --
>> With Kind Regards.
>>
>> Scott McKeown
>> Loadbalancer.org
>> http://www.loadbalancer.org
>> Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
>> Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
>>
>
>
>
> --
> With Kind Regards.
>
> Scott McKeown
> Loadbalancer.org
> http://www.loadbalancer.org
> Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
>



-- 
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)

Re: [PATCH 0/2] MEDIUM: stats: Add JSON output option to show (info|stat)

Hi Guys,
Sorry for the delay I got tied up with some other issues yesterday but I've
just finished with Simons git repo pull.
Simon are you sure this is correct as I thought this was for a 1.8-dev
build which could be why I had problems on Tuesday.

# haproxy -v
HA-Proxy version 1.6-dev1 2015/03/11
Copyright 2000-2015 Willy Tarreau 

I'm still getting the following though:
# echo "show info json" | socat /tmp/haproxy.stat stdio | python -m
json.tool
No JSON object could be decoded

Its a basic build with the following options:
# make TARGET=linux26 USE_STATIC_PCRE=1 USE_LINUX_TPROXY=1

Oh and I'm running with your config file now.

Anything else that I can try or detail you would like to make sure its not
me?


~Scott



On 9 January 2017 at 16:13, Scott McKeown  wrote:

> No problem I'll have another look tomorrow morning and I'll let you all
> know how I get on.
>
>
> On 9 January 2017 at 15:18, Simon Horman  wrote:
>
>> Hi Scott,
>>
>> thanks for testing.
>>
>> For reference the code I am using is here:
>>
>> https://github.com/horms/haproxy.git show-json
>>
>> And my minimal config file is as follows.
>> Would it be possible for you to share you config with me (privately) ?
>>
>> global
>> daemon
>> stats socket /tmp/haproxy.stat mode 600 level admin
>> pidfile /tmp/haproxy.pid
>> log /dev/log local4
>> #tune.bufsize 2048
>> #tune.bufsize 4096
>> tune.bufsize 8192
>> #tune.bufsize 16384
>> tune.maxrewrite 1024
>>
>> defaults
>> mode http
>> balance roundrobin
>> timeout connect 4000
>> timeout client 42000
>> timeout server 43000
>> log global
>>
>>
>> listen VIP_Name
>> bind 127.0.0.1:10080 transparent
>> mode http
>> balance leastconn
>> cookie SERVERID insert nocache indirect
>> server backup 127.0.0.1:9081 backup  non-stick
>> option http-keep-alive
>> option forwardfor
>> option redispatch
>> option abortonclose
>> maxconn 4
>> log global
>> option httplog
>> option log-health-checks
>> server RIP1_Name 127.0.0.1  weight 100  cookie RIP_Name
>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions disabled
>> server RIP2_Name 127.0.0.1  weight 100  cookie RIP_Name
>> agent-check agent-port 12345 agent-inter 2000 check port 80 inter 2000 rise
>> 2 fall 3  minconn 0 maxconn 0s on-marked-down shutdown-sessions
>>
>
>
>
> --
> With Kind Regards.
>
> Scott McKeown
> Loadbalancer.org
> http://www.loadbalancer.org
> Tel (UK) - +44 (0) 3303801064 <0330%20380%201064> (24x7)
> Tel (US) - +1 888.867.9504 <+1%20888-867-9504> (Toll Free)(24x7)
>



-- 
With Kind Regards.

Scott McKeown
Loadbalancer.org
http://www.loadbalancer.org
Tel (UK) - +44 (0) 3303801064 (24x7)
Tel (US) - +1 888.867.9504 (Toll Free)(24x7)

Re: HAproxy / Reverse proxy Debian

Sounds as if you have nginx set up for TLS termination, too.
This does not make sense, because haproxy will already have decrypted the 
traffic.
Make sure nginx does not expect https on what in your config would be 
ip_email_server:888.



-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431


> On 12. Jan. 2017, at 14:14, Thierry  wrote:
> 
> Bonjour Daniel,
> 
> I have resolved my problem, HAproxy do start now (ssl ok).
> But when trying to reach my email server, I now do have a:
> 
> 400 Bad gateway - The plain HTTP request was sent to HTTPS port - Nginx
> 
> It should not be the case because 'reqadd x-forwarded-proto:\ https' suppose 
> to correct this ?? And with 'redirect scheme https if !{ ssl_fc }' it should 
> be 100% full HTTPS.
> 
> frontend email-https
>bind *:444 ssl crt /etc/ssl/private/full_certs.crt
>reqadd X-Forwarded-Proto:\ https
>default_backend https-email
> 
> backend https-email
>redirect scheme https if !{ ssl_fc }
>server email_hostname ip_email_server:888
> 
> Thx
> 
> 
> 
> 
> 
> Le jeudi 12 janvier 2017 à 14:44:19, vous écriviez :
> 
> 
> Re-adding the list.
> 
> And:
> 
> 
> Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??
> 
> Yes. Though I am not sure what file.crt and file.pem are :)
> 
> 
> 
> 
> Cheers,
> Daniel
> 
> 
> -- 
> Daniel Schneller
> Principal Cloud Engineer
> 
> CenterDevice GmbH  | Hochstraße 11
>   | 42697 Solingen
> tel: +49 1754155711| Deutschland
> daniel.schnel...@centerdevice.de    
> | www.centerdevice.de 
> 
> Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
> Michael Rosbach, Handelsregister-Nr.: HRB 18655,
> HR-Gericht: Bonn, USt-IdNr.: DE-815299431
> 
> 
> 
> On 12. Jan. 2017, at 13:27, Thierry  > wrote:
> 
> Hi,
> 
> You are right, I am using the v1.7.1-1 on Debian.
> I do have paid ssl certificate (.key, .crt, .pem). They all are in non 
> world-readable folder.
> Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??
> 
> Thx
> 
> 
> Thierry,
> 
> 
> 
> always helps to know the haproxy version you use.
> As for your error message, do you have private key, your site’s
> certificate and all necessary chain certificates in the crt files you 
> reference in your config?
> 
> 
> 
> IIRC they need to be in the order 
> 
> 
> 
> 1. key
> 2. site cert (“leaf”)
> 3. intermediates
> 
> 
> 
> Make sure to have these files not world-readable as they contain secret 
> crypto material.
> 
> 
> 
> HTH,
> Daniel
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> Cordialement,
> Thierrye-mail : lenai...@maelenn.org 
>

Re: HAproxy / Reverse proxy Debian

Title: Re: HAproxy / Reverse proxy Debian


Bonjour Daniel,

I have resolved my problem, HAproxy do start now (ssl ok).
But when trying to reach my email server, I now do have a:

400 Bad gateway - The plain HTTP request was sent to HTTPS port - Nginx

It should not be the case because 'reqadd x-forwarded-proto:\ https' suppose to correct this ?? And with 'redirect scheme https if !{ ssl_fc }' it should be 100% full HTTPS.

frontend email-https
        bind *:444 ssl crt /etc/ssl/private/full_certs.crt
        reqadd X-Forwarded-Proto:\ https
        default_backend https-email

backend https-email
        redirect scheme https if !{ ssl_fc }
        server email_hostname ip_email_server:888

Thx





Le jeudi 12 janvier 2017 à 14:44:19, vous écriviez :





Re-adding the list.

And:





Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??



Yes. Though I am not sure what file.crt and file.pem are :)








Cheers,
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH                  | Hochstraße 11
                                   | 42697 Solingen
tel: +49 1754155711                | Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431






On 12. Jan. 2017, at 13:27, Thierry  wrote:

Hi,

You are right, I am using the v1.7.1-1 on Debian.
I do have paid ssl certificate (.key, .crt, .pem). They all are in non world-readable folder.
Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??

Thx





Thierry,








always helps to know the haproxy version you use.
As for your error message, do you have private key, your site’s
certificate and all necessary chain certificates in the crt files you reference in your config?








IIRC they need to be in the order 








1. key
2. site cert (“leaf”)
3. intermediates








Make sure to have these files not world-readable as they contain secret crypto material.








HTH,
Daniel
















-- 
Cordialement,
 Thierry                            e-mail : lenai...@maelenn.org

Re: HAproxy / Reverse proxy Debian

Re-adding the list.

And:

> Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??

Yes. Though I am not sure what file.crt and file.pem are :)

Cheers,
Daniel


-- 
Daniel Schneller
Principal Cloud Engineer
 
CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431


> On 12. Jan. 2017, at 13:27, Thierry  wrote:
> 
> Hi,
> 
> You are right, I am using the v1.7.1-1 on Debian.
> I do have paid ssl certificate (.key, .crt, .pem). They all are in non 
> world-readable folder.
> Do I have to "cat file.key file.crt file.pem > certi.chained.crt" ??
> 
> Thx
> 
>> Thierry,
> 
> 
>> always helps to know the haproxy version you use.
>> As for your error message, do you have private key, your site’s
>> certificate and all necessary chain certificates in the crt files you 
>> reference in your config?
> 
> 
>> IIRC they need to be in the order 
> 
> 
>> 1. key
>> 2. site cert (“leaf”)
>> 3. intermediates
> 
> 
>> Make sure to have these files not world-readable as they contain secret 
>> crypto material.
> 
> 
>> HTH,
>> Daniel
> 
> 
> 
> 
> 
>

[PATCH] MINOR: systemd unit works with cfgdir and cfgfile

2017-01-12 Thread Ricardo Fraile

Hello,


As 1.7 release allow to load multiple files from a directory:


https://cbonte.github.io/haproxy-dconv/1.7/management.html

 -f  : adds  to the list of configuration files
to be loaded. If  is a directory, all the files (and only files)
it contains are added in lexical order (using LC_COLLATE=C) to the list
of configuration files to be loaded ; only files with ".cfg" extension
are added, only non hidden files (not prefixed with ".") are added.


I think that the systemd unit can have the configuration directory
instead of the path to the file for allow the same behaviour that the
"-f" option provides.


Thanks in advance,



Regards,
From a4d0ea299144f5f2c5983b1335b8d89241f3c0ec Mon Sep 17 00:00:00 2001
From: Ricardo Fraile 
Date: Thu, 12 Jan 2017 12:29:44 +0100
Subject: [PATCH] MINOR: systemd unit works with cfgdir and cfgfile

---
 contrib/systemd/haproxy.service.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/systemd/haproxy.service.in b/contrib/systemd/haproxy.service.in
index dca81a2..1986083 100644
--- a/contrib/systemd/haproxy.service.in
+++ b/contrib/systemd/haproxy.service.in
@@ -3,7 +3,7 @@ Description=HAProxy Load Balancer
 After=network.target
 
 [Service]
-Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
+Environment="CONFIG=/etc/haproxy/" "PIDFILE=/run/haproxy.pid"
 ExecStartPre=@SBINDIR@/haproxy -f $CONFIG -c -q
 ExecStart=@SBINDIR@/haproxy-systemd-wrapper -f $CONFIG -p $PIDFILE
 ExecReload=@SBINDIR@/haproxy -f $CONFIG -c -q
-- 
2.1.4

Sudden peak of orphaned sockets -> 100% cpu

2017-01-12 Thread Carl Pettersson (BN)

Hi,
We have a haproxy setup consisting of a pair of nodes with keepalived, which
then utilize the proxy protocol to pass requests (roundrobin) to a second pair
of haproxy nodes. The first pair mainly terminates SSL and serves as a highly
available entrypoint, while second pair does all the logic around routing to
the correct application.
Yesterday at 21:30:35 CET the active node of the first pair suddenly
accumulated thousands of orphaned sockets. Historically there have been around
300 orphans at any time on this machine. A few seconds later, the cpu shot up
to 100%, up from 30%. At this point, most requests started timing out. At the
moments just before, we were handling 400-450 req/s. The second pair saw no
increase of load during these problems.
As far as we can tell so far (investigations are ongoing), there was no change
anywhere in the environment for several hours preceding this sudden activity.
We are lucky enough that a large part of the traffic coming in is diagnostic
data, which we can live without for a while, and when we shut down that
application, the situation returned to normal. Before doing that, we tried both
failing over to the other node, as well as both restarting haproxy and a full
reboot of the node. Neither of these helped, the situation returned in less
than a minute.

We had a lot of "kernel: [ 1342.944691] TCP: too many orphaned sockets" in our
logs, as well as a few of these:
kernel: [2352025.865855] TCP: request_sock_TCP: Possible SYN flooding on port
443. Sending cookies. Check SNMP counters.
kernel: [2352068.014861] TCP: request_sock_TCP: Possible SYN flooding on port
80. Sending cookies. Check SNMP counters.

According to the Arbor our hosting provider has, there was no SYN attack,
however. The servers are fronted by Akamai, which by default does two retries
on connection failures or timeouts, so this may have amplified our problems.
This feature will be turned off.

So, some questions:
1. Does it seem reasonable that the orphaned socket could cause this behaviour,
or are they just a symptom?
2. What causes the orphaned sockets? Could haproxy start misbehaving when it is
starved for resources?
3. We were speculating that it could somehow be related to keepalives not being
terminated properly, any merit to that thought?

The nodes are virtual machines running Centos 7 and haproxy 1.6.7, single core
with 2 GB memory. As I mentioned, they have been handling this load without any
hiccups for several months, but we are still considering increasing the specs.
Would a few more cores have helped, or would it just have taken a few more
seconds to chew up?

Below is the haproxy configuration.

Thankful for any insights! Best regards,
Carl Pettersson

global
log 127.0.0.1 local0
maxconn 4000
log-tag haproxy-gateway
server-state-file /var/lib/haproxy/gateway.state
stats socket /var/run/haproxy-gateway.sock
stats timeout 2m
user haproxy
group haproxy

tune.ssl.default-dh-param 2048
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
tune.ssl.cachesize 10
tune.ssl.lifetime 600
tune.ssl.maxrecord 1460

defaults
log global
mode http
option httplog
option dontlognull
load-server-state-from-file global
option http-server-close
option forwardfor
# Redispatch if backend is down
option redispatch
retries 3
# timeouts
timeout http-request 5s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
# Set X-Request-Id on all incoming requests. Magic format taken from docs
unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
unique-id-header X-Request-ID
# Set log format to the same as default, adding the request id on the end
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\
%CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r\ %ID

listen http
bind 10.0.1.21:80
default_backend backend_pair
capture request header Host len 64
capture request header X-Forwarded-For len 64
capture request header User-Agent len 200
capture request header True-Client-IP len 32
listen https
bind 10.0.1.21:443 ssl crt /etc/haproxy/ssl/gateway/
http-request set-header X-Forwarded-Proto https
default_backend backend_pair
capture request header Host len 64
capture request header X-Forwarded-For len 64
capture request header

Re: HAProxy Lua Map.end & reserved keywords

2017-01-12 Thread Robin H. Johnson

On Wed, Jan 11, 2017 at 12:17:26PM +0100, Willy Tarreau wrote:
> On Mon, Jan 09, 2017 at 08:47:17PM +, Robin H. Johnson wrote:
> > Maybe Willy would considering changing the name of the matches to 'prefix'
> > & 'suffix' instead of 'beg' & 'end', and just keep beg/end as legacy.
> Another approach would be to have simple rules when mapping a namespace
> to another one. For example we could easily imagine that all "map_something"
> in haproxy are mapped to their exact equivalent name in Lua (thus we would
> have "map_end" instead of "map.end"), or that we could prefix all these
> words by "_" which would give "map._end" which is not much different from
> the haproxy keyword "map_end" and easy enough to remember.
This is actually why I thought of Map.match_str, with a potential idea
that at some point it MIGHT become a bitfield for additional behaviors
(eg the output types per
https://cbonte.github.io/haproxy-dconv/1.6/configuration.html#7.3.1-map).

The underscore is not sufficiently unique I think, but my knowledge of
Lua is nowhere near as extensive as I'd like.

-- 
Robin Hugh Johnson
E-Mail : robb...@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#   : 30269588 or 41961639
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: HAproxy / Reverse proxy Debian

Thierry,

always helps to know the haproxy version you use.
As for your error message, do you have private key, your site’s certificate and 
all necessary chain certificates in the crt files you reference in your config?

IIRC they need to be in the order 

1. key
2. site cert (“leaf”)
3. intermediates

Make sure to have these files not world-readable as they contain secret crypto 
material.

HTH,
Daniel

-- 
Daniel Schneller
Principal Cloud Engineer

CenterDevice GmbH  | Hochstraße 11
   | 42697 Solingen
tel: +49 1754155711| Deutschland
daniel.schnel...@centerdevice.de   | www.centerdevice.de

Geschäftsführung: Dr. Patrick Peschlow, Dr. Lukas Pustina,
Michael Rosbach, Handelsregister-Nr.: HRB 18655,
HR-Gericht: Bonn, USt-IdNr.: DE-815299431

> On 12. Jan. 2017, at 09:23, Thierry  wrote:
> 
> no SSL certificate specified for bind '*:888' at 
> [/etc/haproxy/haproxy.cfg:52] (use 'crt')

HAproxy / Reverse proxy Debian