haproxy 1.5-dev18 + patch corruption happened!!
Hi all, In /var/log/messages, as below haproxy corruption log was detected.. May 8 14:12:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x012333e0 *** May 8 14:12:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x012333e0 *** May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x014b83e0 *** May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x014b83e0 *** May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x014b83e0 *** May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x014b83e0 *** May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x00e4e3e0 *** May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x00e4e3e0 *** May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x00e4e3e0 *** May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy: double free or corruption (fasttop): 0x00e4e3e0 *** My haproxy information as below HA-Proxy version 1.5-dev18 2013/04/29 Copyright 2000-2013 Willy Tarreau Build options : TARGET = linux2628 CPU = native CC = gcc CFLAGS = -O2 -march=native -g -fno-strict-aliasing OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.7 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.32 2012-11-30 PCRE library supports JIT : yes Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. In haproxy soft reload, corruption happened. Without USE_PCRE_JIT=1, this was not happend!! Thanks in advance. - seri
Add X-Forwarded-For
Hi, I want to move some websites behind cloudfare. They already add an X-Forwarded-For header so I do not want to add it if the request comes from cloudfare, but I do want to add it if the request is not from cloudfare. Since both requests will pass through the same frontend I need some kind of ACL or whatever. Is there a way to do this? Greets, Sander
Re: Add X-Forwarded-For
Replying to myself ;-) On 08.05.2013 10:52, Sander Klein wrote: Hi, I want to move some websites behind cloudfare. They already add an X-Forwarded-For header so I do not want to add it if the request comes from cloudfare, but I do want to add it if the request is not from cloudfare. Since both requests will pass through the same frontend I need some kind of ACL or whatever. Is there a way to do this? I know I can use 'option forwardfor except [network]' but cloudfare uses a lot of networks. Greets, Sander
Re: haproxy 1.5-dev18 + patch corruption happened!!
Hi,
On Wed, May 08, 2013 at 05:05:56PM +0900, ?? wrote:
> Hi all,
>
> In /var/log/messages, as below haproxy corruption log was detected..
>
> May 8 14:12:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x012333e0 ***
> May 8 14:12:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x012333e0 ***
> May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x014b83e0 ***
> May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x014b83e0 ***
> May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x014b83e0 ***
> May 8 14:14:02 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x014b83e0 ***
> May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x00e4e3e0 ***
> May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x00e4e3e0 ***
> May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x00e4e3e0 ***
> May 8 14:28:38 X haproxy: *** glibc detected *** /usr/sbin/haproxy:
> double free or corruption (fasttop): 0x00e4e3e0 ***
>
> My haproxy information as below
>
> HA-Proxy version 1.5-dev18 2013/04/29
> Copyright 2000-2013 Willy Tarreau
> Build options :
> TARGET = linux2628
> CPU = native
> CC = gcc
> CFLAGS = -O2 -march=native -g -fno-strict-aliasing
> OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
> Default settings :
> maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
> Encrypted password support via crypt(3): yes
> Built with zlib version : 1.2.7
> Compression algorithms supported : identity, deflate, gzip
> Built with OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
> Running on OpenSSL version : OpenSSL 1.0.1e 11 Feb 2013
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports prefer-server-ciphers : yes
> Built with PCRE version : 8.32 2012-11-30
> PCRE library supports JIT : yes
> Available polling systems :
> epoll : pref=300, test result OK
>poll : pref=200, test result OK
> select : pref=150, test result OK
> Total: 3 (3 usable), will use epoll.
>
>
>
> In haproxy soft reload, corruption happened.
> Without USE_PCRE_JIT=1, this was not happend!!
OK if it's just during a reload, it's completely harmless as it was
just during the memory deallocation procedure that this happened (the
400 totally useless lines of code that make valgrind happy).
It's interesting that it only happens with JIT, I suspect it's just
because of the free(preg) that is done in regex_free(), which is
followed by another free() in free_pattern(). Could you please try
with the following patch and confirm it fixes the issue for you ?
diff --git a/include/common/regex.h b/include/common/regex.h
index bab1a55..0104019 100644
--- a/include/common/regex.h
+++ b/include/common/regex.h
@@ -79,7 +79,6 @@ static inline void regex_free(regex *preg) {
#ifdef USE_PCRE_JIT
pcre_free_study(preg->extra);
pcre_free(preg->reg);
- free(preg);
#else
regfree(preg);
#endif
Thanks,
Willy
RE: Add X-Forwarded-For
> I know I can use 'option forwardfor except [network]' but cloudfare > uses a lot of networks. Exactly, we would need to trigger forwardfor based on a ACL match, which doesn't seem to be supported currently. Regards, Lukas
Re: Add X-Forwarded-For
On Wed, May 08, 2013 at 10:52:29AM +0200, Sander Klein wrote: > Hi, > > I want to move some websites behind cloudfare. They already add an > X-Forwarded-For header so I do not want to add it if the request comes > from cloudfare, but I do want to add it if the request is not from > cloudfare. > > Since both requests will pass through the same frontend I need some > kind of ACL or whatever. > > Is there a way to do this? You have the optional argument "if-none" for "option forwardfor", but you should not do this with external proxies whose addresses you don't know because anyone could pass one and fool you. In practice you would need them to pass you some information to prove the request comes from them. The best way to do this is to do it over ssl. Cheers, Willy
Re: Add X-Forwarded-For
Hey, You have the optional argument "if-none" for "option forwardfor", but you should not do this with external proxies whose addresses you don't know because anyone could pass one and fool you. This doesnt feel like a good option ;-) In practice you would need them to pass you some information to prove the request comes from them. The best way to do this is to do it over ssl. Well, I know which networks they are using since the provide them on their website. That might be prove enough I didn't test if it's possible to do 'option forwardfor except 192.168.1.0/24 192.168.2.0/24 etc...' Even better would be to load it from a file. Maybe the option from Finn Arne Gangstad might prove good enough for me and I can fix it with some reqidel statements. Greets, Sander
RE: Add X-Forwarded-For
You could also: - always insert/append forwardfor and remove the cloudflare ips in the application code This has the disadvantage that you need to modify the application code. Or another way: - duplicate your backend, one for "direct-mode" and one for cloudflare: select it based on a ACL (which you can feed with the cloudflare ips). - configure "option forwardfor" only on the direct-mode backend and remove it from default/frontend/global sections This has the disadvantage that by duplicating the backend, per server settings like maxconn need to be configured more carefully. Lukas
Re: Add X-Forwarded-For
On 08.05.2013 12:21, Sander Klein wrote: Hey, You have the optional argument "if-none" for "option forwardfor", but you should not do this with external proxies whose addresses you don't know because anyone could pass one and fool you. This doesnt feel like a good option ;-) In practice you would need them to pass you some information to prove the request comes from them. The best way to do this is to do it over ssl. Well, I know which networks they are using since the provide them on their website. That might be prove enough I didn't test if it's possible to do 'option forwardfor except 192.168.1.0/24 192.168.2.0/24 etc...' Even better would be to load it from a file. Maybe the option from Finn Arne Gangstad might prove good enough for me and I can fix it with some reqidel statements. I just found out that they also send an CF-Connecting-IP header. Is there a way to copy the contents of this header to the X-Forwarded-For header? Regards, Sander
Re: Add X-Forwarded-For
On Wed, May 08, 2013 at 12:51:10PM +0200, Sander Klein wrote: > On 08.05.2013 12:21, Sander Klein wrote: > >Hey, > > > >>You have the optional argument "if-none" for "option forwardfor", > >>but you should not do this with external proxies whose addresses > >>you don't know because anyone could pass one and fool you. > > > >This doesnt feel like a good option ;-) > >>In practice you would need them to pass you some information to > >>prove the request comes from them. The best way to do this is to > >>do it over ssl. > > > >Well, I know which networks they are using since the provide them on > >their website. That might be prove enough > > > >I didn't test if it's possible to do 'option forwardfor except > >192.168.1.0/24 192.168.2.0/24 etc...' > > > >Even better would be to load it from a file. > > > >Maybe the option from Finn Arne Gangstad might prove good enough for > >me and I can fix it with some reqidel statements. > > I just found out that they also send an CF-Connecting-IP header. Is > there a way to copy the contents of this header to the X-Forwarded-For > header? Yes, just remove x-forwarded-for and rename cf-connecting-ip to x-forwarded-for :-) Willy
RE: Add X-Forwarded-For
> > I just found out that they also send an CF-Connecting-IP header. Is > > there a way to copy the contents of this header to the X-Forwarded-For > > header? > > Yes, just remove x-forwarded-for and rename cf-connecting-ip to > x-forwarded-for :-) > > Willy But remember that cf-connecting-ip can be spoofed as easily as x-forwarded-for. You will need to check the cloudflare ips somehow and you can do this with with the 2 proposals from my previous mail. Regards, Lukas
Re: Add X-Forwarded-For
The definitive list of cloudflare IPs doesn't appear to be too unmanageable: https://www.cloudflare.com/ips They also provide convenient text files that just contain the IP address lists for easy automation. As Lukas says if you do not validate the IP addresses it's trivial for anyone to forge client IP addresses. -JohnF On Wed, May 8, 2013 at 8:26 AM, Lukas Tribus wrote: > > > I just found out that they also send an CF-Connecting-IP header. Is > > > there a way to copy the contents of this header to the X-Forwarded-For > > > header? > > > > Yes, just remove x-forwarded-for and rename cf-connecting-ip to > > x-forwarded-for :-) > > > > Willy > > > But remember that cf-connecting-ip can be spoofed as easily as > x-forwarded-for. > > You will need to check the cloudflare ips somehow and you can do this with > with the 2 proposals from my previous mail. > > > Regards, > Lukas >
Re: Add X-Forwarded-For
On Wed, May 08, 2013 at 08:29:15AM -0400, John Marrett wrote: > The definitive list of cloudflare IPs doesn't appear to be too unmanageable: > > https://www.cloudflare.com/ips > > They also provide convenient text files that just contain the IP address > lists for easy automation. > > As Lukas says if you do not validate the IP addresses it's trivial for > anyone to forge client IP addresses. I agree, and indeed the list is very small, I thought it was much larger, as akamai's which are much harder to deal with. I think the following method should work, though I have not tested it : acl from_cf src -f cf-ips.txt # list of cf's addresses, one per line reqidel ^x-forwarded-for: if !from_cf option forwardfor if-none It is supposed to remove xff from requests not coming from CF, and to add one only when there is none, which should do the trick. Willy
RE: documentation for stats webinterface
Hi! > Whats the explicit difference between > Disable/Enable/SoftStop/SoftStart/KillSessions functions? Disable/Enable matches the "disable/enable server x" on the unix socket, see documentation at [1] and [2]. SoftStop set the servers weight to zero [3]. > Is the way over the webinterface the "proceed right way" to disable > a web-server in background example for maintenance? *Usually* something like this: - *soft stop* (sets weigth to zero, so no *new* sessions come up) - wait for existing session to finish - *disable* the backend server for maintenance - do the actual maintenance - *enable* the server again - and *soft start* the server again (<-- do not forget this one) Regards, Lukas [1] http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#9-disable%20server [2] http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#9-enable%20server [3] http://cbonte.github.io/haproxy-dconv/configuration-1.4.html#9-set%20weight
Re: haproxy 1.5-dev18 + patch corruption happened!!
Hi, Willy
> diff --git a/include/common/regex.h b/include/common/regex.h
> index bab1a55..0104019 100644
> --- a/include/common/regex.h
> +++ b/include/common/regex.h
> @@ -79,7 +79,6 @@ static inline void regex_free(regex *preg) {
> #ifdef USE_PCRE_JIT
> pcre_free_study(preg->extra);
> pcre_free(preg->reg);
> -free(preg);
> #else
> regfree(preg);
> #endif
I've tested after applying this patch.
It works very well without corrupting.
Thanks,
Seri
Re: haproxy 1.5-dev18 + patch corruption happened!!
On Thu, May 09, 2013 at 01:06:21AM +0900, Seri wrote:
> Hi, Willy
>
> > diff --git a/include/common/regex.h b/include/common/regex.h
> > index bab1a55..0104019 100644
> > --- a/include/common/regex.h
> > +++ b/include/common/regex.h
> > @@ -79,7 +79,6 @@ static inline void regex_free(regex *preg) {
> > #ifdef USE_PCRE_JIT
> > pcre_free_study(preg->extra);
> > pcre_free(preg->reg);
> > -free(preg);
> > #else
> > regfree(preg);
> > #endif
>
> I've tested after applying this patch.
>
> It works very well without corrupting.
Thank you for the report, I'm merging it then.
Best regards,
Willy
Re: 1.5-dev .spec File Issues
Hello again,
On 05/07/2013 01:19 AM, Willy Tarreau wrote:
Hi guys,
On Tue, May 07, 2013 at 12:54:29AM +0200, Lukas Tribus wrote:
Hi Clay!
I had a few more modifications to the spec file that I am
including to get the rpmbuild to finish successfully for us.
Ok.
I'm not familiar with spec files or rpm packaging at all,
so please bear with me. Also I have not tested this.
I'm attaching your patch in git format, so Willy and others
can review the patch.
Its an unspectacular change in examples/*, I think we should
include it.
I'm fine with it, but it will require me to modify my release scripts, so
before I merge it, does anyone know if there is a way in the spec file to
split a variable in two ? That way we could have full_version=1.5-dev18
and build both version and release from this ?
If it's not possible not trivial, then I'll adapt my release scripts.
Note that it proves how long spec files have not been used !
Thanks,
Willy
Is this along the lines of what you had in mind:
%define full_version 1.5-dev18
%define version %(echo %{full_version} | awk -F "-" '{print $1}')
%define release %(echo %{full_version} | awk -F "-" '{print $2}')
Version: %{version}
Release: %{release}
Here is my current diff containing this and the previous discussed
changes in comparison to the existing haproxy.spec in examples/:
# diff -u haproxy.spec.orig haproxy.spec
--- haproxy.spec.orig 2013-05-06 10:19:48.462440897 -0500
+++ haproxy.spec2013-05-06 22:17:08.551225524 -0500
@@ -1,12 +1,16 @@
+%define full_version 1.5-dev18
+%define version %(echo %{full_version} | awk -F "-" '{print $1}')
+%define release %(echo %{full_version} | awk -F "-" '{print $2}')
+
Summary: HA-Proxy is a TCP/HTTP reverse proxy for high availability
environments
Name: haproxy
-Version: 1.5-dev18
-Release: 1
+Version: %{version}
+Release: %{release}
License: GPL
Group: System Environment/Daemons
URL: http://haproxy.1wt.eu/
-Source0:
http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}.tar.gz
-BuildRoot: %{_tmppath}/%{name}-%{version}-root
+Source0:
http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}-%{release}.tar.gz
+BuildRoot: %{_tmppath}/%{name}-%{version}.root
BuildRequires: pcre-devel
Requires: /sbin/chkconfig, /sbin/service
@@ -27,7 +31,7 @@
risking the system's stability.
%prep
-%setup -q
+%setup -q -n %{name}-%{version}-%{release}
# We don't want any perl dependecies in this RPM:
%define __perl_requires /bin/true
Re: 1.5-dev .spec File Issues
On 05/08/2013 11:13 AM, Clayton Keller wrote:
Hello again,
On 05/07/2013 01:19 AM, Willy Tarreau wrote:
Hi guys,
On Tue, May 07, 2013 at 12:54:29AM +0200, Lukas Tribus wrote:
Hi Clay!
I had a few more modifications to the spec file that I am
including to get the rpmbuild to finish successfully for us.
Ok.
I'm not familiar with spec files or rpm packaging at all,
so please bear with me. Also I have not tested this.
I'm attaching your patch in git format, so Willy and others
can review the patch.
Its an unspectacular change in examples/*, I think we should
include it.
I'm fine with it, but it will require me to modify my release scripts, so
before I merge it, does anyone know if there is a way in the spec file to
split a variable in two ? That way we could have full_version=1.5-dev18
and build both version and release from this ?
If it's not possible not trivial, then I'll adapt my release scripts.
Note that it proves how long spec files have not been used !
Thanks,
Willy
Is this along the lines of what you had in mind:
%define full_version 1.5-dev18
%define version %(echo %{full_version} | awk -F "-" '{print $1}')
%define release %(echo %{full_version} | awk -F "-" '{print $2}')
Version: %{version}
Release: %{release}
Here is my current diff containing this and the previous discussed
changes in comparison to the existing haproxy.spec in examples/:
Had a "." slip in there on the "BuildRoot" value, my apologies. Here's
what should have been sent:
# diff -u haproxy.spec.orig haproxy.spec
--- haproxy.spec.orig 2013-05-06 10:19:48.462440897 -0500
+++ haproxy.spec2013-05-06 22:33:24.175855675 -0500
@@ -1,11 +1,15 @@
+%define full_version 1.5-dev18
+%define version %(echo %{full_version} | awk -F "-" '{print $1}')
+%define release %(echo %{full_version} | awk -F "-" '{print $2}')
+
Summary: HA-Proxy is a TCP/HTTP reverse proxy for high availability
environments
Name: haproxy
-Version: 1.5-dev18
-Release: 1
+Version: %{version}
+Release: %{release}
License: GPL
Group: System Environment/Daemons
URL: http://haproxy.1wt.eu/
-Source0:
http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}.tar.gz
+Source0:
http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}-%{release}.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{version}-root
BuildRequires: pcre-devel
Requires: /sbin/chkconfig, /sbin/service
@@ -27,7 +31,7 @@
risking the system's stability.
%prep
-%setup -q
+%setup -q -n %{name}-%{version}-%{release}
# We don't want any perl dependecies in this RPM:
%define __perl_requires /bin/true
Re: 1.5-dev .spec File Issues
Hello Clayton,
On Wed, May 08, 2013 at 11:13:04AM -0500, Clayton Keller wrote:
> Is this along the lines of what you had in mind:
>
> %define full_version 1.5-dev18
> %define version %(echo %{full_version} | awk -F "-" '{print $1}')
> %define release %(echo %{full_version} | awk -F "-" '{print $2}')
>
> Version: %{version}
> Release: %{release}
Yes I didn't know this was possible. Seems a little bit dirty but is
handy. You can even use "cut -d- -f1" / "cut -d- -f2".
> Here is my current diff containing this and the previous discussed
> changes in comparison to the existing haproxy.spec in examples/:
(...)
I'm seeing one issue here :
> -BuildRoot: %{_tmppath}/%{name}-%{version}-root
> +BuildRoot: %{_tmppath}/%{name}-%{version}.root
I think you wanted to use ".../%{name}-%{version}-%{release}-root" instead.
Thanks,
Willy
Re: 1.5-dev .spec File Issues
Hi there,
On Wed, 08 May 2013, 18:19:43 +0200, Clayton Keller wrote:
> [...]
> Had a "." slip in there on the "BuildRoot" value, my apologies.
> Here's what should have been sent:
>From my experience with building RPMs, this is very close to what should
be applied. The rest is nit-picking, such as defining 1.5 as the
full_version, and adding further minor versions attached to it. But the
version you have sent should just work!
Cheers.
l8er
manfred
> # diff -u haproxy.spec.orig haproxy.spec
> --- haproxy.spec.orig 2013-05-06 10:19:48.462440897 -0500
> +++ haproxy.spec2013-05-06 22:33:24.175855675 -0500
> @@ -1,11 +1,15 @@
> +%define full_version 1.5-dev18
> +%define version %(echo %{full_version} | awk -F "-" '{print $1}')
> +%define release %(echo %{full_version} | awk -F "-" '{print $2}')
> +
> Summary: HA-Proxy is a TCP/HTTP reverse proxy for high availability
> environments
> Name: haproxy
> -Version: 1.5-dev18
> -Release: 1
> +Version: %{version}
> +Release: %{release}
> License: GPL
> Group: System Environment/Daemons
> URL: http://haproxy.1wt.eu/
> -Source0:
> http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}.tar.gz
> +Source0:
> http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}-%{release}.tar.gz
> BuildRoot: %{_tmppath}/%{name}-%{version}-root
> BuildRequires: pcre-devel
> Requires: /sbin/chkconfig, /sbin/service
> @@ -27,7 +31,7 @@
> risking the system's stability.
>
> %prep
> -%setup -q
> +%setup -q -n %{name}-%{version}-%{release}
>
> # We don't want any perl dependecies in this RPM:
> %define __perl_requires /bin/true
Re: 1.5-dev .spec File Issues
On 05/08/2013 11:26 AM, Manfred Hollstein wrote: Hi there, On Wed, 08 May 2013, 18:19:43 +0200, Clayton Keller wrote: [...] Had a "." slip in there on the "BuildRoot" value, my apologies. Here's what should have been sent: From my experience with building RPMs, this is very close to what should be applied. The rest is nit-picking, such as defining 1.5 as the full_version, and adding further minor versions attached to it. But the version you have sent should just work! Cheers. l8er manfred Thanks with the 1.5-dev18 naming convention this changes that idea up a bit. I would be typically be used to something like 1.5.1. With .1 being the version and incremented. However he has been using dev18 as that type of incremental version value. And with limitations and restrictions to rpmbuild not allowing a Release value to contain a "-" I've made these adjustments so that we could successfully build from the packaged haproxy.spec file. Clay
Re: 1.5-dev .spec File Issues
Forgot to make sure this went to the list.
On 05/08/2013 11:21 AM, Willy Tarreau wrote:
Hello Clayton,
On Wed, May 08, 2013 at 11:13:04AM -0500, Clayton Keller wrote:
Is this along the lines of what you had in mind:
%define full_version 1.5-dev18
%define version %(echo %{full_version} | awk -F "-" '{print $1}')
%define release %(echo %{full_version} | awk -F "-" '{print $2}')
Version: %{version}
Release: %{release}
Yes I didn't know this was possible. Seems a little bit dirty but is
handy. You can even use "cut -d- -f1" / "cut -d- -f2".
I like the cut variation. I'll attach a patch file this time rather than
muddy up the post with the full diff again with it using the cut command
instead.
Here is my current diff containing this and the previous discussed
changes in comparison to the existing haproxy.spec in examples/:
(...)
I'm seeing one issue here :
-BuildRoot: %{_tmppath}/%{name}-%{version}-root
+BuildRoot: %{_tmppath}/%{name}-%{version}.root
I think you wanted to use ".../%{name}-%{version}-%{release}-root" instead.
Yes, I saw that too. I sent a follow-up email shortly after with the
adjusted/correct diff. This patch includes that as well as the cut
command change.
I hope this has been helpful.
Clay
--- haproxy.spec.orig 2013-05-06 10:19:48.462440897 -0500
+++ haproxy.spec 2013-05-06 22:57:38.567834726 -0500
@@ -1,11 +1,15 @@
+%define full_version 1.5-dev18
+%define version %(echo %{full_version} | cut -d- -f1)
+%define release %(echo %{full_version} | cut -d- -f2)
+
Summary: HA-Proxy is a TCP/HTTP reverse proxy for high availability environments
Name: haproxy
-Version: 1.5-dev18
-Release: 1
+Version: %{version}
+Release: %{release}
License: GPL
Group: System Environment/Daemons
URL: http://haproxy.1wt.eu/
-Source0: http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}.tar.gz
+Source0: http://haproxy.1wt.eu/download/1.5/src/devel/%{name}-%{version}-%{release}.tar.gz
BuildRoot: %{_tmppath}/%{name}-%{version}-root
BuildRequires: pcre-devel
Requires: /sbin/chkconfig, /sbin/service
@@ -27,7 +31,7 @@
risking the system's stability.
%prep
-%setup -q
+%setup -q -n %{name}-%{version}-%{release}
# We don't want any perl dependecies in this RPM:
%define __perl_requires /bin/true
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy,
Could you please let me know what your findings are about the proposed
patch?
Does it need some more work, is it implemented wrongly, or would it help
if i send my current haproxy.cfg file?
If i need to change something please let me know, thanks.
Thanks for your time,
PiBa-NL
Op 3-5-2013 18:03, Willy Tarreau schreef:
Hi,
sorry, I missed it.
I'll give it a look and merge it if it's OK.
Thanks, Willy
Op 27-4-2013 18:08, PiBa-NL schreef:
Hi Willy,
I generated 2 patch files:
-"FreeBSD IP_BINDANY git diff.patch" generated with a git diff
(against a hopefully relatively recent source tree)(i couldnt get it
to fetch http://git.1wt.eu/git/haproxy.git ..)
-"FreeBSD IP_BINDANY diff -urN.patch" generated with diff -urN
(against the 'port source')
I hope one of them can be used by you.
Please take a look and comment if something is amiss.
Greetings
PiBa-NL
diff -urN workoriginal/haproxy-1.5-dev18/include/common/compat.h
work/haproxy-1.5-dev18/include/common/compat.h
--- workoriginal/haproxy-1.5-dev18/include/common/compat.h 2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/include/common/compat.h 2013-04-27
14:56:27.0 +
@@ -93,6 +93,15 @@
#endif /* !IPV6_TRANSPARENT */
#endif /* CONFIG_HAP_LINUX_TPROXY */
+#if (defined(SOL_IP) && defined(IP_TRANSPARENT)) \
+ || (defined(SOL_IPV6) && defined(IPV6_TRANSPARENT)) \
+ || (defined(SOL_IP) && defined(IP_FREEBIND)) \
+ || (defined(IPPROTO_IP) && defined(IP_BINDANY)) \
+ || (defined(IPPROTO_IPV6) && defined(IPV6_BINDANY)) \
+ || (defined(SOL_SOCKET) && defined(SO_BINDANY))
+ #define HAP_TRANSPARENT
+#endif
+
/* We'll try to enable SO_REUSEPORT on Linux 2.4 and 2.6 if not defined.
* There are two families of values depending on the architecture. Those
* are at least valid on Linux 2.4 and 2.6, reason why we'll rely on the
diff -urN workoriginal/haproxy-1.5-dev18/include/types/connection.h
work/haproxy-1.5-dev18/include/types/connection.h
--- workoriginal/haproxy-1.5-dev18/include/types/connection.h 2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/include/types/connection.h 2013-04-27
14:56:30.0 +
@@ -219,7 +219,7 @@
char *iface_name;/* bind interface name or NULL */
struct port_range *sport_range; /* optional per-server TCP source
ports */
struct sockaddr_storage source_addr; /* the address to which we want to
bind for connect() */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT)
struct sockaddr_storage tproxy_addr; /* non-local address we want to
bind to for connect() */
char *bind_hdr_name; /* bind to this header name if
defined */
int bind_hdr_len;/* length of the name of the
header above */
diff -urN workoriginal/haproxy-1.5-dev18/src/backend.c
work/haproxy-1.5-dev18/src/backend.c
--- workoriginal/haproxy-1.5-dev18/src/backend.c2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/src/backend.c2013-04-27 14:56:32.0
+
@@ -884,7 +884,7 @@
*/
static void assign_tproxy_address(struct session *s)
{
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT)
struct server *srv = objt_server(s->target);
struct conn_src *src;
diff -urN workoriginal/haproxy-1.5-dev18/src/cfgparse.c
work/haproxy-1.5-dev18/src/cfgparse.c
--- workoriginal/haproxy-1.5-dev18/src/cfgparse.c 2013-04-26
19:36:15.0 +
+++ work/haproxy-1.5-dev18/src/cfgparse.c 2013-04-27 14:56:33.0
+
@@ -4535,8 +4535,8 @@
cur_arg += 2;
while (*(args[cur_arg])) {
if (!strcmp(args[cur_arg], "usesrc")) {
/* address to use outside */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
-#if !defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(HAP_TRANSPARENT)
+#if !defined(HAP_TRANSPARENT)
if
(!is_addr(&newsrv->conn_src.source_addr)) {
Alert("parsing [%s:%d]
: '%s' requires an explicit '%s' address.\n",
file, linenum,
"usesrc", "source");
@@ -4625,7 +4625,7 @@
newsrv->conn_src.opts
|= CO_SRC_TPROXY_ADDR;
}
global.last_checks |=
LSTCHK_NETADM;
-#if !defined(CONFIG_HAP_LINUX_TPROXY)
+#if !defined(HAP_TRANSPARENT)
global.last_checks |=
LSTCHK_CTTPROXY;
#endif
cur_arg += 2;
@@ -4635,7 +46
Re: 1.5-dev .spec File Issues
On 05/08/2013 12:05 PM, Clayton Keller wrote:
Forgot to make sure this went to the list.
On 05/08/2013 11:21 AM, Willy Tarreau wrote:
Hello Clayton,
On Wed, May 08, 2013 at 11:13:04AM -0500, Clayton Keller wrote:
Is this along the lines of what you had in mind:
%define full_version 1.5-dev18
%define version %(echo %{full_version} | awk -F "-" '{print $1}')
%define release %(echo %{full_version} | awk -F "-" '{print $2}')
Version: %{version}
Release: %{release}
Yes I didn't know this was possible. Seems a little bit dirty but is
handy. You can even use "cut -d- -f1" / "cut -d- -f2".
I like the cut variation. I'll attach a patch file this time rather than
muddy up the post with the full diff again with it using the cut command
instead.
Here is my current diff containing this and the previous discussed
changes in comparison to the existing haproxy.spec in examples/:
(...)
I'm seeing one issue here :
-BuildRoot: %{_tmppath}/%{name}-%{version}-root
+BuildRoot: %{_tmppath}/%{name}-%{version}.root
I think you wanted to use ".../%{name}-%{version}-%{release}-root"
instead.
Yes, I saw that too. I sent a follow-up email shortly after with the
adjusted/correct diff. This patch includes that as well as the cut
command change.
I hope this has been helpful.
I guess keep in mind that once 1.5 drops the -devXX naming convention
and were to fall back in the more standard 1.5.x the use of this is not
as necessary and the Version could again be the entire release number
with the Release being more RPM build related.
Clay
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, On Wed, May 08, 2013 at 07:34:19PM +0200, PiBa-NL wrote: > Hi Willy, > > Could you please let me know what your findings are about the proposed > patch? I was on it this afternoon (didn't have time earlier) :-) I haven't finished reviewing it yet, because I was trying to figure if there would be an easy way to merge the CTTPROXY mode into the other transparent proxy options, but I'm not sure that's really useful. Also I found one issue here : + int ret = 0; + #if defined(SOL_IP) && defined(IP_TRANSPARENT) + ret |= setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0; + #endif + #if defined(SOL_IP) && defined(IP_FREEBIND) + ret |= setsockopt(fd, SOL_IP, IP_FREEBIND, &one, sizeof(one)) == 0; + #endif + #if defined(IPPROTO_IP) && defined(IP_BINDANY) + ret |= setsockopt(fd, IPPROTO_IP, IP_BINDANY, &one, sizeof(one)) == 0; + #endif + #if defined(SOL_SOCKET) && defined(SO_BINDANY) + ret |= setsockopt(fd, SOL_SOCKET, SO_BINDANY, &one, sizeof(one)) == 0; + #endif + if (ret) As you can see, if we have multiple defines, we'll call setsockopt multiple times, which we don't want. I was thinking about something like this instead : if (0 #if cond1 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0 #endif #if cond2 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0 #endif ) ... I'm still on it right now, to ensure we don't break anything. > Does it need some more work, is it implemented wrongly, or would it help > if i send my current haproxy.cfg file? > > If i need to change something please let me know, thanks. I do not think so, I can easily perform the changes above myself, I won't harrass you with another iteration. Overall it's good but since we're changing many things at once, I'm cautious. I'd prefer to break it in two BTW : 1) change existing code to support CONFIG_HAP_TRANSPARENT everywhere 2) add FreeBSD support But if that's OK for you, I'll simply perform the small adjustments before merging it. Cheers, Willy
Re: In SSL environment, HTTP Keepalive timeout issue
On Thu, May 09, 2013 at 03:14:13AM +0900, Seri wrote: > Hi, > > In Haproxy SSL and HTTP KeepAlive environment, > After keepalive timeout, haproxy sends FIN to client, but Client(Windows IE > or Chrome) doesn't send ACK to haproxy server. > > So, haproxy server left TCP FIN_WAIT2 state and client left CLOSE_WAIT state. So this means you have a bug on the client which didn't notice the FIN. It's possible it was doing something else and not monitoring the connection state, and will notice it once it needs to perform another request. Anyway I find this dirty. > How this issue will be solved? Only on the client. Don't worry, this is harmless. FIN_WAIT2 connections don't last long, so even if a firewall in the middle expires its session, the one on the haproxy side will quickly be dropped, and once the client decides itself to take a look at the events it received, it will notice that the connection is closed. Regards, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi Willy, If you make some changes to what you think/know is better and break the change into two parts is fine for me. About calling setsockopt multiple times, i think the "ret |= " would not evaluate the call behind it if "ret" already is 1, not absolutely sure about that.. I didn't think of starting a if statement with "0 ||" which might speed it up a clock tick or two so would be better anyway instead of having a variable assignment in between. Thanks, could you let me know when its ready then ill give it another compile&check on FreeBSD. And provide a little 'documentation' on how i configured the 'ipfw' firewall/nat to make it work. p.s. Ive spotted a issue in my patch with the IPv6 part where i forgot about the OpenBSD part (SOL_SOCKET & SO_BINDANY) should probably be added there also. PiBa-NL Op 8-5-2013 20:18, Willy Tarreau schreef: Hi, On Wed, May 08, 2013 at 07:34:19PM +0200, PiBa-NL wrote: Hi Willy, Could you please let me know what your findings are about the proposed patch? I was on it this afternoon (didn't have time earlier) :-) I haven't finished reviewing it yet, because I was trying to figure if there would be an easy way to merge the CTTPROXY mode into the other transparent proxy options, but I'm not sure that's really useful. Also I found one issue here : + int ret = 0; + #if defined(SOL_IP) && defined(IP_TRANSPARENT) + ret |= setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0; + #endif + #if defined(SOL_IP) && defined(IP_FREEBIND) + ret |= setsockopt(fd, SOL_IP, IP_FREEBIND, &one, sizeof(one)) == 0; + #endif + #if defined(IPPROTO_IP) && defined(IP_BINDANY) + ret |= setsockopt(fd, IPPROTO_IP, IP_BINDANY, &one, sizeof(one)) == 0; + #endif + #if defined(SOL_SOCKET) && defined(SO_BINDANY) + ret |= setsockopt(fd, SOL_SOCKET, SO_BINDANY, &one, sizeof(one)) == 0; + #endif + if (ret) As you can see, if we have multiple defines, we'll call setsockopt multiple times, which we don't want. I was thinking about something like this instead : if (0 #if cond1 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0 #endif #if cond2 || setsockopt(fd, SOL_IP, IP_TRANSPARENT, &one, sizeof(one)) == 0 #endif ) ... I'm still on it right now, to ensure we don't break anything. Does it need some more work, is it implemented wrongly, or would it help if i send my current haproxy.cfg file? If i need to change something please let me know, thanks. I do not think so, I can easily perform the changes above myself, I won't harrass you with another iteration. Overall it's good but since we're changing many things at once, I'm cautious. I'd prefer to break it in two BTW : 1) change existing code to support CONFIG_HAP_TRANSPARENT everywhere 2) add FreeBSD support But if that's OK for you, I'll simply perform the small adjustments before merging it. Cheers, Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
Hi, On Wed, May 08, 2013 at 09:41:33PM +0200, PiBa-NL wrote: > Hi Willy, > > If you make some changes to what you think/know is better and break the > change into two parts is fine for me. OK. > About calling setsockopt multiple times, i think the "ret |= " would not > evaluate the call behind it if "ret" already is 1, not absolutely sure > about that.. No, I can guarantee you that all of them will be called. > I didn't think of starting a if statement with "0 ||" which might speed > it up a clock tick or two so would be better anyway instead of having a > variable assignment in between. It's not a matter of saving a clock cycle but really not to call the setsockopt we don't want to call, while still keeping the ability to fall back to the remaining supported ones when possible. > Thanks, could you let me know when its ready then ill give it another > compile&check on FreeBSD. And provide a little 'documentation' on how i > configured the 'ipfw' firewall/nat to make it work. Perfect, I'll send you the patch back before merging it then. > p.s. > Ive spotted a issue in my patch with the IPv6 part where i forgot about > the OpenBSD part (SOL_SOCKET & SO_BINDANY) should probably be added > there also. No problem, we'll add this as a third patch. It's really important to have one feature per commit, because when for some reason we introduce regressions, users can easily revert just the faulty commit without losing the other ones. Cheers, Willy
Re: Add X-Forwarded-For
Thanks everyone for answering. I'll play around a bit with my config and the suggestions. Greets, Sander On 8 mei 2013, at 15:04, Willy Tarreau wrote: > On Wed, May 08, 2013 at 08:29:15AM -0400, John Marrett wrote: >> The definitive list of cloudflare IPs doesn't appear to be too unmanageable: >> >> https://www.cloudflare.com/ips >> >> They also provide convenient text files that just contain the IP address >> lists for easy automation. >> >> As Lukas says if you do not validate the IP addresses it's trivial for >> anyone to forge client IP addresses. > > I agree, and indeed the list is very small, I thought it was much larger, > as akamai's which are much harder to deal with. > > I think the following method should work, though I have not tested it : > >acl from_cf src -f cf-ips.txt # list of cf's addresses, one per line >reqidel ^x-forwarded-for: if !from_cf >option forwardfor if-none > > It is supposed to remove xff from requests not coming from CF, and to add > one only when there is none, which should do the trick. > > Willy > >
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
> > p.s. > > Ive spotted a issue in my patch with the IPv6 part where i forgot about > > the OpenBSD part (SOL_SOCKET & SO_BINDANY) should probably be added > > there also. > > No problem, we'll add this as a third patch. It's really important to > have one feature per commit, because when for some reason we introduce > regressions, users can easily revert just the faulty commit without > losing the other ones. BTW, I noticed that we can safely remove the following tests : - defined(SOL_IP) - defined(SOL_IPV6) => both are used only on linux and were previously present - defined(IPPROTO_IP) - defined(IPPROTO_IPV6) => we already have many other IPPROTO_* defines (mainly IPPROTO_TCP) so we already have the correct includes to use them. - defined(SOL_SOCKET) => is already referenced a lot, so it's safe. It's nice because it will clear the #ifdef a little bit. Willy
Re: HAProxy on FreeBSD 8.3 with transparent proxying (TProxy?)
OK here's what I came up with. There are 3 patches :
- 0001 : reorganize flags processing
- 0002 : add support for freebsd
- 0003 : add support for openbsd
Please review and test if you can. At least it seems OK on linux here.
I have written all the commit messages. Feel free to change them if you
want, as they're made under your name. If you want to provide additional
doc, let's just add a 4th patch on top of this.
The code is not quite beautiful, but that's always the price to pay
when playing with ifdefs, and there are already a large number of them
in the same functions anyway.
Also, if you could provide a real name for the commits, it would be nice!
Thanks!
Willy
>From fa20b9333c0ee547ec1214da73bdc2d7163442dd Mon Sep 17 00:00:00 2001
From: PiBa-NL
Date: Wed, 8 May 2013 22:49:23 +0200
Subject: REORG: tproxy: prepare the transparent proxy defines for accepting
other OSes
This patch does not change the logic of the code, it only changes the
way OS-specific defines are tested.
At the moment the transparent proxy code heavily depends on Linux-specific
defines. This first patch introduces a new define "CONFIG_HAP_TRANSPARENT"
which is set every time the defines used by transparent proxy are present.
This also means that with an up-to-date libc, it should not be necessary
anymore to force CONFIG_HAP_LINUX_TPROXY during the build, as the flags
will automatically be detected.
The CTTPROXY flags still remain separate because this older API doesn't
work the same way.
A new line has been added in the version output for haproxy -vv to indicate
what transparent proxy support is available.
---
include/common/compat.h| 6 ++
include/types/connection.h | 2 +-
src/backend.c | 2 +-
src/cfgparse.c | 20 +--
src/haproxy.c | 16 +++
src/proto_tcp.c| 49 ++
6 files changed, 70 insertions(+), 25 deletions(-)
diff --git a/include/common/compat.h b/include/common/compat.h
index bb2d010..043a56e 100644
--- a/include/common/compat.h
+++ b/include/common/compat.h
@@ -93,6 +93,12 @@
#endif /* !IPV6_TRANSPARENT */
#endif /* CONFIG_HAP_LINUX_TPROXY */
+#if defined(IP_FREEBIND) \
+ || defined(IP_TRANSPARENT)\
+ || defined(IPV6_TRANSPARENT)
+#define CONFIG_HAP_TRANSPARENT
+#endif
+
/* We'll try to enable SO_REUSEPORT on Linux 2.4 and 2.6 if not defined.
* There are two families of values depending on the architecture. Those
* are at least valid on Linux 2.4 and 2.6, reason why we'll rely on the
diff --git a/include/types/connection.h b/include/types/connection.h
index 255811c..2c7acd1 100644
--- a/include/types/connection.h
+++ b/include/types/connection.h
@@ -219,7 +219,7 @@ struct conn_src {
char *iface_name;/* bind interface name or NULL */
struct port_range *sport_range; /* optional per-server TCP source
ports */
struct sockaddr_storage source_addr; /* the address to which we want to
bind for connect() */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_TRANSPARENT)
struct sockaddr_storage tproxy_addr; /* non-local address we want to
bind to for connect() */
char *bind_hdr_name; /* bind to this header name if
defined */
int bind_hdr_len;/* length of the name of the
header above */
diff --git a/src/backend.c b/src/backend.c
index 9f4e635..9f23018 100644
--- a/src/backend.c
+++ b/src/backend.c
@@ -884,7 +884,7 @@ int assign_server_and_queue(struct session *s)
*/
static void assign_tproxy_address(struct session *s)
{
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_TRANSPARENT)
struct server *srv = objt_server(s->target);
struct conn_src *src;
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 6f2850c..9907bfd 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -1813,7 +1813,7 @@ int cfg_parse_listen(const char *file, int linenum, char
**args, int kwm)
curproxy->conn_src.iface_name =
strdup(defproxy.conn_src.iface_name);
curproxy->conn_src.iface_len =
defproxy.conn_src.iface_len;
curproxy->conn_src.opts = defproxy.conn_src.opts;
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPROXY)
+#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_TRANSPARENT)
curproxy->conn_src.tproxy_addr =
defproxy.conn_src.tproxy_addr;
#endif
}
@@ -4558,8 +4558,8 @@ stats_error_parsing:
cur_arg += 2;
while (*(args[cur_arg])) {
if (!strcmp(args[cur_arg], "usesrc")) {
/* address to use outside */
-#if defined(CONFIG_HAP_CTTPROXY) || defined(CONFIG_HAP_LINUX_TPRO
