Re: Generic backend in HAProxy config with server options as placeholders

[Vijay Bais]

On Thu, Nov 15, 2018 at 4:07 AM Igor Cicimov 
wrote:

>
>
> On Thu, Nov 15, 2018 at 1:36 AM Aleksandar Lazic 
> wrote:
>
>> Hi Vijay.
>>
>> Am 14.11.2018 um 10:14 schrieb Vijay Bais:
>> > Hello Aleksandar,
>> >
>> > We already considered using haproxy maps but we still have to define N
>> backends
>> > for corresponding N keys in the map file.
>> > I'm looking more at an implementation with single backend definition
>> with the
>> > server options as placeholders.
>> >
>> > Ex. Using maps would look something like this
>> >
>> > frontend nat
>> > bind *:1
>> > use_backend %[req.hdr(X-MyHeader), map(/etc/haproxy/my.map)]
>> >
>> > backend example1.com
>> > server myserver1 example1.com:80 source 10.0.0.1
>> >
>> > backend example2.com
>> > server myserver2 example2.com:80 source 10.0.0.2
>> >
>> > backend example3.com
>> > server myserver3 example3.com:80 source 10.0.0.3
>> >
>> >
>> >
>> > Whereas, we are looking for something like below
>> >
>> >
>> > frontend nat
>> > bind *:1
>> > default_backend generic
>> >
>> > backend generic
>> > server myserver %[req.hdr(X-MyHeader)] source %[dst]
>>
>> Ah now concrete examples ;-)
>>
>> Maybe you can use the server template?!
>>
>> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-server-template
>>
>> So you would like to have something like this, is this possible, I don't
>> think so?
>>
>> backend generic
>>   server-template myserver 1-3 %[req.hdr(X-MyHeader)]:80 check source
>> 0.0.0.0
>> usesrc %[dst]
>>
> @Aleksandar Lazic  tried using server-template, but
it won't work


>
>> Which version of HAProxy do you use?
>>
> We are using v1.8.13


>> haproxy -vv
>>
>  HA-Proxy version 1.8.13 2018/07/30
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -fno-strict-overflow -Wno-unused-label
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with network namespace support.
Built with zlib version : 1.2.3
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 7.8 2008-09-05
Running on PCRE version : 8.21 2011-12-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe


>> > Thanks,
>> > Vijay
>>
>> Regards
>> Aleks
>>
>> > On Wed, Nov 14, 2018 at 1:39 PM Aleksandar Lazic > > <mailto:al-hapr...@none.at>> wrote:
>> >
>> > Hi.
>> >
>> > Am 14.11.2018 um 08:46 schrieb Vijay Bais:
>> > > Hello,
>> > >
>> > > We have a requirement wherein a single generic backend with
>> server options
>> > > configured as placeholders, which will resolve on the fly or at
>> runtime.
>> > >
>> > > Currently, we have to define multiple backends (has to be
>> hardcoded) and
>> > select
>> > > them using the /use_backend/ keyword.
>> > >
>> > > Kindly help us with this generic backend implementation in
>> HAProxy and let us
>> > > know if its possible OR any alternative way that this can be
>> achieved.
>> >
>> > Maybe you can use maps for your requirement.
>> >
>> > https://www.haproxy.com/blog/introduction-to-haproxy-maps/
>> >
>> > As an example can you take a look at the openshift router template
>> ;-)
>> >
>> >
>> https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L201-L202
>> >
>> > > Thank you in advance,
>> > > Vijay B
>> >
>> > Regards
>> > Aleks
>>
>
> You will need something dynamic like described here
> https://www.haproxy.com/blog/haproxy-and-consul-with-dns-for-service-discovery/
>
Thanks Igor! Let me explore this and get back :)

Re: Generic backend in HAProxy config with server options as placeholders

[Vijay Bais]

Hello Aleksandar,

We already considered using haproxy maps but we still have to define N
backends for corresponding N keys in the map file.
I'm looking more at an implementation with single backend definition with
the server options as placeholders.

Ex. Using maps would look something like this

frontend nat
bind *:1
use_backend %[req.hdr(X-MyHeader), map(/etc/haproxy/my.map)]

backend example1.com:80
server myserver1 example1.com:80 source 10.0.0.1

backend example2.com:80
server myserver2 example2.com:80 source 10.0.0.2

backend example3.com:80
server myserver3 example3.com:80 source 10.0.0.3



Whereas, we are looking for something like below


frontend nat
bind *:1
default_backend generic

backend generic
server myserver %[req.hdr(X-MyHeader)] source %[dst]


Thanks,
Vijay

On Wed, Nov 14, 2018 at 1:39 PM Aleksandar Lazic  wrote:

> Hi.
>
> Am 14.11.2018 um 08:46 schrieb Vijay Bais:
> > Hello,
> >
> > We have a requirement wherein a single generic backend with server
> options
> > configured as placeholders, which will resolve on the fly or at runtime.
> >
> > Currently, we have to define multiple backends (has to be hardcoded) and
> select
> > them using the /use_backend/ keyword.
> >
> > Kindly help us with this generic backend implementation in HAProxy and
> let us
> > know if its possible OR any alternative way that this can be achieved.
>
> Maybe you can use maps for your requirement.
>
> https://www.haproxy.com/blog/introduction-to-haproxy-maps/
>
> As an example can you take a look at the openshift router template ;-)
>
>
> https://github.com/openshift/origin/blob/master/images/router/haproxy/conf/haproxy-config.template#L201-L202
>
> > Thank you in advance,
> > Vijay B
>
> Regards
> Aleks
>

Generic backend in HAProxy config with server options as placeholders

[Vijay Bais]

Hello,

We have a requirement wherein a single generic backend with server options
configured as placeholders, which will resolve on the fly or at runtime.

Currently, we have to define multiple backends (has to be hardcoded) and
select them using the *use_backend* keyword.

Kindly help us with this generic backend implementation in HAProxy and let
us know if its possible OR any alternative way that this can be achieved.

Thank you in advance,
Vijay B

Re: Issue with HAProxy as a forward proxy

[Vijay Bais]

Thanks a lot, Willy!

Seems the issue was with preference of resolvers itself.
After configuring it to prefer IPv4 as stated by you; it starting working
like a charm.

Ref:
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2-resolve-prefer

Thanks again for your help.


On Thu, Nov 8, 2018 at 1:36 PM Willy Tarreau  wrote:

> Hello Vijay,
>
> I'm having several comments below :
>
> On Thu, Nov 08, 2018 at 08:21:37AM +0530, Vijay Bais wrote:
> > Hello Aleksandar,
> >
> > I tried running haproxy as root, but it still failed with same logs.
> >
> > Btw, here's the global section used,
> > 
> >
> > global
> > log 127.0.0.1 local2
> > chroot  /var/lib/haproxy
> > pidfile /var/run/haproxy.pid
> > maxconn 4000
> > userroot
> > group   root
> > daemon
> > stats socket /var/lib/haproxy/stats
> > stats timeout 2m
> > master-worker
> > nbthread 10
>
> First, I'd be very surprised if you'd need 10 threads for only 4000
> connections, please do not add stuff you don't need, this only
> complicates the troubleshooting and help people can give you.
>
> > defaults
> (...)
> > default-server  resolvers dns
> >
> > resolvers dns
> > nameserver local 127.0.0.1:53 <http://127.0.0.1:53>
> > nameserver ns1   10.0.0.2:53 <http://10.0.0.2:53>
> > hold valid 1s
> >
> > listen c1
> > bind   *10.0.0.26:10001 <http://10.0.0.26:10001>*
> > mode   tcp
> > option tcplog
> > server r1 *ifconfig.co:80 <http://ifconfig.co:80>* source * Public IP>*
>
> So what this means is that the name "ifconfig.co" will be periodically
> resolved using the two resolvers above and that all your connections
> will be sent there from your public address. Please make sure that the
> source address you're forcing is properly bound on your server, and
> usable to reach the address corresponding to "ifconfig.co".
>
> Last, I'm a bit surprised by your statement that you're trying to make a
> forward proxy because at least haproxy is not a forward http proxy, so
> the term is confusing. Your config makes me think that you want to use
> it to always reach ifconfig.co whose address may change overtime, am I
> right ?
>
> I'm also seeing that this host resolves both in IPv4 and IPv6. There's
> something to put in resolvers sections if you want to force v4 only I
> guess but I forgot the option name, you may want to take a look there.
>
> Regards,
> Willy
>

Re: Issue with HAProxy as a forward proxy

[Vijay Bais]

Hello Aleksandar,

I tried running haproxy as root, but it still failed with same logs.

Btw, here's the global section used,


global
log 127.0.0.1 local2
chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
userroot
group   root
daemon
stats socket /var/lib/haproxy/stats
stats timeout 2m
master-worker
nbthread 10



Please let me know if anything is missing here.

Thanks & Regards,
Vijay B

On Wed, Nov 7, 2018 at 4:23 PM Vijay Bais  wrote:

> Thanks for the reply!
>
> Here, the haproxy process runs as a *haproxy* user itself.
>
> Let me try it as root and revert back.
>
> On Tue 6 Nov, 2018, 9:19 PM Aleksandar Lazic,  wrote:
>
>> Hi Vijay.
>>
>> Am 06.11.2018 um 10:06 schrieb Vijay Bais:
>> > Hello,
>> >
>> > I'm using HAProxy 1.8 as a forward proxy with below configuration
>> >
>> > 
>> >
>> > defaults
>> > modetcp
>> > log global
>> > option  tcplog
>> > option  dontlognull
>> > option http-server-close
>> > #option forwardfor   except 127.0.0.0/8 <http://127.0.0.0/8
>> >
>> > option  redispatch
>> > retries 3
>> > timeout http-request10s
>> > timeout queue   1m
>> > timeout connect 10s
>> > timeout client  1m
>> > timeout server  1m
>> > timeout http-keep-alive 10s
>> > timeout check   10s
>> > maxconn 3000
>> > default-server  resolvers dns
>> >
>> > resolvers dns
>> > nameserver local 127.0.0.1:53 <http://127.0.0.1:53>
>> > nameserver ns1   10.0.0.2:53 <http://10.0.0.2:53>
>> > hold valid 1s
>> >
>> > listen c1
>> > bind   *10.0.0.26:10001 <http://10.0.0.26:10001>*
>> > mode   tcp
>> > option tcplog
>> > server r1 *ifconfig.co:80 <http://ifconfig.co:80>* source *> Public IP>*
>> >
>> > 
>> >
>> > But this fails with below log lines for any internet destination (both
>> in TCP
>> > and HTTP mode):
>> >
>> > 10.0.1.79:47437 <http://10.0.1.79:47437>
>> [06/Nov/2018:09:35:31.170] c1 c1/r1
>> > 1/-1/0 0 SC 1/1/0/0/3 0/0
>> > Cannot bind to source address before connect() for backend c1.
>> >
>> >
>> >
>> > Whereas, if the destination is under my control (with my source public
>> IP fully
>> > whitelisted), then the flow works perfectly.
>> >
>> > Any help to know the actual issue would be great.
>>
>> The snipped does not show the global section.
>> I think you will need to run HAProxy as root to be able to do this.
>>
>> Do you run HAProxy as root?
>>
>> > Thanks,
>> > Vijay B
>>
>> Regards
>> Aleks
>>
>

Re: Issue with HAProxy as a forward proxy

[Vijay Bais]

Thanks for the reply!

Here, the haproxy process runs as a *haproxy* user itself.

Let me try it as root and revert back.

On Tue 6 Nov, 2018, 9:19 PM Aleksandar Lazic,  wrote:

> Hi Vijay.
>
> Am 06.11.2018 um 10:06 schrieb Vijay Bais:
> > Hello,
> >
> > I'm using HAProxy 1.8 as a forward proxy with below configuration
> >
> > 
> >
> > defaults
> > modetcp
> > log global
> > option  tcplog
> > option  dontlognull
> > option http-server-close
> > #option forwardfor   except 127.0.0.0/8 <http://127.0.0.0/8>
> > option  redispatch
> > retries 3
> > timeout http-request10s
> > timeout queue   1m
> > timeout connect 10s
> > timeout client  1m
> > timeout server  1m
> > timeout http-keep-alive 10s
> > timeout check   10s
> > maxconn 3000
> > default-server  resolvers dns
> >
> > resolvers dns
> > nameserver local 127.0.0.1:53 <http://127.0.0.1:53>
> > nameserver ns1   10.0.0.2:53 <http://10.0.0.2:53>
> > hold valid 1s
> >
> > listen c1
> > bind   *10.0.0.26:10001 <http://10.0.0.26:10001>*
> > mode   tcp
> > option tcplog
> > server r1 *ifconfig.co:80 <http://ifconfig.co:80>* source * Public IP>*
> >
> > 
> >
> > But this fails with below log lines for any internet destination (both
> in TCP
> > and HTTP mode):
> >
> > 10.0.1.79:47437 <http://10.0.1.79:47437> [06/Nov/2018:09:35:31.170]
> c1 c1/r1
> > 1/-1/0 0 SC 1/1/0/0/3 0/0
> > Cannot bind to source address before connect() for backend c1.
> >
> >
> >
> > Whereas, if the destination is under my control (with my source public
> IP fully
> > whitelisted), then the flow works perfectly.
> >
> > Any help to know the actual issue would be great.
>
> The snipped does not show the global section.
> I think you will need to run HAProxy as root to be able to do this.
>
> Do you run HAProxy as root?
>
> > Thanks,
> > Vijay B
>
> Regards
> Aleks
>

Issue with HAProxy as a forward proxy

[Vijay Bais]

Hello,

I'm using HAProxy 1.8 as a forward proxy with below configuration



defaults
modetcp
log global
option  tcplog
option  dontlognull
option http-server-close
#option forwardfor   except 127.0.0.0/8
option  redispatch
retries 3
timeout http-request10s
timeout queue   1m
timeout connect 10s
timeout client  1m
timeout server  1m
timeout http-keep-alive 10s
timeout check   10s
maxconn 3000
default-server  resolvers dns

resolvers dns
nameserver local 127.0.0.1:53
nameserver ns1   10.0.0.2:53
hold valid 1s

listen c1
bind   *10.0.0.26:10001 *
mode   tcp
option tcplog
server r1 *ifconfig.co:80 * source **



But this fails with below log lines for any internet destination (both in
TCP and HTTP mode):

10.0.1.79:47437 [06/Nov/2018:09:35:31.170] c1 c1/r1 1/-1/0 0 SC 1/1/0/0/3
0/0
Cannot bind to source address before connect() for backend c1.



Whereas, if the destination is under my control (with my source public IP
fully whitelisted), then the flow works perfectly.

Any help to know the actual issue would be great.

Thanks,
Vijay B

Re: Haproxy 1.8 rpm for CentOS 6

[Vijay Bais]

Yes, that's the plan :)

Just need to know if there is any dependency on the system and haproxy
version.

On Mon 18 Jun, 2018, 10:03 PM Gibson, Brian (IMS), 
wrote:

> You’d have to ask the standard repository maintainers for that answer.
> IMO though it’s trivial to compile so if you need it why not just compile
> it yourself?
>
>
>
> *From:* Vijay Bais [mailto:vija...@endurance.com]
> *Sent:* Monday, June 18, 2018 12:28 PM
> *To:* haproxy@formilux.org
> *Subject:* Haproxy 1.8 rpm for CentOS 6
>
>
>
> Hello,
>
>
>
> Is there a particular reason for no standard rpm available of haproxy-1.8
> on CentOS 6?
>
>
>
> Any inputs will be appreciate.
>
>
>
> Thanks,
>
> Vijay B
>
> --
>
> Information in this e-mail may be confidential. It is intended only for
> the addressee(s) identified above. If you are not the addressee(s), or an
> employee or agent of the addressee(s), please note that any dissemination,
> distribution, or copying of this communication is strictly prohibited. If
> you have received this e-mail in error, please notify the sender of the
> error.
>

Haproxy 1.8 rpm for CentOS 6

[Vijay Bais]

Hello,

Is there a particular reason for no standard rpm available of haproxy-1.8
on CentOS 6?

Any inputs will be appreciate.

Thanks,
Vijay B

Re: Issue while using Proxy protocol in TCP mode

[Vijay Bais]

An important thing I missed to mention that *the load balancer is an EC2
instance in AWS*.
Also, consider 5.6.7.8 as an EIP and the frontend uses private IP to bind.

When I tested proxy protocol with similar local setup, it works as expect
(disregard any version of haproxy).

Its likely, AWS to be the component causing the issue of intermittent
errors.

On Wed, Jun 14, 2017 at 5:46 PM, Christopher Faulet <cfau...@haproxy.com>
wrote:

> Le 14/06/2017 à 13:07, Vijay Bais a écrit :
>
>>
>>
>> On Wed, Jun 14, 2017 at 3:06 PM, Christopher Faulet <cfau...@haproxy.com
>> <mailto:cfau...@haproxy.com>> wrote:
>>
>> Ok, If the problem is still there with a stable release, be sure to
>> have the "send-proxy" directive on your server line (the one which
>> forwards the traffic to haproxy itself). If you have any doubt about
>> your configuration, please, share it.
>>
>> Below are the snippets on haproxy configuration of reverse proxy and load
>> balancer:
>>
>> *On reverse proxy(1.2.3.4):*
>> 
>> defaults
>>  log global
>>  mode tcp
>>  option tcplog
>>
>> listen revproxy :80
>>  server load-balancer 5.6.7.8:80 <http://5.6.7.8:80> send-proxy
>> 
>>
>> *On load balancer(5.6.7.8):*
>> 
>> defaults
>>  log global
>>  option forwardfor
>>  mode http
>>  option httplog
>>  option httpclose
>>
>> frontend web
>>  bind 5.6.7.8:80 <http://5.6.7.8:80>
>>  acl revproxy src 1.2.3.4/32 <http://1.2.3.4/32>
>>  tcp-request connection expect-proxy layer4 if revproxy
>>  default_backend web-backend
>>
>> backend web-backend
>>  balance leastconn
>>  server node1 11.11.11.11:8080 <http://11.11.11.11:8080> check
>>  server node2 22.22.22.22:8080 <http://22.22.22.22:8080> check
>> 
>>
>>
>> Let me know if anything seems incorrect here.
>>
>>
> Nothing strange here. In your first message, you said that the error is
> intermittent. does it means that some HTTP requests are correctly handled
> and others fail, all coming from 1.2.3.4 through the listener "revproxy" ?
> With this kind of configuration, all connections from 1.2.3.4 to
> 5.6.7.8:80 must use the PROXY protocol.
>
> Could you start your 2 instances of haproxy with the debug mode enabled
> (-d option) to do some requests and provide their outputs and the logs ?
>
> --
> Christopher Faulet
>

Re: Issue while using Proxy protocol in TCP mode

[Vijay Bais]

On Wed, Jun 14, 2017 at 3:06 PM, Christopher Faulet 
wrote:
>
> Ok, If the problem is still there with a stable release, be sure to have
> the "send-proxy" directive on your server line (the one which forwards the
> traffic to haproxy itself). If you have any doubt about your configuration,
> please, share it.
>
>
Below are the snippets on haproxy configuration of reverse proxy and load
balancer:

*On reverse proxy(1.2.3.4):*

defaults
log global
mode tcp
option tcplog

listen revproxy :80
server load-balancer 5.6.7.8:80 send-proxy


*On load balancer(5.6.7.8):*

defaults
log global
option forwardfor
mode http
option httplog
option httpclose

frontend web
bind 5.6.7.8:80
acl revproxy src 1.2.3.4/32
tcp-request connection expect-proxy layer4 if revproxy
default_backend web-backend

backend web-backend
balance leastconn
server node1 11.11.11.11:8080 check
server node2 22.22.22.22:8080 check



Let me know if anything seems incorrect here.

Thanks & Regards,
Vijay B

Re: Issue while using Proxy protocol in TCP mode

[Vijay Bais]

Yes, I agree it's an old and unstable version; will upgrade to a stable
release.

The client is also haproxy of same old version, which forwards the traffic
using proxy protocol (similar to the link specified earlier).

On 13 Jun 2017 9:49 p.m., "Lukas Tribus" <lu...@gmx.net> wrote:

> Hello Vijay,
>
>
> Am 13.06.2017 um 10:07 schrieb Vijay Bais:
> > Hello,
> >
> > I am using HAProxy version 1.5-dev25-a339395.
>
> This is an unstable, more than 3 years old development version of haproxy.
> There is no way we can support this release here.
>
> Upgrade to a stable release, first of all.
>
>
>
> >
> > Facing issues while using proxy protocol to preserve source IP address
> > despite reverse proxies
> > ref: https://www.haproxy.com/blog/preserve-source-ip-address-
> despite-reverse-proxies/
> >
> > Haproxy log of load-balancer shows below messages intermittently:
> > "Received something which does not look like a PROXY protocol header"
>
> Whos the client, sending (or not) the proxy protocol?
>
>
>
> Regards,
> Lukas
>
>

Issue while using Proxy protocol in TCP mode

[Vijay Bais]

Hello,

I am using HAProxy version 1.5-dev25-a339395.

Facing issues while using proxy protocol to preserve source IP address
despite reverse proxies
ref:
https://www.haproxy.com/blog/preserve-source-ip-address-despite-reverse-proxies/

Haproxy log of load-balancer shows below messages intermittently:
"Received something which does not look like a PROXY protocol header"

This breaks of my application often.

But works swiftly when removed the proxy protocol options.

Any help would be great.

Thanks,
Vijay B