Re: tcpdump and Haproxy SSL Offloading

[Igor Cicimov]

Hi Lukas,

On Sat, Jun 4, 2016 at 3:03 AM, Lukas Tribus  wrote:

> Hello,
>
>
> you can dump the symmetric keys from the browser and import them in
> wireshark to decrypt PFS protected TLS sessions [1]


Yes in case you want to troubleshoot something generic this is a good
approach but if you want to troubleshoot sessions not initiated by your
self, ie particular client connection, this is practically impossible.


> or downgrade your ciphers settings to non-PF ciphers. Properly decrypting
> the TLS session is the only way to really make sure you see what happens,
> even if there is a TLS related bug in the client or server (haproxy).
>
>
> Some other idea's are:
>
> - if your backend traffic is unencrypted, you may want to capture the
> traffic there.
>

Not practical though if you have tens of backend servers. Much better if
you have to troubleshoot on 2 instead of dozen of servers. At least that's
how I understand the question related to tcpdump usage on the haproxy
servers them self.


>
> - if haproxy is rejecting the request, check "show errors" on the admin
> socket.
>
>

As you said, the best solution is to not depend on haproxy specific
> features, as you don't want to modify existing infrastructure in a
> troubleshooting case.Maybe something
>
>
Outside haproxy maybe something like mitmproxy or sslstrip might help. Not
sure though have never used them myself.


>
> Another proxy layer means that you decrypt TLS on the front-end proxy,
> while you sniff the plaintext traffic between the front-end and the second
> tier proxy. You can probably do this with a single haproxy instance
> recirculating the traffic through a unix socket and capture the traffic on
> it, but it would require some trial and error and definitely some testing.
>
>
This will probably be faster but can't use tcpdump in that case.


>
> I believe the SSLKEYLOGFILE approach [1] to be the most efficient and
> simplest approach.
>
>
> cheers,
>
> lukas
>
>
> [1]
> https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-wireshark-the-easy-way/
>
>
>

Re: Haproxy limiting queue / connections

[Lukas Tribus]

There is really no need to set ulimit-n manually, haproxy will do that 
for you (as per the docs "it is recommended not to use this option"). 
Also, configuring maxconn correctly is the single most important 
configuration in haproxy, and will affect the performance as well as 
cause crashes if the OOM killer shots down a miss-configured haproxy 
process.


You should've seen the OOM kills in your dmesg though.


Regards,
Lukas

Re: tcpdump and Haproxy SSL Offloading

[Lukas Tribus]

Hello,


you can dump the symmetric keys from the browser and import them in 
wireshark to decrypt PFS protected TLS sessions [1] or downgrade your 
ciphers settings to non-PF ciphers. Properly decrypting the TLS session 
is the only way to really make sure you see what happens, even if there 
is a TLS related bug in the client or server (haproxy).



Some other idea's are:

- if your backend traffic is unencrypted, you may want to capture the 
traffic there.


- if haproxy is rejecting the request, check "show errors" on the admin 
socket.



As you said, the best solution is to not depend on haproxy specific 
features, as you don't want to modify existing infrastructure in a 
troubleshooting case.



Another proxy layer means that you decrypt TLS on the front-end proxy, 
while you sniff the plaintext traffic between the front-end and the 
second tier proxy. You can probably do this with a single haproxy 
instance recirculating the traffic through a unix socket and capture the 
traffic on it, but it would require some trial and error and definitely 
some testing.



I believe the SSLKEYLOGFILE approach [1] to be the most efficient and 
simplest approach.



cheers,

lukas


[1] 
https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-wireshark-the-easy-way/

external-check stdout ends up in load-balanced traffic, destroying tcp sessions

[Lukas Erlacher]

I am absolutely flabbergasted...

I have a "simple" transparent proxy application with two ldap backends.


Here is my haproxy config:
> erlacher@vmrbg80:~$ cat /etc/haproxy/haproxy.cfg
> # Managed by saltstack
> global
> log /dev/loglocal0
> log /dev/loglocal1 notice
> #   chroot /var/lib/haproxy
> #   user haproxy
> #   group haproxy
> daemon
> external-check
> 
> defaults
> log global
> modetcp
> option  tcplog
> option  dontlognull
> timeout connect 5000
> timeout client 1d
> timeout server 1d
> errorfile 400 /etc/haproxy/errors/400.http
> errorfile 403 /etc/haproxy/errors/403.http
> errorfile 408 /etc/haproxy/errors/408.http
> errorfile 500 /etc/haproxy/errors/500.http
> errorfile 502 /etc/haproxy/errors/502.http
> errorfile 503 /etc/haproxy/errors/503.http
> errorfile 504 /etc/haproxy/errors/504.http
> 
> frontend ft_ldaps
>   bind ldap-ha-ft:636 transparent
>   default_backend bk_ldaps
> backend bk_ldaps
>   balance source
>   source 0.0.0.0 usesrc clientip
>   option external-check
>   external-check command /usr/local/bin/ldapcheck.sh
>   server ldap1 ldap-ha-bk1:636 check
>   server ldap2 ldap-ha-bk2:636 check
> 
> frontend ft_ldap
>   bind ldap-ha-ft:389 transparent
>   default_backend bk_ldap
> backend bk_ldap
>   balance source
>   source 0.0.0.0 usesrc clientip
>   option external-check
>   external-check command /usr/local/bin/ldapcheck.sh
>   server ldap1 ldap-ha-bk1:389 check
>   server ldap2 ldap-ha-bk2:389 check

Here is the ldapcheck.sh script:
> erlacher@vmrbg80:~$ cat /usr/local/bin/ldapcheck.sh 
> #!/bin/bash
> 
> backend=$3
> 
> #/usr/bin/ldapsearch -x -L -H "ldap://$backend"; -b ou=blabliblub '(mail=!)' dn
> 
> exit 0

I will neglect to post all of the sysctl, ip rule, ip route etc. etc. settings, 
they roughly follow the instructions in [1] with my own improvements previously 
posted to this ML and they work.

Everything works in this configuration. HOWEVER, if I delete the "exit 0" and 
uncomment the ldapsearch call so it's actually a functional health check, the 
output of the ldapsearch ends up in the load balanced tcp traffic!

> erlacher@atrbg13 ~ % telnet ldap-ha-ft 389
>   
>   
>  ~
> Trying 131.159.255.33...
> Connected to ldap-ha-ft.
> Escape character is '^]'.
> version: 1
> 
> #
> # LDAPv3
> # base  with scope subtree
> # filter: (mail=!)
> # requesting: dn 
> #
> 
> # search result


I can, of course, fix this by redirecting the output of the ldapsearch command 
to /dev/null, but why the heck does this end up in the tcp traffic anyway? 
Needless to say, this completely clobbers the actual client connection and it 
gets closed immediately.

Best regards,
Luke

:D

[1] 
http://blog.haproxy.com/2013/09/16/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer/


haproxy -vv:
> erlacher@vmrbg80:~$ sudo haproxy -vv
> HA-Proxy version 1.6.5 2016/05/10
> Copyright 2000-2016 Willy Tarreau 
> 
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat 
> -Werror=format-security -D_FORTIFY_SOURCE=2
>   OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Encrypted password support via crypt(3): yes
> Built with zlib version : 1.2.8
> Compression algorithms supported : identity("identity"), deflate("deflate"), 
> raw-deflate("deflate"), gzip("gzip")
> Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
> Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports prefer-server-ciphers : yes
> Built with PCRE version : 8.31 2012-07-06
> PCRE library supports JIT : no (USE_PCRE_JIT not set)
> Built with Lua version : Lua 5.3.1
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
> IP_FREEBIND
> 
> Available polling systems :
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.


-- 

Mit freundlichen Gruessen,
Lukas Erlacher

--
Rechnerbetriebsgruppe der Fakultäten Mathematik und Informatik
Raum 00.05.042
Tel. 089-289-18258
erlac...@in.tum.de
Technische Universität München - Boltzmannstr. 3 - 85748 Garching



smime.p7s
Description: S/MIME Cryptographic Signature

RE: tcpdump and Haproxy SSL Offloading

[mlist]

> If you are not using DHE cyphers (but you should) then you can try ssldump. 
> In case of Diffie-Hellman though new encryption key is generated for each ssl 
> session so you are out of luck here.

I just read about ssldump and wireshark decryption mechanism… I use 
Diffie-Helmlman…

> Not that I'm aware of but you can try chaining a local proxy where you can 
> see the traffic in clear text before you send the traffic to the backend .

I think the best solution is haproxy to allow to see the unencrypted traffic as 
haproxy manage the SSL/TLS communication and termination process.
It was the less invasive and most conservative solution in regard to leave 
infrastructure and flow as the original as possible.
I know this is not simple as many levels are involved and probably It is not 
properly a haproxy task.

Can you explain more about local proxy solution ?

Roberto



From: Igor Cicimov [mailto:ig...@encompasscorporation.com]
Sent: venerdì 3 giugno 2016 03.44
To: mlist 
Cc: haproxy@formilux.org
Subject: Re: tcpdump and Haproxy SSL Offloading



On Fri, Jun 3, 2016 at 3:14 AM, mlist 
mailto:ml...@apsystems.it>> wrote:
Often I need to take tcpdump to analyze haproxy communication to clients and to 
backend servers.
As we use haproxy as SSL termination point (haproxy SSL ofloading), at low 
levels (so tcpdump level)
we see communication with client encrypted.

If you are not using DHE cyphers (but you should) then you can try ssldump. In 
case of Diffie-Hellman though new encryption key is generated for each ssl 
session so you are out of luck here.

There are simple solution so I can do a tcpdump having
unencrypted communication ? Has haproxy some mechanism ?

Not that I'm aware of but you can try chaining a local proxy where you can see 
the traffic in clear text before you send the traffic to the backend .


I have 3 haproxy LBs with 2 L4 LBs balancing on haproxy LBs so I want to avoid 
if possible to make more
complex infrastructure introducing some other intermediate proxy to do that, so 
I make the communication
path as simple and equal to normal request path as possible.

Roberto




--
Il messaggio e' stato analizzato alla ricerca di virus o
contenuti pericolosi da MailScanner, ed e'
risultato non infetto.

Re: What can and cannot be done using Lua

[Thierry FOURNIER]

On Thu, 2 Jun 2016 11:17:09 -0400
Louis Munro  wrote:

> Hi Thierry,
> --
> Louis Munro
> lmu...@inverse.ca  ::  www.inverse.ca 
> +1.514.447.4918 x125  :: +1 (866) 353-6153 x125
> Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence 
> (www.packetfence.org)
> 
> > On Jun 2, 2016, at 7:17 , Thierry FOURNIER  
> > wrote:
> > 
> > Hi, you can do that you describre. First, you can read the general
> > introduction:
> > 
> >   http://www.arpalert.org/haproxy-lua.html
> > 
> > For using redis, you can deal with cosocket, the API documentation
> > provides an example with one redis library:
> > 
> >   
> > http://www.arpalert.org/src/haproxy-lua-api/1.6/index.html#external-lua-libraries
> > 
> 
> 
> I have actually already done that (although I may have missed some of the 
> finer points of the documentation).
> 
> I’ve got HAProxy/ Lua / Redis working at the moment and returning a redirect 
> based on a key in Redis.
> 
> > 
> > You can use an action (who can yield), store the result in a variable,
> > and use this variable for decisions.
> > 
> >   http://www.arpalert.org/src/haproxy-lua-api/1.6/index.html#TXN.set_var
> >   http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#7.3.2-var
> > 
> >   core.register_action("select", { "http-req" }, function(txn)
> >  -- deal with redis
> >  -- ...
> >  -- store the result:
> >  txn:set_var("req.action", "backend1")
> >   end)
> > 
> > In the haproxy configuration:
> > 
> > 
> > Don't hesitate to share your work (if it is possible). I try to list
> > article about haproxy and lua.
> 
> 
> Thank you very much.
> I will definitely write it up if I get it to do what I want.
> 
> The ultimate goal is to integrate this into PacketFence, an open source NAC 
> solution to which I contribute (https://github.com/inverse-inc/packetfence 
> ).
> So the code would end up on github for anyone to reuse.
> 
> I think you just gave me what I needed, which is the reassurance that I am 
> not wasting my time trying to do the impossible.


Ok, it is great.

BR,
Thierry


> 
> Best regards,
> --
> Louis Munro
> lmu...@inverse.ca  ::  www.inverse.ca 
> +1.514.447.4918 x125  :: +1 (866) 353-6153 x125
> Inverse inc. :: Leaders behind SOGo (www.sogo.nu) and PacketFence 
> (www.packetfence.org)
> 


-- 

RE: Haproxy limiting queue / connections

[Haim Ari]

Hello,

I managed to get better performance after doing a bit more configuration
Such as binding the haproxy processes to all cores in the same physical CPU
Also bonded all HTTP to these cores and TCP to the other cores on the second 
physical CPU


I was able to protect the backends (maxconn) now it is up to me to optimize and 
find the magic number to utilize exactly 65% of each backend cpu.
(more than that will cause latency)

As far as I can see in the stats page the total haproxy connections increases 
when the backends hit maxconn limit
And I guess this is what I should expect.

After tuning the global ulimit -n and maxconn to prevent the haproxy itself 
from crashing everything looks better.
(no haproxy crashes) 

I will update (with configuration details) if I'll see this issue again, but I 
think this is the right direction.

Thank you for the info.

Haim

-Original Message-
From: Lukas Tribus [mailto:lu...@gmx.net] 
Sent: Friday, June 3, 2016 12:45 PM
To: Haim Ari ; haproxy@formilux.org
Subject: Re: Haproxy limiting queue / connections

Hi Ari,

> If the backend servers are responding slowly for some reason, all 
> haproxy nodes will crash.
>
> i configured queue to 1 and set up reasonable server and client 
> timeouts on haproxy
>
>
> however if for some reason connections builds up on haproxy  crashes 
> and the machine becomes unresponsive (all haproxy nodes)
>
>
> I don't see any kernel errors.
>
>
> 1. how can i debug this ?
>
> 2. what is the best practice to protect haproxy from such situation ?
>


So there is nothing in the logs, nothing in dmesg? No OOM events, no crashes?

Then how do you know haproxy is crashing, if there is not a single log line 
about it? The node becomes completely unresponsive, like not even ping and ssh 
works? Then it is likely not a haproxy problem (but do provide "top" output 
anyway). Sounds more like a stateful firewall in front of haproxy with 
exhausted resources.

Also, how do you know connections are building up on haproxy? Can you see that 
on the stat interface? If yes, how does that look like exactly?


Please share the haproxy configuration and check your stateful firewalls 
for resource exhaustion.



Regards,

Lukas

Re: Haproxy limiting queue / connections

[Lukas Tribus]

Hi Ari,

If the backend servers are responding slowly for some reason, all 
haproxy nodes will crash.


i configured queue to 1 and set up reasonable server and client 
timeouts on haproxy



however if for some reason connections builds up on haproxy  crashes 
and the machine becomes unresponsive (all haproxy nodes)



I don't see any kernel errors.


1. how can i debug this ?

2. what is the best practice to protect haproxy from such situation ?




So there is nothing in the logs, nothing in dmesg? No OOM events, no 
crashes?


Then how do you know haproxy is crashing, if there is not a single log 
line about it? The node becomes completely unresponsive, like not even 
ping and ssh works? Then it is likely not a haproxy problem (but do 
provide "top" output anyway). Sounds more like a stateful firewall in 
front of haproxy with exhausted resources.


Also, how do you know connections are building up on haproxy? Can you 
see that on the stat interface? If yes, how does that look like exactly?



Please share the haproxy configuration and check your stateful firewalls 
for resource exhaustion.




Regards,

Lukas

ACL et frontend : comportement aléatoire / haproxy 1.5.18-1ppp1

[Kevin Maziere]

Bonjour

Je rencontre un problème d'acl et de redirection vers  backend que je ne
m'explique pas.
J'ai un frontend qui envoi par default sur un backend, et en fonction d'une
acl sur le hostname il envoi sur un autre backend.
Et bien quand je fais un curl/wget/chome/firefox sur ce frontend avec le
hostname qui match l'acl, j'arrive de manière aléatoire sur le
default_backend ou sur le backend spécifique pour ce hostname.

Rien dans les logs haproxy, un coup il me renvoi sur le default, un coup
sur celui attendu, pas de message d'erreur ou autre.

Si j’enlève le default backend, j'arrive à 100% sur le bon backend... je ne
comprends pas.

Si quelqu'un à une idée ?

Kévin

 haproxy  1.5.18-1ppa1~trusty  amd64


Voici le morceau de configuration en question :

 global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
maxconn x
#debug
#quiet
#spread-checks
user haproxy
group haproxy
defaults
log global
modehttp
#option  dontlognull
maxconn 
timeout server  xxm
timeout connect xxm
timeout client  xxm
option redispatch
retries 5
option  httplog
option forwardfor
timeout http-keep-alive xm
timeout http-request xm


frontend 10.0.01-80
bind 10.0.0.1:80
reqadd X-Forwarded-Proto:\ http
option http-server-close

acl host_beg_ttfr  hdr_beg(Host) tt-fra29-2-france
use_backend tt-france-fra29-2 if host_beg_ttfr

default_backend ipv4-fr

backend tt-france-fra29-2
reqirep ^Host:\  tt-france-fra29-2-france.subd.fr.mondomainamoi.fr
Host:\ fra29-2-fra.md.bbb.loca
server labas 192.168.21.5:80

backend ipv4-fr
balance roundrobin
option httpchk GET /
server fr-icietla 192.168.22.4:8080 weight 1 check inter 5000 rise 2
fall 5

9-12th June Guangzhou Lighting Fair Invitation

[jennifer]

Dear Customer,
This is Jennifer from Seamount Technology SZ Co.,Ltd.
Thanks for your support in the past.
Our company will attend Guangzhou international Lighting Fair on 9-12th 
June.And we will show some new products.If you will come to Guangzhou,welcome 
to our booth.

Our booth No. 1.1 Hall B43

Best regards
Jennifer
Tel:86-13823578467
Email:jenni...@seamount.cn

亲爱的客户，
你好！
我是新茂科技深圳有限公司的罗小姐。感谢大家一直以来对我工作的支持。
6月9-12日，我们公司将会去参加广州国际照明展，此次我们将会展出一些新品，如果你回去广州，欢迎到我们的展位上去看看。
我们的展位号 1.1馆B43

致谢！

罗小姐
电话:86-13823578467
邮箱:jenni...@seamount.cn