TProxy debugging

[Eduard Martinescu]

I am trying to get HAProxy up and running with TPROXY support...I've built
1.5-dev18 with the proper flags, and I'm running on a kernel with the
correct support.

My config is something like this:

global
log 127.0.0.1 local0 notice
maxconn 200
daemon

defaults
option tcplog
log global
modehttp
timeout client  5000ms
timeout connect 5ms
timeout server  5ms
option contstats

listen www
bind :80
mode tcp
maxconn 5
source 10.252.168.252 usesrc clientip
server 10.244.46.81
  option redispatch


If I comment out the 'source 10.252.168.252 usesrc clientip' line out,
everything works greatbut the server on the far end receives the IP of
the haproxy node.

If I leave that 'source' line in, I don't seem to get any responses.
 Turning on some packet traces I see the following

*NOTE: my client is 74.112.38.136



1832.467324 74.112.38.136 -> 10.252.168.252 TCP 52916 > https [SYN] Seq=0
Win=65535 Len=0 MSS=1460 WS=4 TSV=1260124448 TSER=0
1832.467385 10.252.168.252 -> 74.112.38.136 TCP https > 52916 [SYN, ACK]
Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 TSV=21847417 TSER=1260124448 WS=7
1832.688956 74.112.38.136 -> 10.252.168.252 TCP 52916 > https [ACK] Seq=1
Ack=1 Win=131760 Len=0 TSV=1260124633 TSER=21847417
1832.689008 74.112.38.136 -> 10.252.168.252 SSL Client Hello
1832.689027 10.252.168.252 -> 74.112.38.136 TCP https > 52916 [ACK] Seq=1
Ack=248 Win=15616 Len=0 TSV=21847472 TSER=1260124633
1832.689418 74.112.38.136 -> 10.244.46.81 TCP 49431 > https [SYN] Seq=0
Win=14600 Len=0 MSS=1460 TSV=21847472 TSER=0 WS=7
1833.686741 74.112.38.136 -> 10.244.46.81 TCP 49431 > https [SYN] Seq=0
Win=14600 Len=0 MSS=1460 TSV=21847722 TSER=0 WS=7
1835.690732 74.112.38.136 -> 10.244.46.81 TCP 49431 > https [SYN] Seq=0
Win=14600 Len=0 MSS=1460 TSV=21848223 TSER=0 WS=7
1839.694742 74.112.38.136 -> 10.244.46.81 TCP 49431 > https [SYN] Seq=0
Win=14600 Len=0 MSS=1460 TSV=21849224 TSER=0 WS=7



I really see no packets at all


So I double checked my configuration on the HAProxy node.  I ran the
following iptables commands:
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100


And made sure ip forwarding was on:
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
echo 1 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 1 > /proc/sys/net/ipv4/conf/eth0/send_redirects

And the firewall rules are:
$ sudo service iptables status
Table: mangle
Chain PREROUTING (policy ACCEPT)
num  target prot opt source   destination
1DIVERT tcp  --  0.0.0.0/00.0.0.0/0socket

Chain INPUT (policy ACCEPT)
num  target prot opt source   destination

Chain FORWARD (policy ACCEPT)
num  target prot opt source   destination

Chain OUTPUT (policy ACCEPT)
num  target prot opt source   destination

Chain POSTROUTING (policy ACCEPT)
num  target prot opt source   destination

Chain DIVERT (1 references)
num  target prot opt source   destination
1MARK   all  --  0.0.0.0/00.0.0.0/0MARK set
0x6f
2ACCEPT all  --  0.0.0.0/00.0.0.0/0



On the web ser
​ver node, I made sure that my haproxy node was the default router
Kernel IP routing table
Destination Gateway Genmask Flags   MSS Window  irtt
Iface
0.0.0.0 10.252.168.252  0.0.0.0 UG0 0  0
eth0
10.244.46.640.0.0.0 255.255.255.192 U 0 0  0
eth0
169.254.169.254 0.0.0.0 255.255.255.255 UH0 0  0
eth0

and

 ip route get 74.112.38.165
74.112.38.165 via 10.252.168.252 dev eth0  src 10.244.46.81
cache  ipid 0xb9ee

And finally, no firewall rules on my webserver:
$ sudo service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num  target prot opt source   destination

Chain FORWARD (policy ACCEPT)
num  target prot opt source   destination

Chain OUTPUT (policy ACCEPT)
num  target prot opt source   destination



I am now at an impasse and don't know what else to check.

​Does anyone have any pointers?

​Ed​


--
✉ Eduard Martinescu  | ✆ (585) 708-9685 | [image:
http://www.salsalabs.com]  - ignite action. fuel
change.

Re: TProxy debugging

[PiBa-NL]

Hi Eduard,

Im not sure about your iptables rules.. using pf/ipfw on FreeBSD myself...
But to me it looks like those last 4 [SYN] packets should have shown in 
a packetcapture on your webserver, unless they are re-routed elsewhere..


You could try a different IP in the source option :
  source 0.0.0.0 usesrc clientip

Could you also remove all special packet re-routing/divert rules from 
the haproxy box.? And check again if the webserver then does receive a 
SYN from the 'client-IP' and sends back a SYN-ACK to the HAProxy server?


It still wont work then because the HAProxy process wont actually 
receive the SYN-ACK but it should show up on the lan-interface of that 
machine.


Then the remaining issue is how to write the proper redirect rule for 
the 'return traffic' coming from the webserver and point it to the 
'local machine'..


As for the iptables, probably some other guy's can help better. But hope 
this helps in the 'debugging' a bit :).
Also i found it usefull to start haproxy with the -d -V parameters to 
show on-screen what happens (told me it couldnt bind to a nonlocal ip at 
first tries..).


Greets
PiBa-NL

Re: TProxy debugging

[Eduard Martinescu]

Thank you for your help.

It turns out the problem was beyond my control and in the network/routing
layer.  There were some rules in place preventing address spoofing.

I'm all set now and things are working correctly.

Ed

--
✉ Eduard Martinescu  | ✆ (585) 708-9685 | [image:
http://www.salsalabs.com]  - ignite action. fuel
change.


On Tue, May 7, 2013 at 4:31 PM, PiBa-NL  wrote:

>  Hi Eduard,
>
> Im not sure about your iptables rules.. using pf/ipfw on FreeBSD myself...
> But to me it looks like those last 4 [SYN] packets should have shown in a
> packetcapture on your webserver, unless they are re-routed elsewhere..
>
> You could try a different IP in the source option :
>   source 0.0.0.0 usesrc clientip
>
> Could you also remove all special packet re-routing/divert rules from the
> haproxy box.? And check again if the webserver then does receive a SYN from
> the 'client-IP' and sends back a SYN-ACK to the HAProxy server?
>
> It still wont work then because the HAProxy process wont actually receive
> the SYN-ACK but it should show up on the lan-interface of that machine.
>
> Then the remaining issue is how to write the proper redirect rule for the
> 'return traffic' coming from the webserver and point it to the 'local
> machine'..
>
> As for the iptables, probably some other guy's can help better. But hope
> this helps in the 'debugging' a bit :).
> Also i found it usefull to start haproxy with the -d -V parameters to show
> on-screen what happens (told me it couldnt bind to a nonlocal ip at first
> tries..).
>
> Greets
> PiBa-NL
>