Re: Kernel NAT issues

[Nathan Aherne]

I am not exactly sure how to draw the setup so it doesn’t confuse the 
situation. The setup is extremely simple (I am not running vimage), jails 
running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. 
The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy 
(nginx) and forwards traffic to each HTTP App jail based on the URL it 
receives. The jails with public IPs are things like database jails which cannot 
be proxied by the Host.

I can happily communicate with any jail from my laptop (externally) but when I 
want one jail to communicate with another jail (for example an App Jail 
communicating with the database jail) the traffic shows as backwards 
(destination:port -> source:port) in the IPFW logs (tshark shows the traffic 
correctly source:port -> destination:port). The jail to jail traffic tries to 
go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs 
of an App jail (10.0.0.25) communicating with the database jail 
(aaa.bbb.ccc.ddd)

IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is 
labelled UNKNOWN even if it is KNOWN traffic)

Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1

tshark output (loopback and wan interface capture for port 5432)

Capturing on 'Loopback' and 'bce0'
  1   0.0010.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 
Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0
  2   3.01390510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142888539 TSecr=0
  3   6.24165810.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142891767 TSecr=0
  4   9.45151610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142894976 TSecr=0
  5  12.65465610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142898180 TSecr=0
  6  15.86390010.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142901389 TSecr=0
  7  22.07665510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142907602 TSecr=0


> If so, what sort of routing is setup on both host and jails?

Routing is what would be added by default (whatever the host system adds when 
adding an IP), there is no custom routing. I have wondered if I need to modify 
the routing table to get this to work. 

Below is the output of netstat -rn

www.xxx.yy.zzz is the gateway address
eee.fff.gg.hhh is the database jail public IP
aaa.bbb.cc.ddd is the public IP for NAT
lll.mmm.nn.ooo is the Hosts public IP


Routing tables

Internet:
DestinationGatewayFlags  Netif Expire
defaultwww.xxx.yy.zzz UGSbce0
10.0.0.1   link#6 UH  lo1
10.0.0.2   link#6 UH  lo1
10.0.0.3   link#6 UH  lo1
10.0.0.4   link#6 UH  lo1
10.0.0.5   link#6 UH  lo1
10.0.0.6   link#6 UH  lo1
10.0.0.7   link#6 UH  lo1
10.0.0.8   link#6 UH  lo1
10.0.0.9   link#6 UH  lo1
10.0.0.10  link#6 UH  lo1
10.0.0.11  link#6 UH  lo1
10.0.0.12  link#6 UH  lo1
10.0.0.13  link#6 UH  lo1
10.0.0.14  link#6 UH  lo1
10.0.0.15  link#6 UH  lo1
10.0.0.16  link#6 UH  lo1
10.0.0.17  link#6 UH  lo1
10.0.0.18  link#6 UH  lo1
10.0.0.19  link#6 UH  lo1
10.0.0.20  link#6 UH  lo1
10.0.0.21  link#6 UH  lo1
10.0.0.22  link#6 UH  lo1

Re: Kernel NAT issues

[Nathan Aherne]

I had a bit of a think about how to describe what I am trying to achieve.

I am treating each jail likes its own little "virtual machine”. The jail 
provides certain services, using things like nginx or nodejs, php-fpm, mysql or 
postgresql. The jails can control connections to themselves by configuring the 
firewall ports that are opened on the IP their IP  (10.0.0.0/16 or a public 
IP). I know the jails have no firewall of their own, the firewall is configured 
from the host.

I want each jail or “virtual machine” to be able to communicate with one 
another and the wider internet. When a jail does a DNS query for another App 
jail, it may get a public IP on its own Host (or it may get another host) and 
it has no issues being able to communicate with another jail on the same host.

At the moment all of the above is working perfectly except for jail to jail 
communication on the same host (when the communication is not directly between 
10.0.0.0/16 IP addresses).

Regards,

Nathan

> On 21 Nov 2015, at 9:12 am, Nathan Aherne  wrote:
> 
> I am not exactly sure how to draw the setup so it doesn’t confuse the 
> situation. The setup is extremely simple (I am not running vimage), jails 
> running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. 
> The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy 
> (nginx) and forwards traffic to each HTTP App jail based on the URL it 
> receives. The jails with public IPs are things like database jails which 
> cannot be proxied by the Host.
> 
> I can happily communicate with any jail from my laptop (externally) but when 
> I want one jail to communicate with another jail (for example an App Jail 
> communicating with the database jail) the traffic shows as backwards 
> (destination:port -> source:port) in the IPFW logs (tshark shows the traffic 
> correctly source:port -> destination:port). The jail to jail traffic tries to 
> go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs 
> of an App jail (10.0.0.25) communicating with the database jail 
> (aaa.bbb.ccc.ddd)
> 
> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is 
> labelled UNKNOWN even if it is KNOWN traffic)
> 
> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> 
> tshark output (loopback and wan interface capture for port 5432)
> 
> Capturing on 'Loopback' and 'bce0'
>   1   0.0010.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 
> Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0
>   2   3.01390510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142888539 TSecr=0
>   3   6.24165810.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142891767 TSecr=0
>   4   9.45151610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142894976 TSecr=0
>   5  12.65465610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142898180 TSecr=0
>   6  15.86390010.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142901389 TSecr=0
>   7  22.07665510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142907602 TSecr=0
> 
> 
>> If so, what sort of routing is setup on both host and jails?
> 
> Routing is what would be added by default (whatever the host system adds when 
> adding an IP), there is no custom routing. I have wondered if I need to 
> modify the routing table to get this to work. 
> 
> Below is the output of netstat -rn
> 
> www.xxx.yy .zzz is the gateway address
> eee.fff.gg.hhh is the database jail public IP
> aaa.bbb.cc.ddd is the public IP for NAT
> lll.mmm.nn.ooo is the Hosts public IP
> 
> 
> Routing tables
> 
> Internet:
> DestinationGatewayFlags