Re: Kernel NAT issues

[Dewayne Geraghty]

Nathan, I've gone the same way that you have, ie bunch of jails that are
individually providing services& kernel Nat.  It takes careful planning and
the knowledge that the default route will be the first IP in your jail.conf
list for each jail.

Getting jails to play nice means fiddling around with all interfaces. If
you can take ipfw out of the equation until you can see tcpdump traffic
doing what you want; the challenge hasn't been ipfw in my experience. (&
yes initially I've had three tcpdumps going at once too; along with old
friends: raw ip & ping )
Enjoy the fun of getting it to work, it's well worth the effort. (And be
thankful that you aren't using pf, another level of complexity but suits my
needs perfectly) ;)
Dewayne
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Nathan Aherne]

Hi Julian,

Thank you for replying. I was completely off grid for a while and only got back 
on it today.

I thought that Vimage was probably the way to achieve what I want. The main 
reason I was staying away from Vimage was the reported bugs with it, another 
reason was the extra overhead. I would like to be able to shutdown jails quite 
regularly so was worried the kernel panic bug or memory leak bug might be a 
problem here. Is there any version of Vimage/FreeBSD which is stable?

Regards,

Nathan

> On 23 Nov 2015, at 5:02 pm, Julian Elischer  wrote:
> 
> On 21/11/2015 10:06 AM, Nathan Aherne wrote:
>> I had a bit of a think about how to describe what I am trying to achieve.
>> 
>> I am treating each jail likes its own little "virtual machine”. The jail 
>> provides certain services, using things like nginx or nodejs, php-fpm, mysql 
>> or postgresql. The jails can control connections to themselves by 
>> configuring the firewall ports that are opened on the IP their IP  
>> (10.0.0.0/16 or a public IP). I know the jails have no firewall of their 
>> own, the firewall is configured from the host.
>> 
>> I want each jail or “virtual machine” to be able to communicate with one 
>> another and the wider internet. When a jail does a DNS query for another App 
>> jail, it may get a public IP on its own Host (or it may get another host) 
>> and it has no issues being able to communicate with another jail on the same 
>> host.
>> 
>> At the moment all of the above is working perfectly except for jail to jail 
>> communication on the same host (when the communication is not directly 
>> between 10.0.0.0/16 IP addresses).
> this is pretty much exactly when vimage/vnet jails could be used to great 
> affect.
> Is there a reason you are not doing that?  Each jail has it's own routing 
> tables, addresses and (virtual) interfaces.
> 
> here's how I'd do it with vimage
> 
>   +--+
>   +---+  | servers
>   |   +--+
>   |
>   |   +--+
>   |  ++  |
>   |  |+--+
>   |  |
> ++ +--+--++
> | iface  | | bridge   |
> |+-+  |
> ++ ++-+
> |
> |
> |
> |
> |
> |
> ++-+
> |  |
> |  |
> |   NAT jail router|
> |  |
> |  |
> +---+++---++
>|||   |
> +--+--+  +--+--+  +--+--+ +--+--+
> | |  | |  | | | |
> | |  | |  | | | |
> | |  | |  | | | |jails
> | |  | |  | | | |
> +-+  +-+  +-+ +-+
> 
> 
> 
> however the hairpin idea might still be useful even in that scenario if they 
> don't know about each other's 'local' addresses, but do NAT'd machines need 
> to talk to each other by externeal addresses?
> 
> i Nathan
>>> On 21 Nov 2015, at 9:12 am, Nathan Aherne  wrote:
>>> 
>>> I am not exactly sure how to draw the setup so it doesn’t confuse the 
>>> situation. The setup is extremely simple (I am not running vimage), jails 
>>> running on the 10.0.0.0/16 (cloned lo1 interface) network or with public 
>>> IPs. The jails with private IPs are the HTTP app jails. The Host runs a 
>>> HTTP Proxy (nginx) and forwards traffic to each HTTP App jail based on the 
>>> URL it receives. The jails with public IPs are things like database jails 
>>> which cannot be proxied by the Host.
>>> 
>>> I can happily communicate with any jail from my laptop (externally) but 
>>> when I want one jail to communicate with another jail (for example an App 
>>> Jail communicating with the database jail) the traffic shows as backwards 
>>> (destination:port -> source:port) in the IPFW logs (tshark shows the 
>>> traffic correctly source:port -> destination:port). The jail to jail 
>>> traffic tries to go over the lo1 interface (backwards) and is blocked. 
>>> Below is some IPFW logs of an App jail (10.0.0.25) communicating with the 
>>> database jail (aaa.bbb.ccc.ddd)
>>> 
>>> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything 
>>> is labelled UNKNOWN even if it is KNOWN traffic)
>>> 
>>> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
>>> 10.0.0.25:42957 out via lo1
>>> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP 

Re: Kernel NAT issues

[Julian Elischer]

On 21/11/2015 10:06 AM, Nathan Aherne wrote:

I had a bit of a think about how to describe what I am trying to achieve.

I am treating each jail likes its own little "virtual machine”. The jail 
provides certain services, using things like nginx or nodejs, php-fpm, mysql or 
postgresql. The jails can control connections to themselves by configuring the 
firewall ports that are opened on the IP their IP  (10.0.0.0/16 or a public IP). I 
know the jails have no firewall of their own, the firewall is configured from the 
host.

I want each jail or “virtual machine” to be able to communicate with one 
another and the wider internet. When a jail does a DNS query for another App 
jail, it may get a public IP on its own Host (or it may get another host) and 
it has no issues being able to communicate with another jail on the same host.

At the moment all of the above is working perfectly except for jail to jail 
communication on the same host (when the communication is not directly between 
10.0.0.0/16 IP addresses).
this is pretty much exactly when vimage/vnet jails could be used to 
great affect.
Is there a reason you are not doing that?  Each jail has it's own 
routing tables, addresses and (virtual) interfaces.


here's how I'd do it with vimage

   +--+
   +---+  | servers
   |   +--+
   |
   |   +--+
   |  ++  |
   |  |+--+
   |  |
 ++ +--+--++
 | iface  | | bridge   |
 |+-+  |
 ++ ++-+
 |
 |
 |
 |
 |
 |
++-+
|  |
|  |
|   NAT jail router|
|  |
|  |
+---+++---++
|||   |
 +--+--+  +--+--+  +--+--+ +--+--+
 | |  | |  | | | |
 | |  | |  | | | |
 | |  | |  | | | |jails
 | |  | |  | | | |
 +-+  +-+  +-+ +-+



however the hairpin idea might still be useful even in that scenario 
if they don't know about each other's 'local' addresses, but do NAT'd 
machines need to talk to each other by externeal addresses?


i Nathan

On 21 Nov 2015, at 9:12 am, Nathan Aherne  wrote:

I am not exactly sure how to draw the setup so it doesn’t confuse the 
situation. The setup is extremely simple (I am not running vimage), jails 
running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. 
The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy 
(nginx) and forwards traffic to each HTTP App jail based on the URL it 
receives. The jails with public IPs are things like database jails which cannot 
be proxied by the Host.

I can happily communicate with any jail from my laptop (externally) but when I want 
one jail to communicate with another jail (for example an App Jail communicating with 
the database jail) the traffic shows as backwards (destination:port -> 
source:port) in the IPFW logs (tshark shows the traffic correctly source:port -> 
destination:port). The jail to jail traffic tries to go over the lo1 interface 
(backwards) and is blocked. Below is some IPFW logs of an App jail (10.0.0.25) 
communicating with the database jail (aaa.bbb.ccc.ddd)

IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is 
labelled UNKNOWN even if it is KNOWN traffic)

Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1

tshark output (loopback and wan interface capture for port 5432)

Capturing on 'Loopback' and 'bce0'
   1   0.0010.0.0.25 -> eee.fff.gg.hhh TCP 64 

Re: Kernel NAT issues

[Julian Elischer]

I note,  no answers yet..
I plan on spending some time tomorrow trying to understand this issue.

just letting you know..



On 21/11/2015 10:06 AM, Nathan Aherne wrote:
I had a bit of a think about how to describe what I am trying to 
achieve.


I am treating each jail likes its own little "virtual machine”. The 
jail provides certain services, using things like nginx or nodejs, 
php-fpm, mysql or postgresql. The jails can control connections to 
themselves by configuring the firewall ports that are opened on the 
IP their IP  (10.0.0.0/16 or a public IP). I know the jails have no 
firewall of their own, the firewall is configured from the host.


I want each jail or “virtual machine” to be able to communicate with 
one another and the wider internet. When a jail does a DNS query for 
another App jail, it may get a public IP on its own Host (or it may 
get another host) and it has no issues being able to communicate 
with another jail on the same host.


At the moment all of the above is working perfectly except for jail 
to jail communication on the same host (when the communication is 
not directly between 10.0.0.0/16 IP addresses).


Regards,

Nathan

On 21 Nov 2015, at 9:12 am, Nathan Aherne > wrote:


I am not exactly sure how to draw the setup so it doesn’t confuse 
the situation. The setup is extremely simple (I am not running 
vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) 
network or with public IPs. The jails with private IPs are the HTTP 
app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic 
to each HTTP App jail based on the URL it receives. The jails with 
public IPs are things like database jails which cannot be proxied 
by the Host.


I can happily communicate with any jail from my laptop (externally) 
but when I want one jail to communicate with another jail (for 
example an App Jail communicating with the database jail) the 
traffic shows as backwards (destination:port -> source:port) in the 
IPFW logs (tshark shows the traffic correctly source:port -> 
destination:port). The jail to jail traffic tries to go over the 
lo1 interface (backwards) and is blocked. Below is some IPFW logs 
of an App jail (10.0.0.25) communicating with the database jail 
(aaa.bbb.ccc.ddd)


IPFW logs. The lines labelled UNKNOWN is the check-state rule 
(everything is labelled UNKNOWN even if it is KNOWN traffic)


Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP 
eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1


tshark output (loopback and wan interface capture for port 5432)

Capturing on 'Loopback' and 'bce0'
  1   0.00 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] 
Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 
TSecr=0
  2   3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP 
Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 
WS=64 SACK_PERM=1 TSval=142888539 TSecr=0
  3   6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP 
Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 
WS=64 SACK_PERM=1 TSval=142891767 TSecr=0
  4   9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP 
Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 
WS=64 SACK_PERM=1 TSval=142894976 TSecr=0
  5  12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP 
Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 
WS=64 SACK_PERM=1 TSval=142898180 TSecr=0
  6  15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP 
Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 
WS=64 SACK_PERM=1 TSval=142901389 TSecr=0
  7  22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP 
Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 
WS=64 SACK_PERM=1 TSval=142907602 TSecr=0




If so, what sort of routing is setup on both host and jails?


Routing is what would be added by default (whatever the host system 
adds when adding an IP), there is no custom routing. I have 
wondered if I need to modify the routing table to get this to work.


Below is the output of netstat -rn

www.xxx.yy .zzz is the gateway address
eee.fff.gg.hhh is the database jail public IP
aaa.bbb.cc.ddd is the public IP for NAT
lll.mmm.nn.ooo is the Hosts public IP


Routing tables

Internet:
Destination 

Re: Kernel NAT issues

[Nathan Aherne]

I am not exactly sure how to draw the setup so it doesn’t confuse the 
situation. The setup is extremely simple (I am not running vimage), jails 
running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. 
The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy 
(nginx) and forwards traffic to each HTTP App jail based on the URL it 
receives. The jails with public IPs are things like database jails which cannot 
be proxied by the Host.

I can happily communicate with any jail from my laptop (externally) but when I 
want one jail to communicate with another jail (for example an App Jail 
communicating with the database jail) the traffic shows as backwards 
(destination:port -> source:port) in the IPFW logs (tshark shows the traffic 
correctly source:port -> destination:port). The jail to jail traffic tries to 
go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs 
of an App jail (10.0.0.25) communicating with the database jail 
(aaa.bbb.ccc.ddd)

IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is 
labelled UNKNOWN even if it is KNOWN traffic)

Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
10.0.0.25:42957 out via lo1

tshark output (loopback and wan interface capture for port 5432)

Capturing on 'Loopback' and 'bce0'
  1   0.0010.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 
Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0
  2   3.01390510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142888539 TSecr=0
  3   6.24165810.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142891767 TSecr=0
  4   9.45151610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142894976 TSecr=0
  5  12.65465610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142898180 TSecr=0
  6  15.86390010.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142901389 TSecr=0
  7  22.07665510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
TSval=142907602 TSecr=0


> If so, what sort of routing is setup on both host and jails?

Routing is what would be added by default (whatever the host system adds when 
adding an IP), there is no custom routing. I have wondered if I need to modify 
the routing table to get this to work. 

Below is the output of netstat -rn

www.xxx.yy.zzz is the gateway address
eee.fff.gg.hhh is the database jail public IP
aaa.bbb.cc.ddd is the public IP for NAT
lll.mmm.nn.ooo is the Hosts public IP


Routing tables

Internet:
DestinationGatewayFlags  Netif Expire
defaultwww.xxx.yy.zzz UGSbce0
10.0.0.1   link#6 UH  lo1
10.0.0.2   link#6 UH  lo1
10.0.0.3   link#6 UH  lo1
10.0.0.4   link#6 UH  lo1
10.0.0.5   link#6 UH  lo1
10.0.0.6   link#6 UH  lo1
10.0.0.7   link#6 UH  lo1
10.0.0.8   link#6 UH  lo1
10.0.0.9   link#6 UH  lo1
10.0.0.10  link#6 UH  lo1
10.0.0.11  link#6 UH  lo1
10.0.0.12  link#6 UH  lo1
10.0.0.13  link#6 UH  lo1
10.0.0.14  link#6 UH  lo1
10.0.0.15  link#6 UH  lo1
10.0.0.16  link#6 UH  lo1
10.0.0.17  link#6 UH  lo1
10.0.0.18  link#6 UH  lo1
10.0.0.19  link#6 UH  lo1
10.0.0.20  link#6 UH  lo1
10.0.0.21  link#6 UH  lo1
10.0.0.22  link#6 UH  lo1

Re: Kernel NAT issues

[Nathan Aherne]

I had a bit of a think about how to describe what I am trying to achieve.

I am treating each jail likes its own little "virtual machine”. The jail 
provides certain services, using things like nginx or nodejs, php-fpm, mysql or 
postgresql. The jails can control connections to themselves by configuring the 
firewall ports that are opened on the IP their IP  (10.0.0.0/16 or a public 
IP). I know the jails have no firewall of their own, the firewall is configured 
from the host.

I want each jail or “virtual machine” to be able to communicate with one 
another and the wider internet. When a jail does a DNS query for another App 
jail, it may get a public IP on its own Host (or it may get another host) and 
it has no issues being able to communicate with another jail on the same host.

At the moment all of the above is working perfectly except for jail to jail 
communication on the same host (when the communication is not directly between 
10.0.0.0/16 IP addresses).

Regards,

Nathan

> On 21 Nov 2015, at 9:12 am, Nathan Aherne  wrote:
> 
> I am not exactly sure how to draw the setup so it doesn’t confuse the 
> situation. The setup is extremely simple (I am not running vimage), jails 
> running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. 
> The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy 
> (nginx) and forwards traffic to each HTTP App jail based on the URL it 
> receives. The jails with public IPs are things like database jails which 
> cannot be proxied by the Host.
> 
> I can happily communicate with any jail from my laptop (externally) but when 
> I want one jail to communicate with another jail (for example an App Jail 
> communicating with the database jail) the traffic shows as backwards 
> (destination:port -> source:port) in the IPFW logs (tshark shows the traffic 
> correctly source:port -> destination:port). The jail to jail traffic tries to 
> go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs 
> of an App jail (10.0.0.25) communicating with the database jail 
> (aaa.bbb.ccc.ddd)
> 
> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is 
> labelled UNKNOWN even if it is KNOWN traffic)
> 
> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> 
> tshark output (loopback and wan interface capture for port 5432)
> 
> Capturing on 'Loopback' and 'bce0'
>   1   0.0010.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 
> Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0
>   2   3.01390510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142888539 TSecr=0
>   3   6.24165810.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142891767 TSecr=0
>   4   9.45151610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142894976 TSecr=0
>   5  12.65465610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142898180 TSecr=0
>   6  15.86390010.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142901389 TSecr=0
>   7  22.07665510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142907602 TSecr=0
> 
> 
>> If so, what sort of routing is setup on both host and jails?
> 
> Routing is what would be added by default (whatever the host system adds when 
> adding an IP), there is no custom routing. I have wondered if I need to 
> modify the routing table to get this to work. 
> 
> Below is the output of netstat -rn
> 
> www.xxx.yy .zzz is the gateway address
> eee.fff.gg.hhh is the database jail public IP
> aaa.bbb.cc.ddd is the public IP for NAT
> lll.mmm.nn.ooo is the Hosts public IP
> 
> 
> Routing tables
> 
> Internet:
> DestinationGatewayFlags  

Re: Kernel NAT issues

[Ian Smith]

On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote:
 > On 11/18/15 8:40 AM, Nathan Aherne wrote:
 > > For some reason hairpin (loopback nat or nat reflection) does not seem to
 > > be working, which is why I chose IPFW in the first place.

 > it would be good to see a diagram of what this actually means.

Anything like ?
http://kb.juniper.net/InfoCenter/index?page=content=KB24639=search

Was this so one jail can only access service/s provided by other jail/s, 
both/all with internal NAT'd addresses, by using only the public address 
and port of the 'router', which IIRC this is a single system with jails?

If so, what sort of routing is setup on both host and jails?

(blindfolded, no idea where I've pinned the donkey's tail :)

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Julian Elischer]

On 11/18/15 8:40 AM, Nathan Aherne wrote:

For some reason hairpin (loopback nat or nat reflection) does not seem to be 
working, which is why I chose IPFW in the first place.

it would be good to see a diagram of what this actually means.



___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Nathan Aherne]

Hi Everyone,

I think I have worked this out and have a working Stateful IPFW NAT (its worked 
fine for a few weeks) rule set. Hopefully this saves someone else a few weeks 
of their lives.

For some reason hairpin (loopback nat or nat reflection) does not seem to be 
working, which is why I chose IPFW in the first place.

I am actually getting some really weird log results from IPFW (for inter jail 
communication ). IPFW logs show traffic flowing in the opposite direction to 
what tshark shows. Tshark shows the traffic in the direction I would expect. 
IPFW is blocking the traffic because it is seeing it as going the wrong way. 
Anyone have an idea how I can solve this issue, I would super appreciate any 
pointers!

To clarify inter jail communication - not using the local jail IPs but DNS - 
they all resolve to the same public IP. I cannot use split DNS.

###
#!/bin/sh

# Configuration
wif="bce0" # WAN interface
jif="lo1" # Jail interface
jsn="10.0.0.0/16" # Jail subnet
plip="10.0.0.1" # Proxy IP
hwip=“aaa.bbb.ccc.ddd" # Host WAN IP
nwip=“www.xxx.yyy.zzz" # NAT WAN IP

# Script variables
cmd="ipfw -q add"
ks="keep-state"
sks="setup keep-state"

# Flush all rules
ipfw -q -f flush

# Enable NAT
ipfw nat 1 config ip $nwip log

# Allow all loopback traffic
$cmd 5 allow ip from any to any via lo0

# Block any traffic not for Host
$cmd 6 deny ip from any to not me in via $wif

$cmd 100 nat 1 log ip from any to $nwip recv $wif
$cmd 101 check-state log

# Host
$cmd 110 allow icmp from any to $hwip in via $wif $ks
$cmd 111 allow tcp from any to $hwip 65222 in via $wif $sks
$cmd 112 allow icmp from $hwip to any out via $wif $ks
$cmd 113 allow tcp from $hwip to any 53, 80, 443, 22, 65222 out via $wif $sks
$cmd 114 allow udp from $hwip to any 53, 123 out via $wif $ks

# Incoming NAT
$cmd 120 skipto 65510 log tcp from any to $jsn recv $wif $sks
$cmd 121 skipto 65510 log udp from any to $jsn recv $wif $ks
# Outgoing NAT
$cmd 122 skipto 65510 log tcp from $jsn to not $jsn xmit $wif $sks
$cmd 123 skipto 65510 log udp from $jsn to not $jsn xmit $wif $ks

# JAILS RULES 200-65000

# Block any other traffic
$cmd 65501 deny log ip from any to any

# Outgoing NAT
$cmd 65510 nat 1 log ip from $jsn to any xmit $wif
$cmd 65511 allow log ip from $nwip to any xmit $wif
# Block any other traffic
#$cmd 65519 deny log ip from any to any

# Incoming NAT
$cmd 65520 allow log ip from any to $jsn recv $wif
# Block any other traffic
$cmd 65522 deny log ip from any to any

###

Regards,

Nathan

> On 21 Oct 2015, at 9:02 am, Nathan Aherne  wrote:
> 
> Hi Ian,
> 
> Thank you very much for your response! Sorry about the late response, I have 
> been offline for a few days.
> 
> I think I may have worked this issue out. I am bringing up a bunch of Jails 
> today to test my firewall rules in the hopes that I have corrected my 
> problem. I will reply back either way.
> 
> Regards,
> 
> Nathan
> 
>> On 15 Oct 2015, at 12:51 am, Ian Smith  wrote:
>> 
>> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
>>> Hi Ian,
>>> 
>>> Thank you for your response.
>>> 
>>> I didnÿÿt post my ruleset because I should be able to fix the issue 
>>> myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
>>> incorrect.
>>> 
>>> I have now included my ruleset below (as well as my initial email).
>> 
>> Hi Nathan,
>> 
>> I was really hoping someone who knows more about stateful rule handling 
>> (and jail networking) might have a go at this.  Oh well I'll try, but 
>> I'm a lousy mindreader, and really don't know which of the below 
>> constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
>> and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?
>> 
>>> # Enable NAT
>>> ipfw nat 1 config ip $jip same_ports log
>> 
>> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
>> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
>> to you, and so traffic to it won't be subject to NAT .. correct?  But 
>> the WWW... address and all 10.0/16 addresses are jails, not any separate 
>> boxes you gateway for, right?  Just the one external interface, right?
>> 
>>> 5 allow ip from any to any via lo0
>>> 6 deny ip from any to not me in via bce0
>>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
>>> 00101 check-state
>> 
>> Ok, inbound from WAN is nat'd and existing stateful flows followed by 
>> executing the rule that originally kept state.  Where this is a skipto, 
>> skipto will be performed.  But where it's a nat rule, I've no idea .. 
>> see below, but you really don't want to add keep-state (again) there.
>> 
>>> 00110 allow icmp from 

Re: Kernel NAT issues

[Nathan Aherne]

Hi Ian,

Thank you very much for your response! Sorry about the late response, I have 
been offline for a few days.

I think I may have worked this issue out. I am bringing up a bunch of Jails 
today to test my firewall rules in the hopes that I have corrected my problem. 
I will reply back either way.

Regards,

Nathan

> On 15 Oct 2015, at 12:51 am, Ian Smith  wrote:
> 
> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
>> Hi Ian,
>> 
>> Thank you for your response.
>> 
>> I didnÿÿt post my ruleset because I should be able to fix the issue 
>> myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
>> incorrect.
>> 
>> I have now included my ruleset below (as well as my initial email).
> 
> Hi Nathan,
> 
> I was really hoping someone who knows more about stateful rule handling 
> (and jail networking) might have a go at this.  Oh well I'll try, but 
> I'm a lousy mindreader, and really don't know which of the below 
> constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
> and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?
> 
>> # Enable NAT
>> ipfw nat 1 config ip $jip same_ports log
> 
> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
> to you, and so traffic to it won't be subject to NAT .. correct?  But 
> the WWW... address and all 10.0/16 addresses are jails, not any separate 
> boxes you gateway for, right?  Just the one external interface, right?
> 
>> 5 allow ip from any to any via lo0
>> 6 deny ip from any to not me in via bce0
>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
>> 00101 check-state
> 
> Ok, inbound from WAN is nat'd and existing stateful flows followed by 
> executing the rule that originally kept state.  Where this is a skipto, 
> skipto will be performed.  But where it's a nat rule, I've no idea .. 
> see below, but you really don't want to add keep-state (again) there.
> 
>> 00110 allow icmp from any to WWW.XXX.YYY .ZZZ recv bce0 
>> keep-state
> 
> Hmm.  I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless 
> rule would make more sense especially for inbound ICMP.  But moving on ..
> 
>> 00111 allow tcp from any to WWW.XXX.YYY .ZZZ dst-port 
>> 65222 recv bce0 setup keep-state
> 
> Ok, but showting why plain text works better than HTML on lists :)
> 
>> 00112 allow icmp from WWW.XXX.YYY .ZZZ to any xmit bce0 
>> keep-state
>> 00113 allow tcp from WWW.XXX.YYY .ZZZ to any dst-port 
>> 53,80,443,22,65222 xmit bce0 setup keep-state
>> 00114 allow udp from WWW.XXX.YYY .ZZZ to any dst-port 
>> 53,123 xmit bce0 keep-state
> 
> Smells ok.
> 
>> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
>> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
> 
> Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's 
> got a problem .. see below.  These two are inbound traffic (recv) and as 
> is, skipping to 65501 will fall through two outbound rules to be denied.
> 
> Either allow them here directly, or likely better, skipto a separate
> target that then allows (or denies) them, if that's what you intended?
> 
>> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
>> setup keep-state
>> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
>> keep-state
> 
> Ok, this traffic does needs to be NAT'd on the way out.
> 
>> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup 
>> keep-state
>> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup 
>> keep-state
>> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
> 
> Not clear why these tcp ports are open inbound and outbound?  Presumably 
> this is jail-to-jail traffic?  Perhaps not relevant to your problem.
> 
>> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup 
>> keep-state
>> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup 
>> keep-state
>> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
>> 65500 deny log ip from any to any
> 
> Ok.
> 
>> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 
> This the target for outbound traffix, xmit bce0, so nat is appropriate.  
> Does jail-to-jail traffic travels via lo1?  Or what?
> 
> This won't do anything to inbound traffic, but that really shouldn't get 
> here except returns as the result of check-state - not from 120 & 121.
> 
> But keep-state is not ok, state was already set on the skipto.  I don't 
> know how this extra keep-state might behave - does anyone have an idea?
> 
> Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic.
> 
>> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
> 
> So, only remaining traffic is 

Re: Kernel NAT issues

[Ian Smith]

On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
 > Hi Ian,
 > 
 > Thank you for your response.
 > 
 > I didnÿÿt post my ruleset because I should be able to fix the issue 
 > myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
 > incorrect.
 > 
 > I have now included my ruleset below (as well as my initial email).

Hi Nathan,

I was really hoping someone who knows more about stateful rule handling 
(and jail networking) might have a go at this.  Oh well I'll try, but 
I'm a lousy mindreader, and really don't know which of the below 
constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?

 > # Enable NAT
 > ipfw nat 1 config ip $jip same_ports log

I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
to you, and so traffic to it won't be subject to NAT .. correct?  But 
the WWW... address and all 10.0/16 addresses are jails, not any separate 
boxes you gateway for, right?  Just the one external interface, right?

 > 5 allow ip from any to any via lo0
 > 6 deny ip from any to not me in via bce0
 > 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
 > 00101 check-state

Ok, inbound from WAN is nat'd and existing stateful flows followed by 
executing the rule that originally kept state.  Where this is a skipto, 
skipto will be performed.  But where it's a nat rule, I've no idea .. 
see below, but you really don't want to add keep-state (again) there.

 > 00110 allow icmp from any to WWW.XXX.YYY .ZZZ recv bce0 
 > keep-state

Hmm.  I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless 
rule would make more sense especially for inbound ICMP.  But moving on ..

 > 00111 allow tcp from any to WWW.XXX.YYY .ZZZ dst-port 
 > 65222 recv bce0 setup keep-state

Ok, but showting why plain text works better than HTML on lists :)

 > 00112 allow icmp from WWW.XXX.YYY .ZZZ to any xmit bce0 
 > keep-state
 > 00113 allow tcp from WWW.XXX.YYY .ZZZ to any dst-port 
 > 53,80,443,22,65222 xmit bce0 setup keep-state
 > 00114 allow udp from WWW.XXX.YYY .ZZZ to any dst-port 
 > 53,123 xmit bce0 keep-state

Smells ok.

 > 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
 > 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state

Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's 
got a problem .. see below.  These two are inbound traffic (recv) and as 
is, skipping to 65501 will fall through two outbound rules to be denied.

Either allow them here directly, or likely better, skipto a separate
target that then allows (or denies) them, if that's what you intended?

 > 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
 > setup keep-state
 > 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
 > keep-state

Ok, this traffic does needs to be NAT'd on the way out.

 > 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup 
 > keep-state
 > 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup 
 > keep-state
 > 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state

Not clear why these tcp ports are open inbound and outbound?  Presumably 
this is jail-to-jail traffic?  Perhaps not relevant to your problem.

 > 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup 
 > keep-state
 > 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup 
 > keep-state
 > 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
 > 65500 deny log ip from any to any

Ok.

 > 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state

This the target for outbound traffix, xmit bce0, so nat is appropriate.  
Does jail-to-jail traffic travels via lo1?  Or what?

This won't do anything to inbound traffic, but that really shouldn't get 
here except returns as the result of check-state - not from 120 & 121.

But keep-state is not ok, state was already set on the skipto.  I don't 
know how this extra keep-state might behave - does anyone have an idea?

Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic.

 > 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state

So, only remaining traffic is outbound from the host itself, and traffic 
that is to 10.0/16, but not from AAA... is to be dropped, correct?

I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp 
states .. myself, I'd go for separate rules for each eg tcp, udp, .. and 
I'd do it somewhere else than as a fall through from outbound nat rule, 
it's confusing here, to me anyway .. unless I've missed the reason?

 > 65534 deny log ip from any to any
 > 65535 deny ip from any to any


Ok, now for your demo of the problem from the later mail, 

Re: Kernel NAT issues

[Nathan Aherne]

Hi Ian,

Thank you for your response.

I didn’t post my ruleset because I should be able to fix the issue myself but I 
see now that my request to explain “how NAT works” was incorrect.

I have now included my ruleset below (as well as my initial email).

# Enable NAT
ipfw nat 1 config ip $jip same_ports log


5 allow ip from any to any via lo0
6 deny ip from any to not me in via bce0
00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
00101 check-state
00110 allow icmp from any to WWW.XXX.YYY .ZZZ recv bce0 
keep-state
00111 allow tcp from any to WWW.XXX.YYY .ZZZ dst-port 
65222 recv bce0 setup keep-state
00112 allow icmp from WWW.XXX.YYY .ZZZ to any xmit bce0 
keep-state
00113 allow tcp from WWW.XXX.YYY .ZZZ to any dst-port 
53,80,443,22,65222 xmit bce0 setup keep-state
00114 allow udp from WWW.XXX.YYY .ZZZ to any dst-port 
53,123 xmit bce0 keep-state
00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup 
keep-state
00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
keep-state
00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state
00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state
00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state
00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state
00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
65500 deny log ip from any to any
65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
65534 deny log ip from any to any
65535 deny ip from any to any

**
I sent through a question to this list a little while ago and have been trying 
to get IPFW NAT working since then. I have had some success but not the success 
I need, everything is working correctly except NAT rules for my particular use 
case. 

I have read every Google result on the first 50 pages when searching for “IPFW 
NAT” or “IPFW kernel NAT”. I would really appreciate it if someone could help 
me out.

My use case is as follows:

1. I need to use hairpin NAT - I am using Jails behind a http proxy and some 
jails need to be able to communicate with each other but only over the WAN IP. 
This is why I have not use PF.
2. Some jails need to be able to communicate with each other on the private 
interface (lo1)
3. IPFW is configured as default deny
4. Each jail has a list of allowed ports for incoming and outgoing connections, 
these are set on the jails private IP (10.0.0.0/16)
5. I am using a stateful firewall.

At the moment I am testing my IPFW ruleset using “host google.com 
 >” I can see the 
traffic leave the Jail, get natted, the response come back from 8.8.8.8 and the 
traffic is then denied. It seems like the state is not being checked or my 
rules are in the wrong place. I feel that I should be able to fix this but I am 
obviously misunderstanding is how NAT works. 

I was under the assumption that traffic flowed like this:

1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for public 
IP, the traffic is natted, it goes out the WAN interface, comes back, is natted 
and switched to lo1 interface, state is checked and it passes as returning 
traffic.

2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for private 
IP, the traffic is not natted, it stays on the lo1 interface and goes directly 
to the 10.0.0.2 Jail.

I know I could answer my last question if “I read the code” and I have tried 
but am not getting it. Is my understanding of IPFW kernel NAT correct?

Regards,

Nathan

___
freebsd-ipfw@freebsd.org  mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw 

To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org 
”

**

Regards,

Nathan

> On 13 Oct 2015, at 1:37 pm, Ian Smith  wrote:
> 
> On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote:
> 
>> I sent through a question to this list a little while ago and have 
>> been trying to get IPFW NAT working since then. I have had some 
>> success but not the success I need, everything is working correctly 
>> except NAT rules for my particular use case.
> 
> Unfortunately the 

Kernel NAT issues

[Nathan Aherne]

I sent through a question to this list a little while ago and have been trying 
to get IPFW NAT working since then. I have had some success but not the success 
I need, everything is working correctly except NAT rules for my particular use 
case. 

I have read every Google result on the first 50 pages when searching for “IPFW 
NAT” or “IPFW kernel NAT”. I would really appreciate it if someone could help 
me out.

My use case is as follows:

1. I need to use hairpin NAT - I am using Jails behind a http proxy and some 
jails need to be able to communicate with each other but only over the WAN IP. 
This is why I have not use PF.
2. Some jails need to be able to communicate with each other on the private 
interface (lo1)
3. IPFW is configured as default deny
4. Each jail has a list of allowed ports for incoming and outgoing connections, 
these are set on the jails private IP (10.0.0.0/16)
5. I am using a stateful firewall.

At the moment I am testing my IPFW ruleset using “host google.com 
” I can see the traffic leave the Jail, get natted, the 
response come back from 8.8.8.8 and the traffic is then denied. It seems like 
the state is not being checked or my rules are in the wrong place. I feel that 
I should be able to fix this but I am obviously misunderstanding is how NAT 
works. 

I was under the assumption that traffic flowed like this:

1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for public 
IP, the traffic is natted, it goes out the WAN interface, comes back, is natted 
and switched to lo1 interface, state is checked and it passes as returning 
traffic.

2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for private 
IP, the traffic is not natted, it stays on the lo1 interface and goes directly 
to the 10.0.0.2 Jail.

I know I could answer my last question if “I read the code” and I have tried 
but am not getting it. Is my understanding of IPFW kernel NAT correct?

Regards,

Nathan

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Nathan Aherne]

To further illustrate my issue, this is a small log output.

I am running “host google.com ” in the jail, which has the 
IP 10.0.0.1. The UNKNOWN line is logging on the check-state rule. I would 
expect the first piece of traffic out would be UNKNOWN (does not have an entry 
in the state table) but it seems the returning traffic is also showing as 
UNKNOWN (the second 101). You can see that the traffic is returning on the same 
port it went out on, so its obviously the returning traffic. I am not sure why 
state is not being kept?

Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 
out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 
8.8.8.8:53 out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out 
via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 
in via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 
10.0.0.1:57446 in via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in 
via bce0

Regards,

Nathan

> On 13 Oct 2015, at 1:50 pm, Nathan Aherne  wrote:
> 
> Hi Ian,
> 
> Thank you for your response.
> 
> I didn’t post my ruleset because I should be able to fix the issue myself but 
> I see now that my request to explain “how NAT works” was incorrect.
> 
> I have now included my ruleset below (as well as my initial email).
> 
> # Enable NAT
> ipfw nat 1 config ip $jip same_ports log
> 
> 
> 5 allow ip from any to any via lo0
> 6 deny ip from any to not me in via bce0
> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
> 00101 check-state
> 00110 allow icmp from any to WWW.XXX.YYY .ZZZ recv bce0 
> keep-state
> 00111 allow tcp from any to WWW.XXX.YYY .ZZZ dst-port 
> 65222 recv bce0 setup keep-state
> 00112 allow icmp from WWW.XXX.YYY .ZZZ to any xmit bce0 
> keep-state
> 00113 allow tcp from WWW.XXX.YYY .ZZZ to any dst-port 
> 53,80,443,22,65222 xmit bce0 setup keep-state
> 00114 allow udp from WWW.XXX.YYY .ZZZ to any dst-port 
> 53,123 xmit bce0 keep-state
> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
> setup keep-state
> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
> keep-state
> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup 
> keep-state
> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup 
> keep-state
> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup 
> keep-state
> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup 
> keep-state
> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
> 65500 deny log ip from any to any
> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
> 65534 deny log ip from any to any
> 65535 deny ip from any to any
> 
> **
> I sent through a question to this list a little while ago and have been 
> trying to get IPFW NAT working since then. I have had some success but not 
> the success I need, everything is working correctly except NAT rules for my 
> particular use case. 
> 
> I have read every Google result on the first 50 pages when searching for 
> “IPFW NAT” or “IPFW kernel NAT”. I would really appreciate it if someone 
> could help me out.
> 
> My use case is as follows:
> 
> 1. I need to use hairpin NAT - I am using Jails behind a http proxy and some 
> jails need to be able to communicate with each other but only over the WAN 
> IP. This is why I have not use PF.
> 2. Some jails need to be able to communicate with each other on the private 
> interface (lo1)
> 3. IPFW is configured as default deny
> 4. Each jail has a list of allowed ports for incoming and outgoing 
> connections, these are set on the jails private IP (10.0.0.0/16)
> 5. I am using a stateful firewall.
> 
> At the moment I am testing my IPFW ruleset using “host google.com 
>  >” I can see the 
> traffic leave the Jail, get natted, the response come back from 8.8.8.8 and 
> the traffic is then denied. It seems like the state is not being checked or 
> my rules are in the wrong place. I feel that I should be able to fix this but 
> I am obviously misunderstanding is how NAT works. 
> 
> I was under the assumption that traffic flowed like this:
> 
> 1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for 
> public IP, the traffic is 

Re: Kernel NAT issues

[Ian Smith]

On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote:

 > I sent through a question to this list a little while ago and have 
 > been trying to get IPFW NAT working since then. I have had some 
 > success but not the success I need, everything is working correctly 
 > except NAT rules for my particular use case.

Unfortunately the rest of your message failed to quote properly here, 
i.e not quoted indented as above, so I'll leave it out for now; perhaps 
it's my old mailer (pine) at fault.  Maybe plain ASCII text would help.

That said, without sharing your actual ruleset with us, sanitised if 
need be, it seems unlikely that anyone will be able to work out what 
might be happening here solely from your textual description.

cheers, Ian
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"