Re: Set a deny rule for a URL in IPFW by its domain name

[Nathan Aherne]

Just use a dns override for the domains you want to block.

Regards,
Nathan
Sent from my iPhone

> On 1 Dec 2015, at 1:52 AM, Julian Elischer  wrote:
> 
>> On 30/11/2015 8:02 PM, Ian Smith wrote:
>> On Mon, 30 Nov 2015 16:48:49 +0530, Kulamani Sethi wrote:
>>  > Hi all,
>>  >I am using ipfw3, can i block a URL by its domain name? When i am
>>  > setting rules in IPFW by its domain name, it simple set rule by its
>>  > corresponding IP.
>>  > Here example how i set
>>  >
>>  > C:>ipfw add 1002 deny log ip  from www.google.com to any
>>  >
>>  > As i know most of the websites uses dynamic IP, it simple changes there IP
>>  > periodically. This rule i set for google is worked for few moment, then it
>>  > allow the packets to my terminal.
> the only way to do this is to make a daemon similar to what I wrote for cisco 
> many years ago.
> it acts as a DNS 'man-in-the-middle' and compares all DNS responses against 
> black/white lists.
> WHen it gets a hit it:
> 1/ returns a suitably altered answer.
> 2/ adds the address found to a black or white table in ipfw.
> 
> Since Secure DNS is getting more popular, it would probably make more sense 
> these days to make unbound or bind
> feed their work through some filter module to do the same thing.
> 
> ___
> freebsd-ipfw@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
> To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Nathan Aherne]

Hi Julian,

Thank you for replying. I was completely off grid for a while and only got back 
on it today.

I thought that Vimage was probably the way to achieve what I want. The main 
reason I was staying away from Vimage was the reported bugs with it, another 
reason was the extra overhead. I would like to be able to shutdown jails quite 
regularly so was worried the kernel panic bug or memory leak bug might be a 
problem here. Is there any version of Vimage/FreeBSD which is stable?

Regards,

Nathan

> On 23 Nov 2015, at 5:02 pm, Julian Elischer <jul...@freebsd.org> wrote:
> 
> On 21/11/2015 10:06 AM, Nathan Aherne wrote:
>> I had a bit of a think about how to describe what I am trying to achieve.
>> 
>> I am treating each jail likes its own little "virtual machine”. The jail 
>> provides certain services, using things like nginx or nodejs, php-fpm, mysql 
>> or postgresql. The jails can control connections to themselves by 
>> configuring the firewall ports that are opened on the IP their IP  
>> (10.0.0.0/16 or a public IP). I know the jails have no firewall of their 
>> own, the firewall is configured from the host.
>> 
>> I want each jail or “virtual machine” to be able to communicate with one 
>> another and the wider internet. When a jail does a DNS query for another App 
>> jail, it may get a public IP on its own Host (or it may get another host) 
>> and it has no issues being able to communicate with another jail on the same 
>> host.
>> 
>> At the moment all of the above is working perfectly except for jail to jail 
>> communication on the same host (when the communication is not directly 
>> between 10.0.0.0/16 IP addresses).
> this is pretty much exactly when vimage/vnet jails could be used to great 
> affect.
> Is there a reason you are not doing that?  Each jail has it's own routing 
> tables, addresses and (virtual) interfaces.
> 
> here's how I'd do it with vimage
> 
>   +--+
>   +---+  | servers
>   |   +--+
>   |
>   |   +--+
>   |  ++  |
>   |  |+--+
>   |  |
> ++ +--+--++
> | iface  | | bridge   |
> |+-+  |
> ++ ++-+
> |
> |
> |
> |
> |
> |
> ++-+
> |  |
> |  |
> |   NAT jail router|
> |  |
> |  |
> +---+++---++
>|||   |
> +--+--+  +--+--+  +--+--+ +--+--+
> | |  | |  | | | |
> | |  | |  | | | |
> | |  | |  | | | |jails
> | |  | |  | | | |
> +-+  +-+  +-+ +-+
> 
> 
> 
> however the hairpin idea might still be useful even in that scenario if they 
> don't know about each other's 'local' addresses, but do NAT'd machines need 
> to talk to each other by externeal addresses?
> 
> i Nathan
>>> On 21 Nov 2015, at 9:12 am, Nathan Aherne <nat...@reddog.com.au> wrote:
>>> 
>>> I am not exactly sure how to draw the setup so it doesn’t confuse the 
>>> situation. The setup is extremely simple (I am not running vimage), jails 
>>> running on the 10.0.0.0/16 (cloned lo1 interface) network or with public 
>>> IPs. The jails with private IPs are the HTTP app jails. The Host runs a 
>>> HTTP Proxy (nginx) and forwards traffic to each HTTP App jail based on the 
>>> URL it receives. The jails with public IPs are things like database jails 
>>> which cannot be proxied by the Host.
>>> 
>>> I can happily communicate with any jail from my laptop (externally) but 
>>> when I want one jail to communicate with another jail (for example an App 
>>> Jail communicating with the database jail) the traffic shows as backwards 
>>> (destination:port -> source:port) in the IPFW logs (tshark shows the 
>>> traffic correctly source:port -> destination:port). The jail to jail 
>>> traffic tries to go over the lo1 interface (backwards) and is block

Re: Kernel NAT issues

[Nathan Aherne]

#6 UH  lo1
10.0.0.23  link#6 UH  lo1
10.0.0.24  link#6 UH  lo1
10.0.0.25  link#6 UH  lo1
10.0.0.26  link#6 UH  lo1
www.xxx.yy.zzz/25  link#1 U  bce0
eee.fff.gg.hhh link#1 UHS lo0
eee.fff.gg.hhh/32  link#1 U  bce0
aaa.bbb.cc.ddd link#1 UHS lo0
aaa.bbb.cc.ddd/32  link#1 U  bce0
lll.mmm.nn.ooo link#1 UHS lo0
127.0.0.1  link#5 UH  lo0

Internet6:
Destination   Gateway   Flags  
Netif Expire
::/96 ::1   UGRSlo0
::1   link#5UH  lo0
:::0.0.0.0/96 ::1   UGRSlo0
fe80::/10 ::1   UGRSlo0
fe80::%lo0/64 link#5U   lo0
fe80::1%lo0   link#5UHS lo0
ff01::%lo0/32 ::1   U   lo0
ff02::/16 ::1   UGRSlo0
ff02::%lo0/32 ::1   U   lo0

> Anything like ?
> http://kb.juniper.net/InfoCenter/index?page=content=KB24639=search 
> <http://kb.juniper.net/InfoCenter/index?page=content=KB24639=search>
Yes just like that.

Regards,

Nathan

> On 19 Nov 2015, at 2:46 am, Ian Smith <smi...@nimnet.asn.au> wrote:
> 
> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote:
>> On 11/18/15 8:40 AM, Nathan Aherne wrote:
>>> For some reason hairpin (loopback nat or nat reflection) does not seem to
>>> be working, which is why I chose IPFW in the first place.
> 
>> it would be good to see a diagram of what this actually means.
> 
> Anything like ?
> http://kb.juniper.net/InfoCenter/index?page=content=KB24639=search
> 
> Was this so one jail can only access service/s provided by other jail/s, 
> both/all with internal NAT'd addresses, by using only the public address 
> and port of the 'router', which IIRC this is a single system with jails?
> 
> If so, what sort of routing is setup on both host and jails?
> 
> (blindfolded, no idea where I've pinned the donkey's tail :)
> 
> cheers, Ian

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Nathan Aherne]

I had a bit of a think about how to describe what I am trying to achieve.

I am treating each jail likes its own little "virtual machine”. The jail 
provides certain services, using things like nginx or nodejs, php-fpm, mysql or 
postgresql. The jails can control connections to themselves by configuring the 
firewall ports that are opened on the IP their IP  (10.0.0.0/16 or a public 
IP). I know the jails have no firewall of their own, the firewall is configured 
from the host.

I want each jail or “virtual machine” to be able to communicate with one 
another and the wider internet. When a jail does a DNS query for another App 
jail, it may get a public IP on its own Host (or it may get another host) and 
it has no issues being able to communicate with another jail on the same host.

At the moment all of the above is working perfectly except for jail to jail 
communication on the same host (when the communication is not directly between 
10.0.0.0/16 IP addresses).

Regards,

Nathan

> On 21 Nov 2015, at 9:12 am, Nathan Aherne <nat...@reddog.com.au> wrote:
> 
> I am not exactly sure how to draw the setup so it doesn’t confuse the 
> situation. The setup is extremely simple (I am not running vimage), jails 
> running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. 
> The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy 
> (nginx) and forwards traffic to each HTTP App jail based on the URL it 
> receives. The jails with public IPs are things like database jails which 
> cannot be proxied by the Host.
> 
> I can happily communicate with any jail from my laptop (externally) but when 
> I want one jail to communicate with another jail (for example an App Jail 
> communicating with the database jail) the traffic shows as backwards 
> (destination:port -> source:port) in the IPFW logs (tshark shows the traffic 
> correctly source:port -> destination:port). The jail to jail traffic tries to 
> go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs 
> of an App jail (10.0.0.25) communicating with the database jail 
> (aaa.bbb.ccc.ddd)
> 
> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is 
> labelled UNKNOWN even if it is KNOWN traffic)
> 
> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 
> 10.0.0.25:42957 out via lo1
> 
> tshark output (loopback and wan interface capture for port 5432)
> 
> Capturing on 'Loopback' and 'bce0'
>   1   0.0010.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 
> Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0
>   2   3.01390510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142888539 TSecr=0
>   3   6.24165810.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142891767 TSecr=0
>   4   9.45151610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142894976 TSecr=0
>   5  12.65465610.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142898180 TSecr=0
>   6  15.86390010.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142901389 TSecr=0
>   7  22.07665510.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 
> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 
> TSval=142907602 TSecr=0
> 
> 
>> If so, what sort of routing is setup on both host and jails?
> 
> Routing is what would be added by default (whatever the host system adds when 
> adding an IP), there is no custom routing. I have wondered if I need to 
> modify the routing table to get this to work. 
> 
> Below is the output of netstat -rn
> 
> www.xxx.yy &

Re: Kernel NAT issues

[Nathan Aherne]

Hi Everyone,

I think I have worked this out and have a working Stateful IPFW NAT (its worked 
fine for a few weeks) rule set. Hopefully this saves someone else a few weeks 
of their lives.

For some reason hairpin (loopback nat or nat reflection) does not seem to be 
working, which is why I chose IPFW in the first place.

I am actually getting some really weird log results from IPFW (for inter jail 
communication ). IPFW logs show traffic flowing in the opposite direction to 
what tshark shows. Tshark shows the traffic in the direction I would expect. 
IPFW is blocking the traffic because it is seeing it as going the wrong way. 
Anyone have an idea how I can solve this issue, I would super appreciate any 
pointers!

To clarify inter jail communication - not using the local jail IPs but DNS - 
they all resolve to the same public IP. I cannot use split DNS.

###
#!/bin/sh

# Configuration
wif="bce0" # WAN interface
jif="lo1" # Jail interface
jsn="10.0.0.0/16" # Jail subnet
plip="10.0.0.1" # Proxy IP
hwip=“aaa.bbb.ccc.ddd" # Host WAN IP
nwip=“www.xxx.yyy.zzz" # NAT WAN IP

# Script variables
cmd="ipfw -q add"
ks="keep-state"
sks="setup keep-state"

# Flush all rules
ipfw -q -f flush

# Enable NAT
ipfw nat 1 config ip $nwip log

# Allow all loopback traffic
$cmd 5 allow ip from any to any via lo0

# Block any traffic not for Host
$cmd 6 deny ip from any to not me in via $wif

$cmd 100 nat 1 log ip from any to $nwip recv $wif
$cmd 101 check-state log

# Host
$cmd 110 allow icmp from any to $hwip in via $wif $ks
$cmd 111 allow tcp from any to $hwip 65222 in via $wif $sks
$cmd 112 allow icmp from $hwip to any out via $wif $ks
$cmd 113 allow tcp from $hwip to any 53, 80, 443, 22, 65222 out via $wif $sks
$cmd 114 allow udp from $hwip to any 53, 123 out via $wif $ks

# Incoming NAT
$cmd 120 skipto 65510 log tcp from any to $jsn recv $wif $sks
$cmd 121 skipto 65510 log udp from any to $jsn recv $wif $ks
# Outgoing NAT
$cmd 122 skipto 65510 log tcp from $jsn to not $jsn xmit $wif $sks
$cmd 123 skipto 65510 log udp from $jsn to not $jsn xmit $wif $ks

# JAILS RULES 200-65000

# Block any other traffic
$cmd 65501 deny log ip from any to any

# Outgoing NAT
$cmd 65510 nat 1 log ip from $jsn to any xmit $wif
$cmd 65511 allow log ip from $nwip to any xmit $wif
# Block any other traffic
#$cmd 65519 deny log ip from any to any

# Incoming NAT
$cmd 65520 allow log ip from any to $jsn recv $wif
# Block any other traffic
$cmd 65522 deny log ip from any to any

#######

Regards,

Nathan

> On 21 Oct 2015, at 9:02 am, Nathan Aherne <nat...@reddog.com.au> wrote:
> 
> Hi Ian,
> 
> Thank you very much for your response! Sorry about the late response, I have 
> been offline for a few days.
> 
> I think I may have worked this issue out. I am bringing up a bunch of Jails 
> today to test my firewall rules in the hopes that I have corrected my 
> problem. I will reply back either way.
> 
> Regards,
> 
> Nathan
> 
>> On 15 Oct 2015, at 12:51 am, Ian Smith <smi...@nimnet.asn.au> wrote:
>> 
>> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
>>> Hi Ian,
>>> 
>>> Thank you for your response.
>>> 
>>> I didnÿÿt post my ruleset because I should be able to fix the issue 
>>> myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
>>> incorrect.
>>> 
>>> I have now included my ruleset below (as well as my initial email).
>> 
>> Hi Nathan,
>> 
>> I was really hoping someone who knows more about stateful rule handling 
>> (and jail networking) might have a go at this.  Oh well I'll try, but 
>> I'm a lousy mindreader, and really don't know which of the below 
>> constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
>> and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?
>> 
>>> # Enable NAT
>>> ipfw nat 1 config ip $jip same_ports log
>> 
>> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
>> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
>> to you, and so traffic to it won't be subject to NAT .. correct?  But 
>> the WWW... address and all 10.0/16 addresses are jails, not any separate 
>> boxes you gateway for, right?  Just the one external interface, right?
>> 
>>> 5 allow ip from any to any via lo0
>>> 6 deny ip from any to not me in via bce0
>>> 00100 nat 1 log ip from any to A

Re: Kernel NAT issues

[Nathan Aherne]

Hi Ian,

Thank you very much for your response! Sorry about the late response, I have 
been offline for a few days.

I think I may have worked this issue out. I am bringing up a bunch of Jails 
today to test my firewall rules in the hopes that I have corrected my problem. 
I will reply back either way.

Regards,

Nathan

> On 15 Oct 2015, at 12:51 am, Ian Smith <smi...@nimnet.asn.au> wrote:
> 
> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
>> Hi Ian,
>> 
>> Thank you for your response.
>> 
>> I didnÿÿt post my ruleset because I should be able to fix the issue 
>> myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
>> incorrect.
>> 
>> I have now included my ruleset below (as well as my initial email).
> 
> Hi Nathan,
> 
> I was really hoping someone who knows more about stateful rule handling 
> (and jail networking) might have a go at this.  Oh well I'll try, but 
> I'm a lousy mindreader, and really don't know which of the below 
> constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
> and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?
> 
>> # Enable NAT
>> ipfw nat 1 config ip $jip same_ports log
> 
> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
> to you, and so traffic to it won't be subject to NAT .. correct?  But 
> the WWW... address and all 10.0/16 addresses are jails, not any separate 
> boxes you gateway for, right?  Just the one external interface, right?
> 
>> 5 allow ip from any to any via lo0
>> 6 deny ip from any to not me in via bce0
>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
>> 00101 check-state
> 
> Ok, inbound from WAN is nat'd and existing stateful flows followed by 
> executing the rule that originally kept state.  Where this is a skipto, 
> skipto will be performed.  But where it's a nat rule, I've no idea .. 
> see below, but you really don't want to add keep-state (again) there.
> 
>> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 
>> keep-state
> 
> Hmm.  I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless 
> rule would make more sense especially for inbound ICMP.  But moving on ..
> 
>> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 
>> 65222 recv bce0 setup keep-state
> 
> Ok, but showting why plain text works better than HTML on lists :)
> 
>> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 
>> keep-state
>> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 
>> 53,80,443,22,65222 xmit bce0 setup keep-state
>> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 
>> 53,123 xmit bce0 keep-state
> 
> Smells ok.
> 
>> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
>> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
> 
> Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's 
> got a problem .. see below.  These two are inbound traffic (recv) and as 
> is, skipping to 65501 will fall through two outbound rules to be denied.
> 
> Either allow them here directly, or likely better, skipto a separate
> target that then allows (or denies) them, if that's what you intended?
> 
>> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
>> setup keep-state
>> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
>> keep-state
> 
> Ok, this traffic does needs to be NAT'd on the way out.
> 
>> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup 
>> keep-state
>> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup 
>> keep-state
>> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
> 
> Not clear why these tcp ports are open inbound and outbound?  Presumably 
> this is jail-to-jail traffic?  Perhaps not relevant to your problem.
> 
>> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup 
>> keep-state
>> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup 
>> keep-state
>> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
>> 65500 deny log ip from any to any
> 
> Ok.
> 
>> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 
> This the target for outbound traffix, xmit bce0, so nat is appropriate.  
> Does jail-to-jail traffic travels via lo1?  Or what?
> 
> This won'

Re: Kernel NAT issues

[Nathan Aherne]

Hi Ian,

Thank you for your response.

I didn’t post my ruleset because I should be able to fix the issue myself but I 
see now that my request to explain “how NAT works” was incorrect.

I have now included my ruleset below (as well as my initial email).

# Enable NAT
ipfw nat 1 config ip $jip same_ports log


5 allow ip from any to any via lo0
6 deny ip from any to not me in via bce0
00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
00101 check-state
00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 
keep-state
00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 
65222 recv bce0 setup keep-state
00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 
keep-state
00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 
53,80,443,22,65222 xmit bce0 setup keep-state
00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 
53,123 xmit bce0 keep-state
00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup 
keep-state
00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
keep-state
00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state
00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state
00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state
00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state
00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
65500 deny log ip from any to any
65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
65534 deny log ip from any to any
65535 deny ip from any to any

**
I sent through a question to this list a little while ago and have been trying 
to get IPFW NAT working since then. I have had some success but not the success 
I need, everything is working correctly except NAT rules for my particular use 
case. 

I have read every Google result on the first 50 pages when searching for “IPFW 
NAT” or “IPFW kernel NAT”. I would really appreciate it if someone could help 
me out.

My use case is as follows:

1. I need to use hairpin NAT - I am using Jails behind a http proxy and some 
jails need to be able to communicate with each other but only over the WAN IP. 
This is why I have not use PF.
2. Some jails need to be able to communicate with each other on the private 
interface (lo1)
3. IPFW is configured as default deny
4. Each jail has a list of allowed ports for incoming and outgoing connections, 
these are set on the jails private IP (10.0.0.0/16)
5. I am using a stateful firewall.

At the moment I am testing my IPFW ruleset using “host google.com 
<http://google.com/> <http://google.com/ <http://google.com/>>” I can see the 
traffic leave the Jail, get natted, the response come back from 8.8.8.8 and the 
traffic is then denied. It seems like the state is not being checked or my 
rules are in the wrong place. I feel that I should be able to fix this but I am 
obviously misunderstanding is how NAT works. 

I was under the assumption that traffic flowed like this:

1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for public 
IP, the traffic is natted, it goes out the WAN interface, comes back, is natted 
and switched to lo1 interface, state is checked and it passes as returning 
traffic.

2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for private 
IP, the traffic is not natted, it stays on the lo1 interface and goes directly 
to the 10.0.0.2 Jail.

I know I could answer my last question if “I read the code” and I have tried 
but am not getting it. Is my understanding of IPFW kernel NAT correct?

Regards,

Nathan

___
freebsd-ipfw@freebsd.org <mailto:freebsd-ipfw@freebsd.org> mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw 
<https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org 
<mailto:freebsd-ipfw-unsubscr...@freebsd.org>”

**

Regards,

Nathan

> On 13 Oct 2015, at 1:37 pm, Ian Smith <smi...@nimnet.asn.au> wrote:
> 
> On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote:
> 
>> I sent through a question to this list a little while ago and have 
>> been trying to get IPFW NAT working since then. I have had some 
>> success but not the success 

Kernel NAT issues

[Nathan Aherne]

I sent through a question to this list a little while ago and have been trying 
to get IPFW NAT working since then. I have had some success but not the success 
I need, everything is working correctly except NAT rules for my particular use 
case. 

I have read every Google result on the first 50 pages when searching for “IPFW 
NAT” or “IPFW kernel NAT”. I would really appreciate it if someone could help 
me out.

My use case is as follows:

1. I need to use hairpin NAT - I am using Jails behind a http proxy and some 
jails need to be able to communicate with each other but only over the WAN IP. 
This is why I have not use PF.
2. Some jails need to be able to communicate with each other on the private 
interface (lo1)
3. IPFW is configured as default deny
4. Each jail has a list of allowed ports for incoming and outgoing connections, 
these are set on the jails private IP (10.0.0.0/16)
5. I am using a stateful firewall.

At the moment I am testing my IPFW ruleset using “host google.com 
” I can see the traffic leave the Jail, get natted, the 
response come back from 8.8.8.8 and the traffic is then denied. It seems like 
the state is not being checked or my rules are in the wrong place. I feel that 
I should be able to fix this but I am obviously misunderstanding is how NAT 
works. 

I was under the assumption that traffic flowed like this:

1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for public 
IP, the traffic is natted, it goes out the WAN interface, comes back, is natted 
and switched to lo1 interface, state is checked and it passes as returning 
traffic.

2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for private 
IP, the traffic is not natted, it stays on the lo1 interface and goes directly 
to the 10.0.0.2 Jail.

I know I could answer my last question if “I read the code” and I have tried 
but am not getting it. Is my understanding of IPFW kernel NAT correct?

Regards,

Nathan

___
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"

Re: Kernel NAT issues

[Nathan Aherne]

To further illustrate my issue, this is a small log output.

I am running “host google.com <http://google.com/>” in the jail, which has the 
IP 10.0.0.1. The UNKNOWN line is logging on the check-state rule. I would 
expect the first piece of traffic out would be UNKNOWN (does not have an entry 
in the state table) but it seems the returning traffic is also showing as 
UNKNOWN (the second 101). You can see that the traffic is returning on the same 
port it went out on, so its obviously the returning traffic. I am not sure why 
state is not being kept?

Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 
out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 
8.8.8.8:53 out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out 
via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 
in via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 
10.0.0.1:57446 in via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in 
via bce0

Regards,

Nathan

> On 13 Oct 2015, at 1:50 pm, Nathan Aherne <nat...@reddog.com.au> wrote:
> 
> Hi Ian,
> 
> Thank you for your response.
> 
> I didn’t post my ruleset because I should be able to fix the issue myself but 
> I see now that my request to explain “how NAT works” was incorrect.
> 
> I have now included my ruleset below (as well as my initial email).
> 
> # Enable NAT
> ipfw nat 1 config ip $jip same_ports log
> 
> 
> 5 allow ip from any to any via lo0
> 6 deny ip from any to not me in via bce0
> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
> 00101 check-state
> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 
> keep-state
> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 
> 65222 recv bce0 setup keep-state
> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 
> keep-state
> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 
> 53,80,443,22,65222 xmit bce0 setup keep-state
> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 
> 53,123 xmit bce0 keep-state
> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
> setup keep-state
> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 
> keep-state
> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup 
> keep-state
> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup 
> keep-state
> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup 
> keep-state
> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup 
> keep-state
> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
> 65500 deny log ip from any to any
> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
> 65534 deny log ip from any to any
> 65535 deny ip from any to any
> 
> **
> I sent through a question to this list a little while ago and have been 
> trying to get IPFW NAT working since then. I have had some success but not 
> the success I need, everything is working correctly except NAT rules for my 
> particular use case. 
> 
> I have read every Google result on the first 50 pages when searching for 
> “IPFW NAT” or “IPFW kernel NAT”. I would really appreciate it if someone 
> could help me out.
> 
> My use case is as follows:
> 
> 1. I need to use hairpin NAT - I am using Jails behind a http proxy and some 
> jails need to be able to communicate with each other but only over the WAN 
> IP. This is why I have not use PF.
> 2. Some jails need to be able to communicate with each other on the private 
> interface (lo1)
> 3. IPFW is configured as default deny
> 4. Each jail has a list of allowed ports for incoming and outgoing 
> connections, these are set on the jails private IP (10.0.0.0/16)
> 5. I am using a stateful firewall.
> 
> At the moment I am testing my IPFW ruleset using “host google.com 
> <http://google.com/> <http://google.com/ <http://google.com/>>” I can see the 
> traffic leave the Jail, get natted, the response come back from 8.8.8.8 and 
> the traffic is then denied. It seems like the state is not being checked or 
> my 

Re: IPFW and in kernel nat

[Nathan Aherne]

I have done some further work on this and I have been able to get kernel nat 
working in ipfw but some functionality I am after didn’t work.

As I said in the below email I have a couple of jails with public ips, those 
jails are unable to communicate with the jails on the private network behind 
nat. The traffic seems to go via the loopback interface, which is then blocked 
because its not going through nat.

I have also found that while jails can communicate with each other (hairpin nat 
is working) some of the jails cannot. The jails that cannot communicate with 
each other are the ones which have ports redirected to them. For instance, I 
have a jail on 10.1.0.9 and it is unable to communicate with the jail on 
10.1.0.2 via port 65432. The traffic seems to be going over the loopback 
interface and not via nat. Any traffic between jails where there is no 
redirected ports (via nat) work fine. I have no custom routes, only what the 
system sets up when adding ips via /etc/rc.conf. It seems this may be a bug 
with ipfw and kernel nat?

Regards,

Nathan Aherne
nat...@reddog.com.au
+61 402 555 080

 On 22 Aug 2015, at 1:19 pm, Nathan Aherne nat...@reddog.com.au wrote:
 
 Hi Everyone,
 
 First time message to this list, so I am sorry if I do something against the 
 rules.
 
 I have posted this questions on the FreeBSD forums in two different places 
 but have not had a single response in several days, so thought I might get 
 more success here. I have spent many days solely reading about IPFW and in 
 kernel nat in the hopes that I would be able to get what I want to achieve 
 working without having to ask for help. I have found it extremely hard to 
 find what I would think is a regular use use case for IPFW and nat. There are 
 examples but there are just so many that are bad or hard to follow, including 
 the handbook and IPFW that its extremely easy to get confused, as I have. 
 Because of this, I will be posting my eventual solution back to the forums in 
 the hopes that it stops someone from wasting days.
 
 I have a single wan interface (bce0) that has two public IPs attached to it. 
 I would like to use one of these IPs for the host (wanip1) and jails and 
 another for a jail (wanip2) that requires a public IP. I have a cloned lo0 
 interface, lo1 with a subnet of 10.1.0.0/24 which all the jails reside on. I 
 would like to forward ports 80,443 coming in on wanip1 to the jail proxy, 
 which then forwards the traffic off to the correct jail. I would like for the 
 host server to be able to have its own firewall and also receive traffic on 
 port 65222. I have a couple of other jails that require some other ports as 
 you will see in my ipfw.rules script. I would also like each jail to be able 
 to connect to the internet.
 
 I would like the jails to be able to have their own firewalls, which I 
 currently allow by including a firewall script from within each jail.
 
 I have net.inet.ip.fw.one_pass=0 set in /etc/sysctl.conf
 
 Below is my ipfw.rules script. I would super appreciate it if someone could 
 show me where I have gone wrong.
 
 *
 #!/bin/sh
 
 ##
 # Configuration
 wif=bce0 # WAN interface
 wip=119.111.111.111 # WAN IP
 
 j1if=lo1 # Jails Interface
 j1net=10.1.0.0/24 # Jails Network
 skip=skipto 3
 jcmd=ipfw -q add 1” # Jails rules are inserted here
 ##
 # IPFW variables
 cmd=ipfw -q add
 ks=keep-state
 sks=setup keep-state
 ##
 ipfw -q -f flush # Flush all rules
 ##
 # NAT on Jail1 WAN IP
 ipfw nat 1 config ip $wip same_ports unreg_only reset \
 redirect_port tcp 10.1.0.1:80 80 \
 redirect_port tcp 10.1.0.1:443 443 \
 redirect_port tcp 10.1.0.2:65432 65432 \
 redirect_port tcp 10.1.0.3:65444 65444 \
 redirect_port tcp 10.1.0.3:65333 65333
 
 # Jail1 Network - allow all traffic
 $cmd 10 allow ip from any to any via $j1if
 ##
 # Allow all traffic on Loopback
 $cmd 999 allow ip from any to any via lo0
 
 # NAT Rule for incoming packets on WAN IP
 $cmd 1000 nat 1 ip4 from any to any in via $wif
 
 # Check stage table
 $cmd 2000 check-state
 ##
 # HOST ONLY
 # Ping
 $cmd 2100 allow icmp from $wip to any out $ks
 $cmd 2101 allow icmp from any to $wip in $ks
 # DNS
 $cmd 2102 allow tcp from $wip to any 53 out $sks
 $cmd 2103 allow udp from $wip to any 53 out $ks
 # Ports
 $cmd 2104 allow tcp from $wip to any 80 out $ks
 $cmd 2105 allow tcp from $wip to any 433 out $ks
 # SSH
 $cmd 2106 allow tcp