On 21/11/2015 10:06 AM, Nathan Aherne wrote:
I had a bit of a think about how to describe what I am trying to achieve.
I am treating each jail likes its own little "virtual machine”. The jail
provides certain services, using things like nginx or nodejs, php-fpm, mysql or
postgresql. The jails can control connections to themselves by configuring the
firewall ports that are opened on the IP their IP (10.0.0.0/16 or a public IP). I
know the jails have no firewall of their own, the firewall is configured from the
host.
I want each jail or “virtual machine” to be able to communicate with one
another and the wider internet. When a jail does a DNS query for another App
jail, it may get a public IP on its own Host (or it may get another host) and
it has no issues being able to communicate with another jail on the same host.
At the moment all of the above is working perfectly except for jail to jail
communication on the same host (when the communication is not directly between
10.0.0.0/16 IP addresses).
this is pretty much exactly when vimage/vnet jails could be used to
great affect.
Is there a reason you are not doing that? Each jail has it's own
routing tables, addresses and (virtual) interfaces.
here's how I'd do it with vimage
+--------------+
+---------------+ | servers
| +--------------+
|
| +--------------+
| +--------+ |
| | +--------------+
| |
+--------+ +--+------+----+
| iface | | bridge |
| +-----+ |
+--------+ +----+---------+
|
|
|
|
|
|
+------------------------+---------------------+
| |
| |
| NAT jail router |
| |
| |
+-------+--------+--------+-------+------------+
| | | |
+--+--+ +--+--+ +--+--+ +--+--+
| | | | | | | |
| | | | | | | |
| | | | | | | | jails
| | | | | | | |
+-----+ +-----+ +-----+ +-----+
however the hairpin idea might still be useful even in that scenario
if they don't know about each other's 'local' addresses, but do NAT'd
machines need to talk to each other by externeal addresses?
i Nathan
On 21 Nov 2015, at 9:12 am, Nathan Aherne <nat...@reddog.com.au> wrote:
I am not exactly sure how to draw the setup so it doesn’t confuse the
situation. The setup is extremely simple (I am not running vimage), jails
running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs.
The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy
(nginx) and forwards traffic to each HTTP App jail based on the URL it
receives. The jails with public IPs are things like database jails which cannot
be proxied by the Host.
I can happily communicate with any jail from my laptop (externally) but when I want
one jail to communicate with another jail (for example an App Jail communicating with
the database jail) the traffic shows as backwards (destination:port ->
source:port) in the IPFW logs (tshark shows the traffic correctly source:port ->
destination:port). The jail to jail traffic tries to go over the lo1 interface
(backwards) and is blocked. Below is some IPFW logs of an App jail (10.0.0.25)
communicating with the database jail (aaa.bbb.ccc.ddd)
IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is
labelled UNKNOWN even if it is KNOWN traffic)
Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432
10.0.0.25:42957 out via lo1
tshark output (loopback and wan interface capture for port 5432)
Capturing on 'Loopback' and 'bce0'
1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0
Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0
2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission]
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142888539
TSecr=0
3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission]
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142891767
TSecr=0
4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission]
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142894976
TSecr=0
5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission]
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142898180
TSecr=0
6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission]
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142901389
TSecr=0
7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission]
42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142907602
TSecr=0
If so, what sort of routing is setup on both host and jails?
Routing is what would be added by default (whatever the host system adds when
adding an IP), there is no custom routing. I have wondered if I need to modify
the routing table to get this to work.
Below is the output of netstat -rn
www.xxx.yy <http://www.xxx.yy/>.zzz is the gateway address
eee.fff.gg.hhh is the database jail public IP
aaa.bbb.cc.ddd is the public IP for NAT
lll.mmm.nn.ooo is the Hosts public IP
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default www.xxx.yy <http://www.xxx.yy/>.zzz UGS bce0
10.0.0.1 link#6 UH lo1
10.0.0.2 link#6 UH lo1
10.0.0.3 link#6 UH lo1
10.0.0.4 link#6 UH lo1
10.0.0.5 link#6 UH lo1
10.0.0.6 link#6 UH lo1
10.0.0.7 link#6 UH lo1
10.0.0.8 link#6 UH lo1
10.0.0.9 link#6 UH lo1
10.0.0.10 link#6 UH lo1
10.0.0.11 link#6 UH lo1
10.0.0.12 link#6 UH lo1
10.0.0.13 link#6 UH lo1
10.0.0.14 link#6 UH lo1
10.0.0.15 link#6 UH lo1
10.0.0.16 link#6 UH lo1
10.0.0.17 link#6 UH lo1
10.0.0.18 link#6 UH lo1
10.0.0.19 link#6 UH lo1
10.0.0.20 link#6 UH lo1
10.0.0.21 link#6 UH lo1
10.0.0.22 link#6 UH lo1
10.0.0.23 link#6 UH lo1
10.0.0.24 link#6 UH lo1
10.0.0.25 link#6 UH lo1
10.0.0.26 link#6 UH lo1
www.xxx.yy.zzz/25 <http://www.xxx.yy.zzz/25> link#1 U bce0
eee.fff.gg.hhh link#1 UHS lo0
eee.fff.gg.hhh/32 link#1 U bce0
aaa.bbb.cc <http://aaa.bbb.cc/>.ddd link#1 UHS lo0
aaa.bbb.cc.ddd/32 link#1 U bce0
lll.mmm.nn.ooo link#1 UHS lo0
127.0.0.1 link#5 UH lo0
Internet6:
Destination Gateway Flags
Netif Expire
::/96 ::1 UGRS lo0
::1 link#5 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#5 U lo0
fe80::1%lo0 link#5 UHS lo0
ff01::%lo0/32 ::1 U lo0
ff02::/16 ::1 UGRS lo0
ff02::%lo0/32 ::1 U lo0
Anything like ?
http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search
<http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search>
Yes just like that.
Regards,
Nathan
On 19 Nov 2015, at 2:46 am, Ian Smith <smi...@nimnet.asn.au
<mailto:smi...@nimnet.asn.au>> wrote:
On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote:
On 11/18/15 8:40 AM, Nathan Aherne wrote:
For some reason hairpin (loopback nat or nat reflection) does not seem to
be working, which is why I chose IPFW in the first place.
it would be good to see a diagram of what this actually means.
Anything like ?
http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search
<http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search>
Was this so one jail can only access service/s provided by other jail/s,
both/all with internal NAT'd addresses, by using only the public address
and port of the 'router', which IIRC this is a single system with jails?
If so, what sort of routing is setup on both host and jails?
(blindfolded, no idea where I've pinned the donkey's tail :)
cheers, Ian
_______________________________________________
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
_______________________________________________
freebsd-ipfw@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
