Hi Julian, Thank you for replying. I was completely off grid for a while and only got back on it today.
I thought that Vimage was probably the way to achieve what I want. The main reason I was staying away from Vimage was the reported bugs with it, another reason was the extra overhead. I would like to be able to shutdown jails quite regularly so was worried the kernel panic bug or memory leak bug might be a problem here. Is there any version of Vimage/FreeBSD which is stable? Regards, Nathan > On 23 Nov 2015, at 5:02 pm, Julian Elischer <jul...@freebsd.org> wrote: > > On 21/11/2015 10:06 AM, Nathan Aherne wrote: >> I had a bit of a think about how to describe what I am trying to achieve. >> >> I am treating each jail likes its own little "virtual machine”. The jail >> provides certain services, using things like nginx or nodejs, php-fpm, mysql >> or postgresql. The jails can control connections to themselves by >> configuring the firewall ports that are opened on the IP their IP >> (10.0.0.0/16 or a public IP). I know the jails have no firewall of their >> own, the firewall is configured from the host. >> >> I want each jail or “virtual machine” to be able to communicate with one >> another and the wider internet. When a jail does a DNS query for another App >> jail, it may get a public IP on its own Host (or it may get another host) >> and it has no issues being able to communicate with another jail on the same >> host. >> >> At the moment all of the above is working perfectly except for jail to jail >> communication on the same host (when the communication is not directly >> between 10.0.0.0/16 IP addresses). > this is pretty much exactly when vimage/vnet jails could be used to great > affect. > Is there a reason you are not doing that? Each jail has it's own routing > tables, addresses and (virtual) interfaces. > > here's how I'd do it with vimage > > +--------------+ > +---------------+ | servers > | +--------------+ > | > | +--------------+ > | +--------+ | > | | +--------------+ > | | > +--------+ +--+------+----+ > | iface | | bridge | > | +-----+ | > +--------+ +----+---------+ > | > | > | > | > | > | > +------------------------+---------------------+ > | | > | | > | NAT jail router | > | | > | | > +-------+--------+--------+-------+------------+ > | | | | > +--+--+ +--+--+ +--+--+ +--+--+ > | | | | | | | | > | | | | | | | | > | | | | | | | | jails > | | | | | | | | > +-----+ +-----+ +-----+ +-----+ > > > > however the hairpin idea might still be useful even in that scenario if they > don't know about each other's 'local' addresses, but do NAT'd machines need > to talk to each other by externeal addresses? > > i Nathan >>> On 21 Nov 2015, at 9:12 am, Nathan Aherne <nat...@reddog.com.au> wrote: >>> >>> I am not exactly sure how to draw the setup so it doesn’t confuse the >>> situation. The setup is extremely simple (I am not running vimage), jails >>> running on the 10.0.0.0/16 (cloned lo1 interface) network or with public >>> IPs. The jails with private IPs are the HTTP app jails. The Host runs a >>> HTTP Proxy (nginx) and forwards traffic to each HTTP App jail based on the >>> URL it receives. The jails with public IPs are things like database jails >>> which cannot be proxied by the Host. >>> >>> I can happily communicate with any jail from my laptop (externally) but >>> when I want one jail to communicate with another jail (for example an App >>> Jail communicating with the database jail) the traffic shows as backwards >>> (destination:port -> source:port) in the IPFW logs (tshark shows the >>> traffic correctly source:port -> destination:port). The jail to jail >>> traffic tries to go over the lo1 interface (backwards) and is blocked. >>> Below is some IPFW logs of an App jail (10.0.0.25) communicating with the >>> database jail (aaa.bbb.ccc.ddd) >>> >>> IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything >>> is labelled UNKNOWN even if it is KNOWN traffic) >>> >>> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 >>> 10.0.0.25:42957 out via lo1 >>> >>> tshark output (loopback and wan interface capture for port 5432) >>> >>> Capturing on 'Loopback' and 'bce0' >>> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 >>> Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0 >>> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] >>> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 >>> TSval=142888539 TSecr=0 >>> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] >>> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 >>> TSval=142891767 TSecr=0 >>> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] >>> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 >>> TSval=142894976 TSecr=0 >>> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] >>> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 >>> TSval=142898180 TSecr=0 >>> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] >>> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 >>> TSval=142901389 TSecr=0 >>> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] >>> 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 >>> TSval=142907602 TSecr=0 >>> >>> >>>> If so, what sort of routing is setup on both host and jails? >>> Routing is what would be added by default (whatever the host system adds >>> when adding an IP), there is no custom routing. I have wondered if I need >>> to modify the routing table to get this to work. >>> >>> Below is the output of netstat -rn >>> >>> www.xxx.yy <http://www.xxx.yy/>.zzz is the gateway address >>> eee.fff.gg.hhh is the database jail public IP >>> aaa.bbb.cc.ddd is the public IP for NAT >>> lll.mmm.nn.ooo is the Hosts public IP >>> >>> >>> Routing tables >>> >>> Internet: >>> Destination Gateway Flags Netif Expire >>> default www.xxx.yy <http://www.xxx.yy/>.zzz UGS bce0 >>> 10.0.0.1 link#6 UH lo1 >>> 10.0.0.2 link#6 UH lo1 >>> 10.0.0.3 link#6 UH lo1 >>> 10.0.0.4 link#6 UH lo1 >>> 10.0.0.5 link#6 UH lo1 >>> 10.0.0.6 link#6 UH lo1 >>> 10.0.0.7 link#6 UH lo1 >>> 10.0.0.8 link#6 UH lo1 >>> 10.0.0.9 link#6 UH lo1 >>> 10.0.0.10 link#6 UH lo1 >>> 10.0.0.11 link#6 UH lo1 >>> 10.0.0.12 link#6 UH lo1 >>> 10.0.0.13 link#6 UH lo1 >>> 10.0.0.14 link#6 UH lo1 >>> 10.0.0.15 link#6 UH lo1 >>> 10.0.0.16 link#6 UH lo1 >>> 10.0.0.17 link#6 UH lo1 >>> 10.0.0.18 link#6 UH lo1 >>> 10.0.0.19 link#6 UH lo1 >>> 10.0.0.20 link#6 UH lo1 >>> 10.0.0.21 link#6 UH lo1 >>> 10.0.0.22 link#6 UH lo1 >>> 10.0.0.23 link#6 UH lo1 >>> 10.0.0.24 link#6 UH lo1 >>> 10.0.0.25 link#6 UH lo1 >>> 10.0.0.26 link#6 UH lo1 >>> www.xxx.yy.zzz/25 <http://www.xxx.yy.zzz/25> link#1 U >>> bce0 >>> eee.fff.gg.hhh link#1 UHS lo0 >>> eee.fff.gg.hhh/32 link#1 U bce0 >>> aaa.bbb.cc <http://aaa.bbb.cc/>.ddd link#1 UHS lo0 >>> aaa.bbb.cc.ddd/32 link#1 U bce0 >>> lll.mmm.nn.ooo link#1 UHS lo0 >>> 127.0.0.1 link#5 UH lo0 >>> >>> Internet6: >>> Destination Gateway Flags >>> Netif Expire >>> ::/96 ::1 UGRS >>> lo0 >>> ::1 link#5 UH >>> lo0 >>> ::ffff:0.0.0.0/96 ::1 UGRS >>> lo0 >>> fe80::/10 ::1 UGRS >>> lo0 >>> fe80::%lo0/64 link#5 U >>> lo0 >>> fe80::1%lo0 link#5 UHS >>> lo0 >>> ff01::%lo0/32 ::1 U >>> lo0 >>> ff02::/16 ::1 UGRS >>> lo0 >>> ff02::%lo0/32 ::1 U >>> lo0 >>> >>>> Anything like ? >>>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>>> <http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search> >>> Yes just like that. >>> >>> Regards, >>> >>> Nathan >>> >>>> On 19 Nov 2015, at 2:46 am, Ian Smith <smi...@nimnet.asn.au >>>> <mailto:smi...@nimnet.asn.au>> wrote: >>>> >>>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>>>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>>>> For some reason hairpin (loopback nat or nat reflection) does not seem to >>>>>> be working, which is why I chose IPFW in the first place. >>>>> it would be good to see a diagram of what this actually means. >>>> Anything like ? >>>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>>> <http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search> >>>> >>>> Was this so one jail can only access service/s provided by other jail/s, >>>> both/all with internal NAT'd addresses, by using only the public address >>>> and port of the 'router', which IIRC this is a single system with jails? >>>> >>>> If so, what sort of routing is setup on both host and jails? >>>> >>>> (blindfolded, no idea where I've pinned the donkey's tail :) >>>> >>>> cheers, Ian >> _______________________________________________ >> freebsd-ipfw@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw >> To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org" >> >> > > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org" _______________________________________________ freebsd-ipfw@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw To unsubscribe, send any mail to "freebsd-ipfw-unsubscr...@freebsd.org"
