Re: natd in a jail
SOLVED: Thanks all for your assistance. SUMMARY: - Kernel rebuilt with option IPFIREWALL and friends turned on (not necessary if your ipfw modules work you should just be able to load them, mine didn't for reasons I don't really have the time or inclination to track down) - OpenVPN configurations modified to use a specific tun device ('device tun' directive replaced with 'device tun0') - OpenVPN configurations modified to run the following script prior to dropping privs (via the 'up' directive); /usr/local/etc/openvpn/up.sh --- ipfw -q flush pfw nat 1 config if tun0 reset same_ports deny_in ipfw add 500 nat 1 ip from any to any via tun0 --- This script assumes that option IPFILTER_DEFAULT_TO_ACCEPT or the equivalent sysctl frob is set, this is most probably *not* what you want to do in the "real world". Modify as needed, and be sure to set the permissions on the file appropriately as the script will be executed by root. A warning though; this is a total hack, the ipfw stuff should be moved to /etc/ipfw.rules or similar and processed by ipfw at boot but I'm not sure how it'll react if you try to do this config before the tun device is created, I expect it'll be unhappy so you'll need to create a static tun device for openvpn, this is the "right" way to do things but I'm being exceedingly lazy. The script above is a filthy hack, and potentially dangerous. - Normal requirements for gateway operation also apply here (which essentially means set gateway_enable=YES in rc.conf on the host and all router jails). ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Sat, Nov 24, 2012 at 5:44 PM, Morgan Reed wrote: > Works like a charm, just one last thing I'd like to get squared away > here though, currently OpenVPN is using a dynamically created tun > device, I'd like to have a static /dev/tun0 exist prior to the > /etc/rc.d/natd start launching (because as it is I have to restart > natd after the openvpn tunnel comes up), not sure what the best way to > achieve this is in a jailed environment though. Scratch that, I definitely need a holiday... natd_enable removed from rc.conf, appropriate ipfw script being run by openvpn prior to dropping its privs (by way of the up directive) and it "just works" (tm) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Sat, Nov 24, 2012 at 9:16 PM, Morgan Reed wrote: >> And with ipfw nat you won't be needing ipdivert. Again, no harm. > > Yeah, I didn't think it should be necessary but something was trying > to load it from within the jails and throwing an error, probably the > natd startup script, not sure why, I might do some digging if I get > bored at some point. *facepalm* or I could just remove natd_enable from rc.conf since I don't need it anymore... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Sat, Nov 24, 2012 at 7:26 PM, Ian Smith wrote: > Unless you needed to include FIREWALL_FORWARD, you really didn't need to > build ipfw into the kernel, it's all loadable by module. No harm, but. The ipfw_nat module was causing an instant panic at load and I was going to have to rebuild my kernel to debug that anyway, went with the sledgehammer approach and built it in, this box won't be doing anything else so it's no problem. > And with ipfw nat you won't be needing ipdivert. Again, no harm. Yeah, I didn't think it should be necessary but something was trying to load it from within the jails and throwing an error, probably the natd startup script, not sure why, I might do some digging if I get bored at some point. > If the address of the tunX interface is fixed in the jail, you can > specify it by IP instead of the interface in the nat setup, like: > > ipfw nat 1 config ip $address same_ports deny_in > ipfw add 500 nat 1 ip from any to any via $address > > Your use of 'reset' in nat config makes me wonder if it's a variable > address though? If IP varies you will need to specify the interface. Dynamically assigned IP address, I don't control the remote end of the tunnel, IP changes each time the tunnel connects. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Sat, 24 Nov 2012 17:44:30 +1100, Morgan Reed wrote: > On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed wrote: > > So it turns out I'd not bought bpf into the jails, however even with > > that and raw_sockets enabled I'm still having no joy with natd. > > > > I've been looking at ipfw a bit today but I've run into an issue, > > loading ipfw_nat causes my kernel to instantly panic, I need to > > recompile with KDB and DDB turned on so I can actually catch the trace > > though... Might look at netgraph before going too far down that path. > > Rebuilt the kernel with option IPFIREWALL and friends turned on > (including IPFILTER_DEFAULT_TO_ACCEPT or whatever it is). Unless you needed to include FIREWALL_FORWARD, you really didn't need to build ipfw into the kernel, it's all loadable by module. No harm, but. > Throw ipfw_nat_load="YES" and ipdivert_load="YES" into > /boot/loader.conf so the modules are available for the jails. And with ipfw nat you won't be needing ipdivert. Again, no harm. > Run a quick and dirty ipfw script (running out of an 'up' script I > wrote into the OpenVPN config); > ipfw nat 1 config if tun0 reset same_ports deny_in > ipfw add 500 nat 1 ip from any to any via tun0 > > Works like a charm, just one last thing I'd like to get squared away > here though, currently OpenVPN is using a dynamically created tun > device, I'd like to have a static /dev/tun0 exist prior to the > /etc/rc.d/natd start launching (because as it is I have to restart > natd after the openvpn tunnel comes up), not sure what the best way to > achieve this is in a jailed environment though. If the address of the tunX interface is fixed in the jail, you can specify it by IP instead of the interface in the nat setup, like: ipfw nat 1 config ip $address same_ports deny_in ipfw add 500 nat 1 ip from any to any via $address Your use of 'reset' in nat config makes me wonder if it's a variable address though? If IP varies you will need to specify the interface. > The next trick will be migrating from my spaghetti script into rc > launched jails... cheers, Ian ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed wrote: > So it turns out I'd not bought bpf into the jails, however even with > that and raw_sockets enabled I'm still having no joy with natd. > > I've been looking at ipfw a bit today but I've run into an issue, > loading ipfw_nat causes my kernel to instantly panic, I need to > recompile with KDB and DDB turned on so I can actually catch the trace > though... Might look at netgraph before going too far down that path. Rebuilt the kernel with option IPFIREWALL and friends turned on (including IPFILTER_DEFAULT_TO_ACCEPT or whatever it is). Throw ipfw_nat_load="YES" and ipdivert_load="YES" into /boot/loader.conf so the modules are available for the jails. Run a quick and dirty ipfw script (running out of an 'up' script I wrote into the OpenVPN config); ipfw nat 1 config if tun0 reset same_ports deny_in ipfw add 500 nat 1 ip from any to any via tun0 Works like a charm, just one last thing I'd like to get squared away here though, currently OpenVPN is using a dynamically created tun device, I'd like to have a static /dev/tun0 exist prior to the /etc/rc.d/natd start launching (because as it is I have to restart natd after the openvpn tunnel comes up), not sure what the best way to achieve this is in a jailed environment though. The next trick will be migrating from my spaghetti script into rc launched jails... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Fri, Nov 23, 2012 at 7:48 PM, Andreas Nilsson wrote: > Why not just load the module? Yeah, you got beaten to the punch on that one offlist, it's late in the day here ;) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Fri, Nov 23, 2012 at 7:22 AM, Morgan Reed wrote: > On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed > wrote: > > So it turns out I'd not bought bpf into the jails, however even with > > that and raw_sockets enabled I'm still having no joy with natd. > > > > I've been looking at ipfw a bit today but I've run into an issue, > > loading ipfw_nat causes my kernel to instantly panic, I need to > > recompile with KDB and DDB turned on so I can actually catch the trace > > though... Might look at netgraph before going too far down that path. > > Scratch that, netgtaph isn't in the GENERIC kernel, so I'll have to > rebuild anyway. > > Why not just load the module? Best regards Andreas ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed wrote: > So it turns out I'd not bought bpf into the jails, however even with > that and raw_sockets enabled I'm still having no joy with natd. > > I've been looking at ipfw a bit today but I've run into an issue, > loading ipfw_nat causes my kernel to instantly panic, I need to > recompile with KDB and DDB turned on so I can actually catch the trace > though... Might look at netgraph before going too far down that path. Scratch that, netgtaph isn't in the GENERIC kernel, so I'll have to rebuild anyway. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Thu, Nov 22, 2012 at 10:36 PM, Morgan Reed wrote: > BPF is enabled for the jails, and the traffic is getting to where it > needs to (but not via natd). I'll try enabling raw_sockets in the > jails, it is entirely conceivable that natd requires that > functionality. So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On 11/22/2012 6:00 AM, Morgan Reed wrote: Hi All, Hi, [snipped content] Any suggestions here? A quick one. Could you make a try using "ipfw nat" instead of natd? I am not sure about divert socket and natd per jail, but NATing using ipfw and libalias(which natd uses as well) works. HTH, Nikos ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Thu, Nov 22, 2012 at 10:32 PM, Teske, Devin wrote: > I have created a boot script for managing vimages (downloadable as a FreeBSD > package) and made a little write-up on how to use it... > http://druidbsd.sf.net/vimage.shtml As noted elsewhere, these are VIMAGE jails, but I'm managing them manually with a spaghetti script at the moment (just proof-of-concept at this point), I'll have a look at the script, might make my life easier. > Note that I use netgraph for bridging (not if_bridge+epair method which seems > to be popular in some other setups -- we've benchmarked netgraph and it > scales well). Not to mention that "ngctl dot | dot -Tsvg -o network.svg" can > produce nice pretty graphs of your vimage structure when using my setup. Hmmm, I've not done anything with netgraph before, I'll have a look into it, if it is an issue of the appropriate interfaces not being exposed to natd from the epair/bridge setup that might be an alternate solution, not hugely concerned about scale, it'll pretty much only be my traffic that gets routed this way, but I am interested in making it as efficient as possible (no sense adding additional latency unnecessarily when one already has the tunnel latency to deal with). Thanks, Morgan ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Thu, Nov 22, 2012 at 9:38 PM, Simon Dick wrote: > I've not used it myself, but this sound like something VIMAGE may be good > for, basically it's a virtual tcp stack per jail, there's some docs at > http://wiki.freebsd.org/Image but I seem to remember a more up to date one > elsewhere but can't find it at the moment! These are all VIMAGE jails :) I originally tried to do this without VIMAGE but OpenVPN won't work properly in that environment as if it updated the kernel routing table (which ISTR it couldn't, makes sense given the nature of jail) it would have changed it on the host and all jail images. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On Nov 22, 2012, at 2:43 AM, wrote: >> I've not used it myself, but this sound like something VIMAGE may be good >> for, basically it's a virtual tcp stack per jail, there's some docs at >> http://wiki.freebsd.org/Image but I seem to remember a more up to date one >> elsewhere but can't find it at the moment! I have created a boot script for managing vimages (downloadable as a FreeBSD package) and made a little write-up on how to use it... http://druidbsd.sf.net/vimage.shtml Note that I use netgraph for bridging (not if_bridge+epair method which seems to be popular in some other setups -- we've benchmarked netgraph and it scales well). Not to mention that "ngctl dot | dot -Tsvg -o network.svg" can produce nice pretty graphs of your vimage structure when using my setup. > AFAIK, VIMAGE is still experimental feature. Works great, tho, seriously! We're multiplexing hardware 20:1 and could probably push it further (but have conservatively kept things at about 2-3x the number of logical CPUs for number-of-vimages (tho, we have benchmarked up to 65530 nodes on a single bridged network connection before netgraph would refuse to make another (impressive -- but not nearly as impressive as the ~90 minutes it took ifconfig to list all the interfaces lol?). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
> I've not used it myself, but this sound like something VIMAGE may be good > for, basically it's a virtual tcp stack per jail, there's some docs at > http://wiki.freebsd.org/Image but I seem to remember a more up to date one > elsewhere but can't find it at the moment! AFAIK, VIMAGE is still experimental feature. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: natd in a jail
On 22 November 2012 04:00, Morgan Reed wrote: > Hi All, > > I've a bit of an odd query which I hope somebody may be able to > assist with. > > I'm looking to set up several OpenVPN tunnels on a single machine > (each residing in its own jail) and route data to different > destinations over different tunnels by selectively routing the traffic > via a particular jail. > > I have three jails set up with OpenVPN tunnels terminated in each, > they all work as expected from the "local" machine. > > I can't do a straight forward route over the VPN tunnel as I don't > control the other end of the tunnel, I need to treat it as a > point-to-point connection as a result, hence I need to use NAT. > > I've tested this setup with a single tunnel running off a "real" > machine with natd providing NAT, it works like a charm, however, when > I move the config into a jail I run into issues, natd doesn't seem to > be able to see the incoming traffic, nothing shows up in the logs at > all. > > I'm not even sure if this is actually possible, I'm starting to > suspect that natd can't hook in low enough from the jails to access > the incoming traffic. > > Traffic gets into the jail by way of an epair interface between the > host and the jail, bridged to the ethernet adapter by way of a bridge > device, I can see the traffic attempting to route over the tun > interface in the jail (but obviously it's not being NATted so nothing > comes back) so the traffic is making it in and through the routing > engine, just not via natd. > > Any suggestions here? > > The host is FreeBSD-8.3. > I've not used it myself, but this sound like something VIMAGE may be good for, basically it's a virtual tcp stack per jail, there's some docs at http://wiki.freebsd.org/Image but I seem to remember a more up to date one elsewhere but can't find it at the moment! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"