Re: natd in a jail
On Sat, 24 Nov 2012 17:44:30 +1100, Morgan Reed wrote: On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed morgan.s.r...@gmail.com wrote: So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path. Rebuilt the kernel with option IPFIREWALL and friends turned on (including IPFILTER_DEFAULT_TO_ACCEPT or whatever it is). Unless you needed to include FIREWALL_FORWARD, you really didn't need to build ipfw into the kernel, it's all loadable by module. No harm, but. Throw ipfw_nat_load=YES and ipdivert_load=YES into /boot/loader.conf so the modules are available for the jails. And with ipfw nat you won't be needing ipdivert. Again, no harm. Run a quick and dirty ipfw script (running out of an 'up' script I wrote into the OpenVPN config); ipfw nat 1 config if tun0 reset same_ports deny_in ipfw add 500 nat 1 ip from any to any via tun0 Works like a charm, just one last thing I'd like to get squared away here though, currently OpenVPN is using a dynamically created tun device, I'd like to have a static /dev/tun0 exist prior to the /etc/rc.d/natd start launching (because as it is I have to restart natd after the openvpn tunnel comes up), not sure what the best way to achieve this is in a jailed environment though. If the address of the tunX interface is fixed in the jail, you can specify it by IP instead of the interface in the nat setup, like: ipfw nat 1 config ip $address same_ports deny_in ipfw add 500 nat 1 ip from any to any via $address Your use of 'reset' in nat config makes me wonder if it's a variable address though? If IP varies you will need to specify the interface. The next trick will be migrating from my spaghetti script into rc launched jails... cheers, Ian ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Sat, Nov 24, 2012 at 7:26 PM, Ian Smith smi...@nimnet.asn.au wrote: Unless you needed to include FIREWALL_FORWARD, you really didn't need to build ipfw into the kernel, it's all loadable by module. No harm, but. The ipfw_nat module was causing an instant panic at load and I was going to have to rebuild my kernel to debug that anyway, went with the sledgehammer approach and built it in, this box won't be doing anything else so it's no problem. And with ipfw nat you won't be needing ipdivert. Again, no harm. Yeah, I didn't think it should be necessary but something was trying to load it from within the jails and throwing an error, probably the natd startup script, not sure why, I might do some digging if I get bored at some point. If the address of the tunX interface is fixed in the jail, you can specify it by IP instead of the interface in the nat setup, like: ipfw nat 1 config ip $address same_ports deny_in ipfw add 500 nat 1 ip from any to any via $address Your use of 'reset' in nat config makes me wonder if it's a variable address though? If IP varies you will need to specify the interface. Dynamically assigned IP address, I don't control the remote end of the tunnel, IP changes each time the tunnel connects. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Sat, Nov 24, 2012 at 9:16 PM, Morgan Reed morgan.s.r...@gmail.com wrote: And with ipfw nat you won't be needing ipdivert. Again, no harm. Yeah, I didn't think it should be necessary but something was trying to load it from within the jails and throwing an error, probably the natd startup script, not sure why, I might do some digging if I get bored at some point. *facepalm* or I could just remove natd_enable from rc.conf since I don't need it anymore... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Sat, Nov 24, 2012 at 5:44 PM, Morgan Reed morgan.s.r...@gmail.com wrote: Works like a charm, just one last thing I'd like to get squared away here though, currently OpenVPN is using a dynamically created tun device, I'd like to have a static /dev/tun0 exist prior to the /etc/rc.d/natd start launching (because as it is I have to restart natd after the openvpn tunnel comes up), not sure what the best way to achieve this is in a jailed environment though. Scratch that, I definitely need a holiday... natd_enable removed from rc.conf, appropriate ipfw script being run by openvpn prior to dropping its privs (by way of the up directive) and it just works (tm) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
SOLVED: Thanks all for your assistance.
SUMMARY:
- Kernel rebuilt with option IPFIREWALL and friends turned on (not
necessary if your ipfw modules work you should just be able to load
them, mine didn't for reasons I don't really have the time or
inclination to track down)
- OpenVPN configurations modified to use a specific tun device
('device tun' directive replaced with 'device tun0')
- OpenVPN configurations modified to run the following script prior
to dropping privs (via the 'up' directive);
/usr/local/etc/openvpn/up.sh
---
ipfw -q flush
pfw nat 1 config if tun0 reset same_ports deny_in
ipfw add 500 nat 1 ip from any to any via tun0
---
This script assumes that option IPFILTER_DEFAULT_TO_ACCEPT or the
equivalent sysctl frob is set, this is most probably *not* what you
want to do in the real world. Modify as needed, and be sure to set
the permissions on the file appropriately as the script will be
executed by root.
A warning though; this is a total hack, the ipfw stuff should be moved
to /etc/ipfw.rules or similar and processed by ipfw at boot but I'm
not sure how it'll react if you try to do this config before the tun
device is created, I expect it'll be unhappy so you'll need to create
a static tun device for openvpn, this is the right way to do things
but I'm being exceedingly lazy. The script above is a filthy hack, and
potentially dangerous.
- Normal requirements for gateway operation also apply here (which
essentially means set gateway_enable=YES in rc.conf on the host and
all router jails).
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Fri, Nov 23, 2012 at 7:22 AM, Morgan Reed morgan.s.r...@gmail.comwrote: On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed morgan.s.r...@gmail.com wrote: So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path. Scratch that, netgtaph isn't in the GENERIC kernel, so I'll have to rebuild anyway. Why not just load the module? Best regards Andreas ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Fri, Nov 23, 2012 at 7:48 PM, Andreas Nilsson andrn...@gmail.com wrote: Why not just load the module? Yeah, you got beaten to the punch on that one offlist, it's late in the day here ;) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed morgan.s.r...@gmail.com wrote: So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path. Rebuilt the kernel with option IPFIREWALL and friends turned on (including IPFILTER_DEFAULT_TO_ACCEPT or whatever it is). Throw ipfw_nat_load=YES and ipdivert_load=YES into /boot/loader.conf so the modules are available for the jails. Run a quick and dirty ipfw script (running out of an 'up' script I wrote into the OpenVPN config); ipfw nat 1 config if tun0 reset same_ports deny_in ipfw add 500 nat 1 ip from any to any via tun0 Works like a charm, just one last thing I'd like to get squared away here though, currently OpenVPN is using a dynamically created tun device, I'd like to have a static /dev/tun0 exist prior to the /etc/rc.d/natd start launching (because as it is I have to restart natd after the openvpn tunnel comes up), not sure what the best way to achieve this is in a jailed environment though. The next trick will be migrating from my spaghetti script into rc launched jails... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On 22 November 2012 04:00, Morgan Reed morgan.s.r...@gmail.com wrote: Hi All, I've a bit of an odd query which I hope somebody may be able to assist with. I'm looking to set up several OpenVPN tunnels on a single machine (each residing in its own jail) and route data to different destinations over different tunnels by selectively routing the traffic via a particular jail. I have three jails set up with OpenVPN tunnels terminated in each, they all work as expected from the local machine. I can't do a straight forward route over the VPN tunnel as I don't control the other end of the tunnel, I need to treat it as a point-to-point connection as a result, hence I need to use NAT. I've tested this setup with a single tunnel running off a real machine with natd providing NAT, it works like a charm, however, when I move the config into a jail I run into issues, natd doesn't seem to be able to see the incoming traffic, nothing shows up in the logs at all. I'm not even sure if this is actually possible, I'm starting to suspect that natd can't hook in low enough from the jails to access the incoming traffic. Traffic gets into the jail by way of an epair interface between the host and the jail, bridged to the ethernet adapter by way of a bridge device, I can see the traffic attempting to route over the tun interface in the jail (but obviously it's not being NATted so nothing comes back) so the traffic is making it in and through the routing engine, just not via natd. Any suggestions here? The host is FreeBSD-8.3. I've not used it myself, but this sound like something VIMAGE may be good for, basically it's a virtual tcp stack per jail, there's some docs at http://wiki.freebsd.org/Image but I seem to remember a more up to date one elsewhere but can't find it at the moment! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
I've not used it myself, but this sound like something VIMAGE may be good for, basically it's a virtual tcp stack per jail, there's some docs at http://wiki.freebsd.org/Image but I seem to remember a more up to date one elsewhere but can't find it at the moment! AFAIK, VIMAGE is still experimental feature. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Nov 22, 2012, at 2:43 AM, nickolas...@gmail.com wrote: I've not used it myself, but this sound like something VIMAGE may be good for, basically it's a virtual tcp stack per jail, there's some docs at http://wiki.freebsd.org/Image but I seem to remember a more up to date one elsewhere but can't find it at the moment! I have created a boot script for managing vimages (downloadable as a FreeBSD package) and made a little write-up on how to use it... http://druidbsd.sf.net/vimage.shtml Note that I use netgraph for bridging (not if_bridge+epair method which seems to be popular in some other setups -- we've benchmarked netgraph and it scales well). Not to mention that ngctl dot | dot -Tsvg -o network.svg can produce nice pretty graphs of your vimage structure when using my setup. AFAIK, VIMAGE is still experimental feature. Works great, tho, seriously! We're multiplexing hardware 20:1 and could probably push it further (but have conservatively kept things at about 2-3x the number of logical CPUs for number-of-vimages (tho, we have benchmarked up to 65530 nodes on a single bridged network connection before netgraph would refuse to make another (impressive -- but not nearly as impressive as the ~90 minutes it took ifconfig to list all the interfaces lol?). -- Devin _ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Thu, Nov 22, 2012 at 9:38 PM, Simon Dick sim...@irrelevant.org wrote: I've not used it myself, but this sound like something VIMAGE may be good for, basically it's a virtual tcp stack per jail, there's some docs at http://wiki.freebsd.org/Image but I seem to remember a more up to date one elsewhere but can't find it at the moment! These are all VIMAGE jails :) I originally tried to do this without VIMAGE but OpenVPN won't work properly in that environment as if it updated the kernel routing table (which ISTR it couldn't, makes sense given the nature of jail) it would have changed it on the host and all jail images. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Fwd: natd in a jail
Hmm, list was missing from reply-to on this one. -- Forwarded message -- From: Morgan Reed morgan.s.r...@gmail.com Date: Thu, Nov 22, 2012 at 10:36 PM Subject: Re: natd in a jail To: Dewayne Geraghty dewayne.gerag...@heuristicsystems.com.au On Thu, Nov 22, 2012 at 9:33 PM, Dewayne Geraghty dewayne.gerag...@heuristicsystems.com.au wrote: We run a lot of jails with kernel nat and ipfw ( ipsec but that's not what you need here). Some of the hosts haven't migrated from natd to kernel nat, so we're probably similar to your setup. Sounds very similar, just substituting OpenVPN for IPSec. 90% of our jails have an 192.168/16 that nat via an external interface with a routable address, and an internal non-routeable address (ie non-RFC1918); which is probably what you're doing for your VPN stuff. Our openvpn's all use tun, I would suggest that your natd isn't doing exactly like you'd wish - on a good day it can be tricky to get right and tcpdump is your friend, which should be monitored in both your host environment and within the jail. You'll need to enable allow.raw_sockets and you'll probably want to enable bpf to be available in your jail, if you haven't already done so. BPF is enabled for the jails, and the traffic is getting to where it needs to (but not via natd). I'll try enabling raw_sockets in the jails, it is entirely conceivable that natd requires that functionality. Thanks for your assistance, I'll see how I go and report back. Best Regards, Morgan Reed -- They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -- Benjamin Franklin, 1759 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Thu, Nov 22, 2012 at 10:32 PM, Teske, Devin devin.te...@fisglobal.com wrote: I have created a boot script for managing vimages (downloadable as a FreeBSD package) and made a little write-up on how to use it... http://druidbsd.sf.net/vimage.shtml As noted elsewhere, these are VIMAGE jails, but I'm managing them manually with a spaghetti script at the moment (just proof-of-concept at this point), I'll have a look at the script, might make my life easier. Note that I use netgraph for bridging (not if_bridge+epair method which seems to be popular in some other setups -- we've benchmarked netgraph and it scales well). Not to mention that ngctl dot | dot -Tsvg -o network.svg can produce nice pretty graphs of your vimage structure when using my setup. Hmmm, I've not done anything with netgraph before, I'll have a look into it, if it is an issue of the appropriate interfaces not being exposed to natd from the epair/bridge setup that might be an alternate solution, not hugely concerned about scale, it'll pretty much only be my traffic that gets routed this way, but I am interested in making it as efficient as possible (no sense adding additional latency unnecessarily when one already has the tunnel latency to deal with). Thanks, Morgan ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On 11/22/2012 6:00 AM, Morgan Reed wrote: Hi All, Hi, [snipped content] Any suggestions here? A quick one. Could you make a try using ipfw nat instead of natd? I am not sure about divert socket and natd per jail, but NATing using ipfw and libalias(which natd uses as well) works. HTH, Nikos ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Thu, Nov 22, 2012 at 10:36 PM, Morgan Reed morgan.s.r...@gmail.com wrote: BPF is enabled for the jails, and the traffic is getting to where it needs to (but not via natd). I'll try enabling raw_sockets in the jails, it is entirely conceivable that natd requires that functionality. So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: natd in a jail
On Fri, Nov 23, 2012 at 5:16 PM, Morgan Reed morgan.s.r...@gmail.com wrote: So it turns out I'd not bought bpf into the jails, however even with that and raw_sockets enabled I'm still having no joy with natd. I've been looking at ipfw a bit today but I've run into an issue, loading ipfw_nat causes my kernel to instantly panic, I need to recompile with KDB and DDB turned on so I can actually catch the trace though... Might look at netgraph before going too far down that path. Scratch that, netgtaph isn't in the GENERIC kernel, so I'll have to rebuild anyway. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
natd in a jail
Hi All, I've a bit of an odd query which I hope somebody may be able to assist with. I'm looking to set up several OpenVPN tunnels on a single machine (each residing in its own jail) and route data to different destinations over different tunnels by selectively routing the traffic via a particular jail. I have three jails set up with OpenVPN tunnels terminated in each, they all work as expected from the local machine. I can't do a straight forward route over the VPN tunnel as I don't control the other end of the tunnel, I need to treat it as a point-to-point connection as a result, hence I need to use NAT. I've tested this setup with a single tunnel running off a real machine with natd providing NAT, it works like a charm, however, when I move the config into a jail I run into issues, natd doesn't seem to be able to see the incoming traffic, nothing shows up in the logs at all. I'm not even sure if this is actually possible, I'm starting to suspect that natd can't hook in low enough from the jails to access the incoming traffic. Traffic gets into the jail by way of an epair interface between the host and the jail, bridged to the ethernet adapter by way of a bridge device, I can see the traffic attempting to route over the tun interface in the jail (but obviously it's not being NATted so nothing comes back) so the traffic is making it in and through the routing engine, just not via natd. Any suggestions here? The host is FreeBSD-8.3. Thanks, Morgan ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
