Hello, I have a vps on vultr.com <http://vultr.com/> running FreeBSD 10.1-p9 and a generic kernel:
% uname -a FreeBSD tzar 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr 7 01:09:46 UTC 2015 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 My goal is to run multiple services in jails (hopefully using ezjail or other convenient manager) and make them accessible from the Internet only on arbitrary ports (like 80 for http(s) server). So far my approach is as follows: I clone the lo0 interface and assign IP from 127.0.0.0/8 space to the jail and redirect port in ipfw nat definition to given address (example in configs below, I tried also with other addresses on vtnet0, which is my base network interface, with similar issues). Unfortunately this configuration doesn’t work for me. I tested this for znc (IRC bouncer) and nginx. If I run them on main host (without NAT in front of them) everything works fine. However, if I run them in jail, behind NAT and send a HTTP(S) request to get some file, connections get dropped (znc has a web admin module, which is broken because of that). It works fine for small files but breaks for larger (I haven’t check the threshold but can do this if this is necessary). For example given curl command and znc service: % curl http://$my_ip:6697/pub/jquery-1.11.2.min.js > /dev/null # (stats cut out) curl: (18) transfer closed with 58648 bytes remaining to read if I tcpdump connection, transfer looks fine for some time and then ends with a following sequence (run on main host, jail master): % sudo tcpdump port 6697 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 bytes 23:37:28.621409 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [S], seq 3967654146, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 601353780 ecr 0,sackOK,eol], length 0 23:37:28.621468 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [S.], seq 517055725, ack 3967654147, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 2553669008 ecr 601353780], length 0 23:37:28.635788 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [.], ack 1, win 4117, options [nop,nop,TS val 601353791 ecr 2553669008], length 0 23:37:28.635865 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val 601353791 ecr 2553669008], length 108 23:37:28.636122 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val 2553669022 ecr 601353791], length 17 23:37:28.650153 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [.], ack 18, win 4117, options [nop,nop,TS val 601353805 ecr 2553669022], length 0 23:37:29.123244 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [.], seq 18:1466, ack 109, win 1040, options [nop,nop,TS val 2553669510 ecr 601353805], length 1448 (transfer goes normally) 23:37:35.519163 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [F.], seq 37666, ack 109, win 1040, options [nop,nop,TS val 2553675906 ecr 601360615], length 0 23:37:35.531004 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [.], ack 33322, win 4096, options [nop,nop,TS val 601360640 ecr 2553675880], length 0 23:37:36.165352 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [.], seq 33322:34770, ack 109, win 1040, options [nop,nop,TS val 2553676552 ecr 601360640], length 1448 23:37:36.184582 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [.], ack 34770, win 4050, options [nop,nop,TS val 601361283 ecr 2553676552], length 0 23:37:36.801437 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [.], seq 34770:36218, ack 109, win 1040, options [nop,nop,TS val 2553677188 ecr 601361283], length 1448 23:37:36.910742 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [.], ack 36218, win 4050, options [nop,nop,TS val 601362012 ecr 2553677188], length 0 23:37:36.910796 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [FP.], seq 36218:37666, ack 109, win 1040, options [nop,nop,TS val 2553677297 ecr 601362012], length 1448 23:37:36.922685 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: Flags [F.], seq 109, ack 37667, win 4096, options [nop,nop,TS val 601362025 ecr 2553677297], length 0 23:37:36.922742 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: Flags [.], ack 110, win 1040, options [nop,nop,TS val 2553677309 ecr 601362025], length 0 My ipfw log doesn’t show any rejected packages in this case. For comparison when I run service on the main host (without NAT and port redirection) sending transfer is longer and ending sequence looks like follows: % sudo tcpdump port 6696 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 bytes 23:44:54.151840 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [S], seq 3928283896, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 601797509 ecr 0,sackOK,eol], length 0 23:44:54.151879 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [S.], seq 3837375255, ack 3928283897, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1729228577 ecr 601797509], length 0 23:44:54.167015 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [.], ack 1, win 4117, options [nop,nop,TS val 601797522 ecr 1729228577], length 0 23:44:54.167052 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val 601797522 ecr 1729228577], length 108 23:44:54.167508 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val 1729228593 ecr 601797522], length 17 23:44:54.167573 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [P.], seq 18:4479, ack 109, win 1040, options [nop,nop,TS val 1729228593 ecr 601797522], length 4461 23:44:54.167585 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [P.], seq 4479:8575, ack 109, win 1040, options [nop,nop,TS val 1729228593 ecr 601797522], length 4096 23:44:54.167599 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [P.], seq 8575:12671, ack 109, win 1040, options [nop,nop,TS val 1729228593 ecr 601797522], length 4096 23:44:54.167611 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [.], seq 12671:14481, ack 109, win 1040, options [nop,nop,TS val 1729228593 ecr 601797522], length 1810 23:44:54.178628 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [.], ack 14481, win 4039, options [nop,nop,TS val 601797540 ecr 1729228593], length 0 (transfer goes normally) 23:44:54.257686 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [P.], seq 85433:96314, ack 109, win 1040, options [nop,nop,TS val 1729228683 ecr 601797608], length 10881 23:44:54.271581 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [.], ack 86881, win 4096, options [nop,nop,TS val 601797622 ecr 1729228683], length 0 23:44:54.271614 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [F.], seq 96314, ack 109, win 1040, options [nop,nop,TS val 1729228697 ecr 601797622], length 0 23:44:54.276925 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [F.], seq 109, ack 96314, win 4096, options [nop,nop,TS val 601797624 ecr 1729228683], length 0 23:44:54.276939 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [F.], seq 96314, ack 110, win 1040, options [nop,nop,TS val 1729228703 ecr 601797624], length 0 23:44:54.286060 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [F.], seq 109, ack 96315, win 4096, options [nop,nop,TS val 601797633 ecr 1729228697], length 0 23:44:54.286085 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [.], ack 110, win 1040, options [nop,nop,TS val 1729228712 ecr 601797624], length 0 23:44:54.291528 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [.], ack 96315, win 4096, options [nop,nop,TS val 601797637 ecr 1729228697], length 0 23:44:54.720336 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: Flags [.], ack 109, win 0, length 0 23:44:54.974955 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: Flags [.], ack 96315, win 4096, options [nop,nop,TS val 601798310 ecr 1729228712], length 0 Of course file transfer in the latter case works just fine. Connections initiated from the jail seem to work just fine (I am able to download larger files, like distfiles to build ports). I’m grateful for every help. I’m happy to fix my configs or even change approach as long as I can achieve my goals. My configs are included below. % cat /etc/rc.conf hostname=“foobar" ifconfig_vtnet0="dhcp" sshd_enable="YES" static_routes=linklocal route_linklocal="-net 169.254.0.0/16 -interface vtnet0" crypto_load=YES cryptodev_load=YES aesni_load=YES virtio_random_load=YES ifconfig_vtnet0_ipv6="inet6 2001:aaaa:bbbb:cccc::64 prefixlen 64" rtsold_enable=YES ipv6_activate_all_interfaces=YES rtsold_flags="-aF" # Firewall configuration firewall_enable="Yes" firewall_script="/etc/ipfw.rules" firewall_logging="YES" firewall_nat_enable="YES" firewall_nat_interface="vtnet0" # Internal network configuration cloned_interfaces="${cloned_interfaces} lo1" gateway_enable="YES" # enables the gateway # (ezjail is not here as I run it with: service ezjail onestart command) % cat /etc/sysctl.conf net.inet.ip.fw.one_pass=0 # Allow multiple passes for packet, necessary for fw+nat to work net.inet.ip.fw.verbose_limit=25 % cat /etc/ipfw.rules #!/bin/sh ipfw -q -f flush # Set rules command prefix cmd="ipfw add" pif="vtnet0" # interface name of NIC attached to Internet skip="skipto 10000" mip=“111.222.111.222" # public ipv4 address ipfw nat 100 config ip $mip log reset same_ports \ redirect_port tcp 127.0.66.1:6697 6697 \ redirect_port tcp 127.0.66.1:80 80 ipfw add 1 allow log ip from any to any via lo0 $cmd 0003 allow tcp from $my_home_ip/32 to me 22 in setup keep-state $cmd 5 allow icmp6 from any to any via vtnet0 # NAT for incoming packets $cmd 100 nat 100 log ip from any to $mip in $cmd 101 check-state log $cmd 500 $skip log tcp from 127.0.0.0/8 to any out setup keep-state $cmd 501 allow log tcp from me to any out setup keep-state $cmd 502 $skip log udp from 127.0.0.0/8 to any out keep-state $cmd 503 allow log udp from me to any out keep-state $cmd 504 $skip log icmp from 127.0.0.0/8 to any out keep-state $cmd 505 allow log icmp from me to any out keep-state $cmd 506 allow log icmp6 from me to any out keep-state $cmd 610 $skip log tcp from any to me 6697 in setup keep-state $cmd 620 $skip log tcp from any to me 80 in setup keep-state $cmd 630 allow log tcp from any to me 6696 in setup keep-state # Deny rest $cmd 9998 deny log all from any to any in via $pif $cmd 9999 deny log all from any to any out via $pif $cmd 10000 nat 100 log ip4 from 127.0.0.0/8 to any out $cmd 10001 allow log ip from any to any $cmd 10002 deny log ip from any to any % cat /usr/local/etc/ezjail/znc export jail_znc_hostname="znc" export jail_znc_ip="lo1|127.0.66.1" # export jail_znc_ip="lo1|127.0.66.1,lo1|::6697,vtnet0|10.0.0.66,vtnet0|2001:aaaa:bbbb:cccc::6697" #export jail_znc_ip="vtnet0|10.0.0.66/0xffffff00,vtnet0|10.0.0.66,vtnet0|2001:aaaa:bbbb:cccc::6697" export jail_znc_rootdir="/usr/jails/znc" export jail_znc_exec_start="/bin/sh /etc/rc" export jail_znc_exec_stop="" export jail_znc_mount_enable="YES" export jail_znc_devfs_enable="YES" export jail_znc_devfs_ruleset="devfsrules_jail" export jail_znc_procfs_enable="YES" export jail_znc_fdescfs_enable="YES" export jail_znc_image="" export jail_znc_imagetype="" export jail_znc_attachparams="" export jail_znc_attachblocking="" export jail_znc_forceblocking="" export jail_znc_zfs_datasets="" export jail_znc_cpuset="" export jail_znc_fib="" export jail_znc_parentzfs="" export jail_znc_parameters="" export jail_znc_post_start_script="" export jail_znc_retention_policy="" ### Only for debugging export jail_znc_parameters="allow.raw_sockets=1” If you need any more configs or tests, please tell me. Thanks for your help! -- BRs, Wojciech Wojtyniak _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
