Hello,
I have a vps on vultr.com http://vultr.com/ running FreeBSD 10.1-p9 and a
generic kernel:
% uname -a
FreeBSD tzar 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr 7 01:09:46
UTC 2015 r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
amd64
My goal is to run multiple services in jails (hopefully using ezjail or other
convenient manager) and make them accessible from the Internet only on
arbitrary ports (like 80 for http(s) server). So far my approach is as follows:
I clone the lo0 interface and assign IP from 127.0.0.0/8 space to the jail and
redirect port in ipfw nat definition to given address (example in configs
below, I tried also with other addresses on vtnet0, which is my base network
interface, with similar issues). Unfortunately this configuration doesn’t work
for me.
I tested this for znc (IRC bouncer) and nginx. If I run them on main host
(without NAT in front of them) everything works fine. However, if I run them in
jail, behind NAT and send a HTTP(S) request to get some file, connections get
dropped (znc has a web admin module, which is broken because of that). It works
fine for small files but breaks for larger (I haven’t check the threshold but
can do this if this is necessary). For example given curl command and znc
service:
% curl http://$my_ip:6697/pub/jquery-1.11.2.min.js /dev/null
# (stats cut out)
curl: (18) transfer closed with 58648 bytes remaining to read
if I tcpdump connection, transfer looks fine for some time and then ends with a
following sequence (run on main host, jail master):
% sudo tcpdump port 6697
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
23:37:28.621409 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[S], seq 3967654146, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val
601353780 ecr 0,sackOK,eol], length 0
23:37:28.621468 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[S.], seq 517055725, ack 3967654147, win 65535, options [mss 1460,nop,wscale
6,sackOK,TS val 2553669008 ecr 601353780], length 0
23:37:28.635788 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[.], ack 1, win 4117, options [nop,nop,TS val 601353791 ecr 2553669008], length 0
23:37:28.635865 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val 601353791 ecr
2553669008], length 108
23:37:28.636122 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val 2553669022 ecr
601353791], length 17
23:37:28.650153 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[.], ack 18, win 4117, options [nop,nop,TS val 601353805 ecr 2553669022],
length 0
23:37:29.123244 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[.], seq 18:1466, ack 109, win 1040, options [nop,nop,TS val 2553669510 ecr
601353805], length 1448
(transfer goes normally)
23:37:35.519163 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[F.], seq 37666, ack 109, win 1040, options [nop,nop,TS val 2553675906 ecr
601360615], length 0
23:37:35.531004 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[.], ack 33322, win 4096, options [nop,nop,TS val 601360640 ecr 2553675880],
length 0
23:37:36.165352 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[.], seq 33322:34770, ack 109, win 1040, options [nop,nop,TS val 2553676552 ecr
601360640], length 1448
23:37:36.184582 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[.], ack 34770, win 4050, options [nop,nop,TS val 601361283 ecr 2553676552],
length 0
23:37:36.801437 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[.], seq 34770:36218, ack 109, win 1040, options [nop,nop,TS val 2553677188 ecr
601361283], length 1448
23:37:36.910742 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[.], ack 36218, win 4050, options [nop,nop,TS val 601362012 ecr 2553677188],
length 0
23:37:36.910796 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[FP.], seq 36218:37666, ack 109, win 1040, options [nop,nop,TS val 2553677297
ecr 601362012], length 1448
23:37:36.922685 IP $my_home_host.51256 $my_vultr_host.vultr.com.6697: Flags
[F.], seq 109, ack 37667, win 4096, options [nop,nop,TS val 601362025 ecr
2553677297], length 0
23:37:36.922742 IP $my_vultr_host.vultr.com.6697 $my_home_host.51256: Flags
[.], ack 110, win 1040, options [nop,nop,TS val 2553677309 ecr 601362025],
length 0
My ipfw log doesn’t show any rejected packages in this case. For comparison
when I run service on the main host (without NAT and port redirection) sending
transfer is longer and ending sequence looks like follows:
% sudo tcpdump port 6696
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
