Re: Jail with public IP alias
On 29/08/2013 02:08, Alejandro Imass wrote: On Wed, Aug 28, 2013 at 4:11 PM, Frank Leonhardt fra...@fjl.co.uk wrote: On 28/08/2013 19:42, Patrick wrote: On Wed, Aug 28, 2013 at 7:25 AM, Alejandro Imass aim...@yabarana.com wrote: On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: [...] Sorry guys - I had not intention of upsetting the EzJail fan club! No worries there I just think it's an awesome tool. We used plain old jails before, and we even went through the service jail path once, but EzJail is a lot more than just lightweight easy-to-use jailing. The fact remains that I've tried to recreate this problem on what comes to a similar set-up, but without EzJail, and I can't. I've only tested it on FreeBSD 8.2 so far, and I've only tested it from INSIDE a jail. I completely understood what you were saying about it doing weird stuff outside a jail, but my point is that this may or may not be related. Actually you can replicate it easily. Assign a number of IPs to any interface but that the interface has a default route. It will always use the primary or default IP on the other end. You can probably see this effect even on a private network provided all the aliases route through the same gateway. You will not be able to see this effect using aliases on the loopback AFAIK. You don't say what version you're running. I can try and recreate it on another version. It doesn't matter, it's a very basic network issue with aliases in FreeBSD, Linux and other OSs. Look here: http://serverfault.com/questions/12285/when-ip-aliasing-how-does-the-os-determine-which-ip-address-will-be-used-as-sour I would like to know how people deal with this on FBSD Okay, I'm trying here. I tried to recreate it thus: b1# ifconfig bge0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=8009bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE ether 00:21:9b:fd:30:8b inet xx.yy.41.196 netmask 0xffc0 broadcast xx.yy.41.255 inet xx.yy.41.197 netmask 0x broadcast xx.yy.41.197 inet xx.yy.41.198 netmask 0x broadcast xx.yy.41.198 inet xx.yy.41.199 netmask 0x broadcast xx.yy.41.199 inet xx.yy.41.200 netmask 0x broadcast xx.yy.41.200 inet xx.yy.41.201 netmask 0x broadcast xx.yy.41.201 inet xx.yy.41.202 netmask 0x broadcast xx.yy.41.202 inet xx.yy.41.203 netmask 0x broadcast xx.yy.41.203 inet xx2.yy2.76.62 netmask 0xffc0 broadcast xx2.yy2.76.63 inet xx.yy.41.207 netmask 0x broadcast xx.yy.41.207 inet xx.yy.41.206 netmask 0x broadcast xx.yy.41.206 media: Ethernet autoselect (100baseTX full-duplex,flowcontrol,rxpause,txpause) status: active etc... Then: b1# ssh -b xx.yy.41.197 b2 -l myname Open new session and... b1# ssh -b xx.yy.41.198 b2 -l myname Open new session and... b1# ssh -b xx.yy.41.199 b2 -l myname An so on Then on b2: b2# w -n 9:43AM up 803 days, 22:47, 5 users, load averages: 0.07, 0.06, 0.02 USER TTY FROM LOGIN@ IDLE WHAT myname p0 ns0.domainname.org.uk9:28AM14 -csh (csh) myname p1 ns1.domainname.net 9:29AM14 -csh (csh) myname p5 xx.yy.41.199 9:29AM13 -csh (csh) myname p6 xx.yy.41.201 9:30AM - w -n myname p7 xx.yy.41.207 9:30AM11 -csh (csh) The only problem I can see there is that the -n option isn't working on w! I'll look in to that. The reverse lookups match the IP addressed dialled in on. b2 has the same sshd bound to all IP addresses, incidentally. b1 has more than one interface, but all the IP addresses I used are on the same one. My guess, if you're not getting this, is that you're configuring the aliases in a different way, so the output of ipconfig might help, even if it just convinces me the netmask is correct and stops me worrying. I've obviously obfuscated the first part of mine. Or have I misunderstood the problem? Regards, Frank. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On 29/08/2013 09:52, Frank Leonhardt wrote: On 29/08/2013 02:08, Alejandro Imass wrote: On Wed, Aug 28, 2013 at 4:11 PM, Frank Leonhardt fra...@fjl.co.uk wrote: On 28/08/2013 19:42, Patrick wrote: On Wed, Aug 28, 2013 at 7:25 AM, Alejandro Imass aim...@yabarana.com wrote: On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: [...] Sorry guys - I had not intention of upsetting the EzJail fan club! No worries there I just think it's an awesome tool. We used plain old jails before, and we even went through the service jail path once, but EzJail is a lot more than just lightweight easy-to-use jailing. The fact remains that I've tried to recreate this problem on what comes to a similar set-up, but without EzJail, and I can't. I've only tested it on FreeBSD 8.2 so far, and I've only tested it from INSIDE a jail. I completely understood what you were saying about it doing weird stuff outside a jail, but my point is that this may or may not be related. Actually you can replicate it easily. Assign a number of IPs to any interface but that the interface has a default route. It will always use the primary or default IP on the other end. You can probably see this effect even on a private network provided all the aliases route through the same gateway. You will not be able to see this effect using aliases on the loopback AFAIK. You don't say what version you're running. I can try and recreate it on another version. It doesn't matter, it's a very basic network issue with aliases in FreeBSD, Linux and other OSs. Look here: http://serverfault.com/questions/12285/when-ip-aliasing-how-does-the-os-determine-which-ip-address-will-be-used-as-sour I would like to know how people deal with this on FBSD Okay, I'm trying here. I tried to recreate it thus: b1# ifconfig bge0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=8009bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE ether 00:21:9b:fd:30:8b inet xx.yy.41.196 netmask 0xffc0 broadcast xx.yy.41.255 inet xx.yy.41.197 netmask 0x broadcast xx.yy.41.197 inet xx.yy.41.198 netmask 0x broadcast xx.yy.41.198 inet xx.yy.41.199 netmask 0x broadcast xx.yy.41.199 inet xx.yy.41.200 netmask 0x broadcast xx.yy.41.200 inet xx.yy.41.201 netmask 0x broadcast xx.yy.41.201 inet xx.yy.41.202 netmask 0x broadcast xx.yy.41.202 inet xx.yy.41.203 netmask 0x broadcast xx.yy.41.203 inet xx2.yy2.76.62 netmask 0xffc0 broadcast xx2.yy2.76.63 inet xx.yy.41.207 netmask 0x broadcast xx.yy.41.207 inet xx.yy.41.206 netmask 0x broadcast xx.yy.41.206 media: Ethernet autoselect (100baseTX full-duplex,flowcontrol,rxpause,txpause) status: active etc... Then: b1# ssh -b xx.yy.41.197 b2 -l myname Open new session and... b1# ssh -b xx.yy.41.198 b2 -l myname Open new session and... b1# ssh -b xx.yy.41.199 b2 -l myname An so on Then on b2: b2# w -n 9:43AM up 803 days, 22:47, 5 users, load averages: 0.07, 0.06, 0.02 USER TTY FROM LOGIN@ IDLE WHAT myname p0 ns0.domainname.org.uk9:28AM14 -csh (csh) myname p1 ns1.domainname.net 9:29AM14 -csh (csh) myname p5 xx.yy.41.199 9:29AM13 -csh (csh) myname p6 xx.yy.41.201 9:30AM - w -n myname p7 xx.yy.41.207 9:30AM11 -csh (csh) The only problem I can see there is that the -n option isn't working on w! I'll look in to that. The reverse lookups match the IP addressed dialled in on. b2 has the same sshd bound to all IP addresses, incidentally. b1 has more than one interface, but all the IP addresses I used are on the same one. My guess, if you're not getting this, is that you're configuring the aliases in a different way, so the output of ipconfig might help, even if it just convinces me the netmask is correct and stops me worrying. I've obviously obfuscated the first part of mine. Or have I misunderstood the problem? Regards, Frank. P.S. Just for completeness: b1# netstat -r Routing tables Internet: DestinationGatewayFlagsRefs Use Netif Expire defaultxx.yy.41.193 UGS112374 7203472736 bge0 etc... The default route does go through that interface. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Thu, Aug 29, 2013 at 5:03 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On 29/08/2013 09:52, Frank Leonhardt wrote: Hi Frank thanks for taking the time to try to replicate this. Here is all the detailed info 8.1-RELEASE em0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=209bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC ether 00:31:88:bd:b9:3a inet xxx.yyy.52.74 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.70 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.71 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.73 netmask 0xff80 broadcast xxx.yyy.52.127 media: Ethernet autoselect (1000baseT full-duplex) status: active I use rc.conf standard practice for aliases: ifconfig_em0=inet xxx.yyy.52.74 netmask 255.255.255.128 -tso ifconfig_em0_alias0=inet xxx.yyy.52.70 netmask 255.255.255.128 -tso ifconfig_em0_alias1=inet xxx.yyy.52.71 netmask 255.255.255.128 -tso ifconfig_em0_alias2=inet xxx.yyy.52.73 netmask 255.255.255.128 -tso nune# netstat -rn Routing tables Internet: DestinationGatewayFlagsRefs Use Netif Expire defaultxxx.yyy.52.1 UGS 168 182183463em0 127.0.0.1 link#4 UH 00lo0 [... internal aliases to lo0 here...] xxx.yyy.52.0/25link#1 U 068581em0 xxx.yyy.52.70 link#1 UHS 014363lo0 xxx.yyy.52.71 link#1 UHS 064765lo0 xxx.yyy.52.73 link#1 UHS 00lo0 xxx.yyy.52.74 link#1 UHS 029170lo0 Note the Netif Expire on 71,73,74 are showing lo0 could this be the problem? nune# ssh -b xxx.yyy.52.71 foo@bar Password: w -n 3:15PM up 130 days, 22:30, 3 users, load averages: 0.00, 0.02, 0.00 USER TTY FROM LOGIN@ IDLE WHAT [...] foo pts/24 xxx.yyy.52.74 3:14PM - w -n I don't know why mine is showing 74 and from your example it should be showing 71. Did you see the article below? http://serverfault.com/questions/12285/when-ip-aliasing-how-does-the-os-determine-which-ip-address-will-be-used-as-sour This seems to be a pretty common issue or it's just a miss-configuration problem? Thanks! Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Thu, Aug 29, 2013 at 12:07 PM, Alejandro Imass aim...@yabarana.com wrote: On Thu, Aug 29, 2013 at 5:03 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On 29/08/2013 09:52, Frank Leonhardt wrote: Hi Frank thanks for taking the time to try to replicate this. Here is all the detailed info 8.1-RELEASE em0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=209bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC ether 00:31:88:bd:b9:3a inet xxx.yyy.52.74 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.70 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.71 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.73 netmask 0xff80 broadcast xxx.yyy.52.127 media: Ethernet autoselect (1000baseT full-duplex) status: active I use rc.conf standard practice for aliases: ifconfig_em0=inet xxx.yyy.52.74 netmask 255.255.255.128 -tso ifconfig_em0_alias0=inet xxx.yyy.52.70 netmask 255.255.255.128 -tso ifconfig_em0_alias1=inet xxx.yyy.52.71 netmask 255.255.255.128 -tso ifconfig_em0_alias2=inet xxx.yyy.52.73 netmask 255.255.255.128 -tso nune# netstat -rn Routing tables Internet: DestinationGatewayFlagsRefs Use Netif Expire defaultxxx.yyy.52.1 UGS 168 182183463em0 127.0.0.1 link#4 UH 00lo0 [... internal aliases to lo0 here...] xxx.yyy.52.0/25link#1 U 068581em0 xxx.yyy.52.70 link#1 UHS 014363lo0 xxx.yyy.52.71 link#1 UHS 064765lo0 xxx.yyy.52.73 link#1 UHS 00lo0 xxx.yyy.52.74 link#1 UHS 029170lo0 Note the Netif Expire on 71,73,74 are showing lo0 could this be the problem? nune# ssh -b xxx.yyy.52.71 foo@bar Password: w -n 3:15PM up 130 days, 22:30, 3 users, load averages: 0.00, 0.02, 0.00 USER TTY FROM LOGIN@ IDLE WHAT [...] foo pts/24 xxx.yyy.52.74 3:14PM - w -n I don't know why mine is showing 74 and from your example it should be showing 71. Did you see the article below? http://serverfault.com/questions/12285/when-ip-aliasing-how-does-the-os-determine-which-ip-address-will-be-used-as-sour This seems to be a pretty common issue or it's just a miss-configuration problem? Thanks! Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org Aliases should have a netmask of 255.255.255.255. What you seeing is not typical behaviour on FreeBSD. http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-virtual-hosts.html Patrick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Thu, Aug 29, 2013 at 5:07 PM, Patrick gibblert...@gmail.com wrote: On Thu, Aug 29, 2013 at 12:07 PM, Alejandro Imass aim...@yabarana.com wrote: On Thu, Aug 29, 2013 at 5:03 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On 29/08/2013 09:52, Frank Leonhardt wrote: [...] Aliases should have a netmask of 255.255.255.255. What you seeing is not typical behaviour on FreeBSD. http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-virtual-hosts.html Patrick Thanks for pointing this out, the manual is effectively very clear on this. So, I changed the masks for ALL the aliases on that server to /32. It alone has more than 30 aliases on lo0 and 4 public IPs. I tested and still has the same problem. So I rebooted just in case and the problem still persists: $ ifconfig em0 em0: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=209bRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC ether 00:30:48:bd:b9:1a inet xxx.yyy.52.74 netmask 0xff80 broadcast xxx.yyy.52.127 inet xxx.yyy.52.70 netmask 0x broadcast xxx.yyy.52.70 inet xxx.yyy.52.71 netmask 0x broadcast xxx.yyy.52.71 inet xxx.yyy.52.73 netmask 0x broadcast xxx.yyy.52.73 media: Ethernet autoselect (1000baseT full-duplex) status: active $ ssh -b xxx.yyy.52.70 foo@bar Password: 7:58PM up 131 days, 3:14, 1 user, load averages: 0.02, 0.01, 0.00 USER TTY FROM LOGIN@ IDLE WHAT foo pts/14 xxx.yyy.52.74 7:58PM - w -n $ ssh -b xxx.yyy.52.71 foo@bar Password: 7:58PM up 131 days, 3:14, 1 user, load averages: 0.02, 0.01, 0.00 USER TTY FROM LOGIN@ IDLE WHAT foo pts/14 xxx.yyy.52.74 7:58PM - w -n $ ssh -b xxx.yyy.52.73 foo@bar Password: 7:58PM up 131 days, 3:14, 1 user, load averages: 0.02, 0.01, 0.00 USER TTY FROM LOGIN@ IDLE WHAT foo pts/14 xxx.yyy.52.74 7:58PM - w -n I don't understand why I get different results than yours and Frank's. We run a pretty standard set-up so why is this not working for us. Could it be because we turned off TCO on the NIC ? One of you asked about NAT. We are using natd to nat some public ports to other ports on the private IPs that are aliases of lo0. This is for the jails that don't have public IPs we just forward some ports to the jail's ports like this: For example: redirect_port tcp 192.168.101.123:22 12322 redirect_port tcp 192.168.101.123:80 12380 Could this have an effect on OUTBOUND connections?? Seems unlikely to me but I think one of you asked about NAT I suspect for a good reason. I'll turn off the natting temporarily and test. Best, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Thu, Aug 29, 2013 at 7:53 PM, Alejandro Imass aim...@yabarana.com wrote: On Thu, Aug 29, 2013 at 5:07 PM, Patrick gibblert...@gmail.com wrote: On Thu, Aug 29, 2013 at 12:07 PM, Alejandro Imass aim...@yabarana.com wrote: On Thu, Aug 29, 2013 at 5:03 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On 29/08/2013 09:52, Frank Leonhardt wrote: [...] Aliases should have a netmask of 255.255.255.255. What you seeing is not typical behaviour on FreeBSD. [...] One of you asked about NAT. We are using natd to nat some public ports to other ports on the private IPs that are aliases of lo0. This is for the jails that don't have public IPs we just forward some ports to the jail's ports like this: For example: redirect_port tcp 192.168.101.123:22 12322 redirect_port tcp 192.168.101.123:80 12380 Could this have an effect on OUTBOUND connections?? Seems unlikely to me but I think one of you asked about NAT I suspect for a good reason. I'll turn off the natting temporarily and test. I can confirm that the culprit was natd. Now the question becomes why does natd affect the source IP for an outbound connection?? Is there a way to fix it and keep natd? Seems that Patrick's NAT hunch on hist first reply was right on the money. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On28/08/2013 00:19, Patrick wrote: On Tue, Aug 27, 2013 at 3:42 PM, Alejandro Imass aim...@yabarana.com wrote: On Tue, Aug 27, 2013 at 6:28 PM, Patrick gibblert...@gmail.com wrote: That's not the behaviour I see. My jail has a private and public IP. Hi Patrick, thanks for your reply. The issue is actually more basic and it's because the same network card has multiple IPs on the same subnet so the routing table always chooses the primary IP assigned to that interface. I'm trying to figure out if I can fix it in the routing table or will need IPFW to re-write the source address. Thanks, -- Alejandro Imass Hi Alejandro, That's how I've got things setup, too, but I'm not seeing the same behaviour. So I was wondering if there was something different about your setup such as using NAT to allow a jail with a private IP to access the internet at large. Patrick (Tidied up so all now bottom posted) I can confirm that you shouldn't be seeing this behaviour because I don't. I don't use EzJail - i prefer vi. Seriously, setting up a jail is very straightforward anyway, and when I tried ezjail I found it was doing stuff I didn't like, so dropped it early on. It was a long time ago and I've forgotten the specifics. I guess if you're using it your new to this particular game, so please excuse me pointing out a few basics here. Although I can't exactly see how this would cause a problem, remember that many service will bind to ALL IP addresses when they start up, and if they pinch a port any subsequent jail trying to take the same one will fail. For SSH, edit /etc/ssh/sshd_config on the host OS and set the ListenAddress to the one you want to use instead of the default, which means all of them. I can't see a mechanism that would get the results you're seeing, but I don't know what ezjail might be doing. I suspect your problem is with ezjail or something bizzare on your network config; can you try it manually? Regards, Frank. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On28/08/2013 00:19, Patrick wrote: On Tue, Aug 27, 2013 at 3:42 PM, Alejandro Imass aim...@yabarana.com wrote: [...] (Tidied up so all now bottom posted) I can confirm that you shouldn't be seeing this behaviour because I don't. I don't use EzJail - i prefer vi. Seriously, setting up a jail is very straightforward anyway, and when I tried ezjail I found it was doing stuff I didn't like, so dropped it early on. It was a long time ago and I've forgotten the specifics. I guess if you're using it your new to this particular game, so please excuse me pointing out a few basics here. We use Ezjail not because it's easy or because we're new to jails, I think you might be confused on what EzJail actually is and why people use it. We use it because we manage a private cloud exclusively based on FBSD with about a dozen servers with a couple dozen jails each. I use EzJail because it allows us to manage just shy of 300 separate environments with only a couple of sysadmins, and with optimized system resources. We use it because IT ROCKS. Although I can't exactly see how this would cause a problem, remember that many service will bind to ALL IP addresses when they start up, and if they [...] I can't see a mechanism that would get the results you're seeing, but I don't know what ezjail might be doing. I suspect your problem is with ezjail or something bizzare on your network config; can you try it manually? After my OP I immediately sent out second mail stating that the problem is not with Jails or EzJail and it's related to the way that aliases behave on a network interface card. When you have aliases that are on the same subnet, the source IP is the primary IP , that is the first IP set on that network device. You can test this with out jails with a simple ssh connection to another server and then typing who. Even if you force ssh to bind to a particular IP using -b it will still show the primary IP. If you have aliases on different subnets this will not happen. Best, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Wed, Aug 28, 2013 at 7:25 AM, Alejandro Imass aim...@yabarana.com wrote: On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On28/08/2013 00:19, Patrick wrote: On Tue, Aug 27, 2013 at 3:42 PM, Alejandro Imass aim...@yabarana.com wrote: [...] (Tidied up so all now bottom posted) I can confirm that you shouldn't be seeing this behaviour because I don't. I don't use EzJail - i prefer vi. Seriously, setting up a jail is very straightforward anyway, and when I tried ezjail I found it was doing stuff I didn't like, so dropped it early on. It was a long time ago and I've forgotten the specifics. I guess if you're using it your new to this particular game, so please excuse me pointing out a few basics here. We use Ezjail not because it's easy or because we're new to jails, I think you might be confused on what EzJail actually is and why people use it. We use it because we manage a private cloud exclusively based on FBSD with about a dozen servers with a couple dozen jails each. I use EzJail because it allows us to manage just shy of 300 separate environments with only a couple of sysadmins, and with optimized system resources. We use it because IT ROCKS. Although I can't exactly see how this would cause a problem, remember that many service will bind to ALL IP addresses when they start up, and if they [...] I can't see a mechanism that would get the results you're seeing, but I don't know what ezjail might be doing. I suspect your problem is with ezjail or something bizzare on your network config; can you try it manually? After my OP I immediately sent out second mail stating that the problem is not with Jails or EzJail and it's related to the way that aliases behave on a network interface card. When you have aliases that are on the same subnet, the source IP is the primary IP , that is the first IP set on that network device. You can test this with out jails with a simple ssh connection to another server and then typing who. Even if you force ssh to bind to a particular IP using -b it will still show the primary IP. If you have aliases on different subnets this will not happen. I don't think that's true though in the case of jails. On the host system, yes, but when a jail is bound to a particular IP, outbound connections originate from that bound IP. At least they do for me in all of my experience. Still wondering if you're using NAT with your jails, as that could change things. (FWIW, we use ezjail as well. It doesn't do anything special except make having lots of jails easy and lightweight.) Patrick ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On 28/08/2013 19:42, Patrick wrote: On Wed, Aug 28, 2013 at 7:25 AM, Alejandro Imass aim...@yabarana.com wrote: On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On28/08/2013 00:19, Patrick wrote: On Tue, Aug 27, 2013 at 3:42 PM, Alejandro Imass aim...@yabarana.com wrote: [...] (Tidied up so all now bottom posted) I can confirm that you shouldn't be seeing this behaviour because I don't. I don't use EzJail - i prefer vi. Seriously, setting up a jail is very straightforward anyway, and when I tried ezjail I found it was doing stuff I didn't like, so dropped it early on. It was a long time ago and I've forgotten the specifics. I guess if you're using it your new to this particular game, so please excuse me pointing out a few basics here. We use Ezjail not because it's easy or because we're new to jails, I think you might be confused on what EzJail actually is and why people use it. We use it because we manage a private cloud exclusively based on FBSD with about a dozen servers with a couple dozen jails each. I use EzJail because it allows us to manage just shy of 300 separate environments with only a couple of sysadmins, and with optimized system resources. We use it because IT ROCKS. Although I can't exactly see how this would cause a problem, remember that many service will bind to ALL IP addresses when they start up, and if they [...] I can't see a mechanism that would get the results you're seeing, but I don't know what ezjail might be doing. I suspect your problem is with ezjail or something bizzare on your network config; can you try it manually? After my OP I immediately sent out second mail stating that the problem is not with Jails or EzJail and it's related to the way that aliases behave on a network interface card. When you have aliases that are on the same subnet, the source IP is the primary IP , that is the first IP set on that network device. You can test this with out jails with a simple ssh connection to another server and then typing who. Even if you force ssh to bind to a particular IP using -b it will still show the primary IP. If you have aliases on different subnets this will not happen. I don't think that's true though in the case of jails. On the host system, yes, but when a jail is bound to a particular IP, outbound connections originate from that bound IP. At least they do for me in all of my experience. Still wondering if you're using NAT with your jails, as that could change things. (FWIW, we use ezjail as well. It doesn't do anything special except make having lots of jails easy and lightweight.) Sorry guys - I had not intention of upsetting the EzJail fan club! The fact remains that I've tried to recreate this problem on what comes to a similar set-up, but without EzJail, and I can't. I've only tested it on FreeBSD 8.2 so far, and I've only tested it from INSIDE a jail. I completely understood what you were saying about it doing weird stuff outside a jail, but my point is that this may or may not be related. You don't say what version you're running. I can try and recreate it on another version. Again basic, but when you set up an alias, what subnet do you use? Same subnet is ringing alarm bells here. The output of ifconfig might help. Regards, Frank. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Wed, Aug 28, 2013 at 2:42 PM, Patrick gibblert...@gmail.com wrote: On Wed, Aug 28, 2013 at 7:25 AM, Alejandro Imass aim...@yabarana.com wrote: On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: On28/08/2013 00:19, Patrick wrote: [...] I don't think that's true though in the case of jails. On the host system, yes, but when a jail is bound to a particular IP, outbound connections originate from that bound IP. At least they do for me in all of my experience. Still wondering if you're using NAT with your jails, as that could change things. Nope, no NAT. I verified what you said using the aliases in lo0 and it does in fact use the correct private IP, and that is well, no surprise because we rarely have jails actually public IPs so I didn't notice this strange behaviour before. Actually, not so strange once you understand what's going on: It doesn't work the same using the public IP because, the public IP goes through a gateway so it's a different case. In that case it will use the primary IP assigned to the device in that subnet that goes through that routing rule. You can test this if you want but you will need to re-create a scenario where you have multiples IPs assigned to a physical network card and that routes through a common gateway. In this case, it will use only the primary IP assigned to network card. If you actually test it you will see it's not a jail issue, it simply works that way,and it will be consistent on a jail or the base system. The only ways to fix this are either through the routing table or source address re-writing with IPFW or similar. (FWIW, we use ezjail as well. It doesn't do anything special except make having lots of jails easy and lightweight.) It does a lot more than that! We use flavours and have pre-loaded environments for easy deployment, much like people use VMWare. For example we do a lot of development in Catalyst and it takes forever to install a working Catalyst env which we only have to do once and then create Cat flavoured jails in minutes. We also, archive and re-instatiate jails in other servers or add more capacity in an existing env just by archiving and creating a clone jail on another server. So basically with EzJail we have our own cloud-type environment but running on the real hardware and with much more granular control. We also use Amazon AWS but not for anything that's core ot the company. We do a ton of other stuff that relies on EzJails tools, for example update one jail to test and the simply re-create that one to replace all the others. Plain old jails will do the same thing for sure, but if you manage hundreds you'll probably wind up re-inventing EzJail in the first place. Best, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Wed, Aug 28, 2013 at 4:11 PM, Frank Leonhardt fra...@fjl.co.uk wrote: On 28/08/2013 19:42, Patrick wrote: On Wed, Aug 28, 2013 at 7:25 AM, Alejandro Imass aim...@yabarana.com wrote: On Wed, Aug 28, 2013 at 5:42 AM, Frank Leonhardt fra...@fjl.co.uk wrote: [...] Sorry guys - I had not intention of upsetting the EzJail fan club! No worries there I just think it's an awesome tool. We used plain old jails before, and we even went through the service jail path once, but EzJail is a lot more than just lightweight easy-to-use jailing. The fact remains that I've tried to recreate this problem on what comes to a similar set-up, but without EzJail, and I can't. I've only tested it on FreeBSD 8.2 so far, and I've only tested it from INSIDE a jail. I completely understood what you were saying about it doing weird stuff outside a jail, but my point is that this may or may not be related. Actually you can replicate it easily. Assign a number of IPs to any interface but that the interface has a default route. It will always use the primary or default IP on the other end. You can probably see this effect even on a private network provided all the aliases route through the same gateway. You will not be able to see this effect using aliases on the loopback AFAIK. You don't say what version you're running. I can try and recreate it on another version. It doesn't matter, it's a very basic network issue with aliases in FreeBSD, Linux and other OSs. Look here: http://serverfault.com/questions/12285/when-ip-aliasing-how-does-the-os-determine-which-ip-address-will-be-used-as-sour I would like to know how people deal with this on FBSD Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Tue, Aug 27, 2013 at 4:59 PM, Alejandro Imass aim...@yabarana.com wrote: Hi, I have a machine with several public IPs on the same NIC and I bound one of those IPs to a jail created with EzJail. Suppose the scenario is something like this: em0 190.100.100.1 190.100.100.2 190.100.100.3 190.100.100.4 In the jail we are bound only to 190.100.100.4 The default router is correctly set on the jail, etc. But when we ssh out of that jail, or send an email, the receiving end always sees 190.100.100.1 not 190.100.100.4 which is the IP the jail is bound to. I think my problem is actually more basic than this. The problem actually occurs on the base system as well and I think it's because all the IPs are on the same subnet, then the kernel assumes to use the primary IP as the source address. For the sake and usefulness of the mail archives I will end this thread here and start another one with a more appropriate title, not before researching to see if this can be done with the routing table or if I need to use ipfw to re-write the source address. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
That's not the behaviour I see. My jail has a private and public IP. $ ifconfig bce1 bce1: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=c01bbRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE ether a4:ba:db:29:7a:1b inet 192.168.42.23 netmask 0x broadcast 192.168.42.23 media: Ethernet autoselect (1000baseT full-duplex) status: active If I ssh into another host on the 192.168.42.0 network, I see: $ who patrick ttyp1Aug 27 15:21 (192.168.42.23) The host of the jail has multiple IPs on that private subnet: $ ifconfig bce1 bce1: flags=8843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST metric 0 mtu 1500 options=c01bbRXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE ether a4:ba:db:29:7a:1b inet 192.168.42.17 netmask 0xff00 broadcast 192.168.42.255 inet 192.168.42.18 netmask 0x broadcast 192.168.42.18 inet 192.168.42.19 netmask 0x broadcast 192.168.42.19 inet 192.168.42.20 netmask 0x broadcast 192.168.42.20 inet 192.168.42.21 netmask 0x broadcast 192.168.42.21 inet 192.168.42.23 netmask 0x broadcast 192.168.42.23 inet 192.168.42.24 netmask 0x broadcast 192.168.42.24 media: Ethernet autoselect (1000baseT full-duplex) status: active Are you using NAT from your jail to the outside world? Patrick On Tue, Aug 27, 2013 at 2:21 PM, Alejandro Imass aim...@yabarana.com wrote: On Tue, Aug 27, 2013 at 4:59 PM, Alejandro Imass aim...@yabarana.com wrote: Hi, I have a machine with several public IPs on the same NIC and I bound one of those IPs to a jail created with EzJail. Suppose the scenario is something like this: em0 190.100.100.1 190.100.100.2 190.100.100.3 190.100.100.4 In the jail we are bound only to 190.100.100.4 The default router is correctly set on the jail, etc. But when we ssh out of that jail, or send an email, the receiving end always sees 190.100.100.1 not 190.100.100.4 which is the IP the jail is bound to. I think my problem is actually more basic than this. The problem actually occurs on the base system as well and I think it's because all the IPs are on the same subnet, then the kernel assumes to use the primary IP as the source address. For the sake and usefulness of the mail archives I will end this thread here and start another one with a more appropriate title, not before researching to see if this can be done with the routing table or if I need to use ipfw to re-write the source address. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
On Tue, Aug 27, 2013 at 6:28 PM, Patrick gibblert...@gmail.com wrote: That's not the behaviour I see. My jail has a private and public IP. Hi Patrick, thanks for your reply. The issue is actually more basic and it's because the same network card has multiple IPs on the same subnet so the routing table always chooses the primary IP assigned to that interface. I'm trying to figure out if I can fix it in the routing table or will need IPFW to re-write the source address. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: Jail with public IP alias
Hi Alejandro, That's how I've got things setup, too, but I'm not seeing the same behaviour. So I was wondering if there was something different about your setup such as using NAT to allow a jail with a private IP to access the internet at large. Patrick On Tue, Aug 27, 2013 at 3:42 PM, Alejandro Imass aim...@yabarana.com wrote: On Tue, Aug 27, 2013 at 6:28 PM, Patrick gibblert...@gmail.com wrote: That's not the behaviour I see. My jail has a private and public IP. Hi Patrick, thanks for your reply. The issue is actually more basic and it's because the same network card has multiple IPs on the same subnet so the routing table always chooses the primary IP assigned to that interface. I'm trying to figure out if I can fix it in the routing table or will need IPFW to re-write the source address. Thanks, -- Alejandro Imass ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
