Re: [lxc-users] Networking issues with LXC containers in EC2
On 01/12/2016 07:03 PM, Fajar A. Nugraha wrote: On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele wrote: On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote: On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele wrote: I should have added that I have no issue running our software on a single EC2 instance with containers running on that instance. We can assign multiple IPs to the instance itself, as well as to the containers running under the instance, and the containers can all communicate with each other as well as with the host. can the containers in that setup communicate with systems outside the host (e.g. access the internet)? if "no", then you might hit the multiple mac problem Sadly the answer is no. They cannot even ping another host in the same VPC... Looks like multiple mac problem. As in, EC2 only allows one mac from your interface. Proxyarp should work: (1) Make SURE your EC2 instances (I'd call them "host" from now on) supports multiple IPs (private or elastic/public IPs, depending on your needs). The easiest way is to add those IPs to your host interface, make sure that that new IP can be accessed (e.g. ping that IP from another host), and then remove it. (2) Enable proxy arp on the host echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp It turned out that proxy arp was indeed the solution here, but a few other parameters had to be set as well. I just need to run the following commands on each EC2 instance: echo 1 > /proc/sys/net/ipv4/conf/br0/forwarding echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp_pvlan echo 1 > /proc/sys/net/ipv4/conf/br0/proxy_arp echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects echo 0 > /proc/sys/net/ipv4/conf/br0/send_redirects With these settings, my containers and hosts can all talk to each other just like they were all residing on the same subnet. An easy solution in the end. Peter ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Networking issues with LXC containers in EC2
On Wed, Jan 13, 2016 at 10:03 AM, Fajar A. Nugraha wrote: > On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele wrote: >> On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote: >>> >>> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele wrote: I should have added that I have no issue running our software on a single EC2 instance with containers running on that instance. We can assign multiple IPs to the instance itself, as well as to the containers running under the instance, and the containers can all communicate with each other as well as with the host. >>> >>> >>> can the containers in that setup communicate with systems outside the >>> host (e.g. access the internet)? >>> >>> if "no", then you might hit the multiple mac problem >> >> Sadly the answer is no. They cannot even ping another host in the same >> VPC... > > Looks like multiple mac problem. As in, EC2 only allows one mac from > your interface. > > (3) See > https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02380.html Actually my reply on your past thread should be simpler: https://lists.linuxcontainers.org/pipermail/lxc-users/2015-September/010069.html -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Networking issues with LXC containers in EC2
On Tue, Jan 12, 2016 at 9:29 PM, Peter Steele wrote: > On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote: >> >> On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele wrote: >>> >>> I should have added that I have no issue running our software on a single >>> EC2 instance with containers running on that instance. We can assign >>> multiple IPs to the instance itself, as well as to the containers running >>> under the instance, and the containers can all communicate with each >>> other >>> as well as with the host. >> >> >> can the containers in that setup communicate with systems outside the >> host (e.g. access the internet)? >> >> if "no", then you might hit the multiple mac problem > > Sadly the answer is no. They cannot even ping another host in the same > VPC... Looks like multiple mac problem. As in, EC2 only allows one mac from your interface. Proxyarp should work: (1) Make SURE your EC2 instances (I'd call them "host" from now on) supports multiple IPs (private or elastic/public IPs, depending on your needs). The easiest way is to add those IPs to your host interface, make sure that that new IP can be accessed (e.g. ping that IP from another host), and then remove it. (2) Enable proxy arp on the host echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp of course, adjust to your environment (e.g. change interface name if needed). You can also add entries in /etc/sysctl.conf or /etc/sysctl.d so that this setting will persist on reboot. (3) See https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02380.html This should make all outgoing packets use eth0's MAC, and the host will effectively function as a router. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Networking issues with LXC containers in EC2
On 01/12/2016 05:59 AM, Fajar A. Nugraha wrote: On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele wrote: I should have added that I have no issue running our software on a single EC2 instance with containers running on that instance. We can assign multiple IPs to the instance itself, as well as to the containers running under the instance, and the containers can all communicate with each other as well as with the host. can the containers in that setup communicate with systems outside the host (e.g. access the internet)? if "no", then you might hit the multiple mac problem Sadly the answer is no. They cannot even ping another host in the same VPC... Peter ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Networking issues with LXC containers in EC2
On Tue, Jan 12, 2016 at 8:40 PM, Peter Steele wrote: > I should have added that I have no issue running our software on a single > EC2 instance with containers running on that instance. We can assign > multiple IPs to the instance itself, as well as to the containers running > under the instance, and the containers can all communicate with each other > as well as with the host. can the containers in that setup communicate with systems outside the host (e.g. access the internet)? if "no", then you might hit the multiple mac problem ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Networking issues with LXC containers in EC2
I should have added that I have no issue running our software on a single EC2 instance with containers running on that instance. We can assign multiple IPs to the instance itself, as well as to the containers running under the instance, and the containers can all communicate with each other as well as with the host. The problem occurs when we have more than one EC2 instance and need to have the containers in separate instances to communicate with each other. You're right though: If no one on this list has actually dealt with this issue themselves, the quickest answer is probably to talk to AWS directly. Thanks. Peter On 01/11/2016 06:55 PM, Fajar A. Nugraha wrote: On Tue, Jan 12, 2016 at 6:31 AM, Peter Steele wrote: From what I've read, I understand that Amazon has implemented some special/restricted behavior for the networking stack of EC2 instances. The question I have is whether I can accomplish what I've attempted here, specifically, can I access a LXC container hosted on one EC2 instance directly from another EC2 instance or from another LXC container hosted on another EC2 instance? You might want to ask them first. Looks like it's only available for VPC setup: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI If they do allow multiple IP address, then the next step is to check whether they allow multiple MACs (which is what you get when you use bridge). There's a workaround for this if the ONLY limitation is the MAC, using proxyarp. ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] Networking issues with LXC containers in EC2
On Tue, Jan 12, 2016 at 6:31 AM, Peter Steele wrote: > From what I've read, I understand that Amazon has implemented some > special/restricted behavior for the networking stack of EC2 instances. The > question I have is whether I can accomplish what I've attempted here, > specifically, can I access a LXC container hosted on one EC2 instance > directly from another EC2 instance or from another LXC container hosted on > another EC2 instance? You might want to ask them first. Looks like it's only available for VPC setup: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI If they do allow multiple IP address, then the next step is to check whether they allow multiple MACs (which is what you get when you use bridge). There's a workaround for this if the ONLY limitation is the MAC, using proxyarp. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users