I believe this link describes the exact problem I've been experiencing:
https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/785668
and although the original post targets KVM, later in the thread it moves
to LXC. This is an old bug report and I'm surprised that this has not
been addressed in recent kernels.
Peter
On 09/10/2015 02:52 PM, Peter Steele wrote:
I've configured a standard CentOS bridge/bond, the exact same setup
that I use for creating VMs. VMs on different hosts communicate
through the bridge without issues. Containers that use the identical
bridge however cannot reliably connect to containers on different
hosts. We've determined that it is some kind of ARP table issue, or at
least appears to be. I have reproduced a scenario where container A on
host X is unable to ping container B on host Y. If I connect to B from
host Y and ping say the gateway of our subnet, suddenly container A on
host X can ping container B. Here's a specific case. Container A on
host X has the IP 172.16.110.106. Its arp table looks like this:
# arp -v -n
Address HWtype HWaddress Flags
Mask Iface
172.16.110.112 ether 00:16:3e:21:c6:c5
C eth0
172.16.110.111 ether 00:16:3e:31:2a:f0
C eth0
172.16.110.110 ether 00:16:3e:5c:62:a9
C eth0
Entries: 4 Skipped: 0 Found: 4
Container B on host Y has the IP 172.16.110.113. Its arp table looks
like this:
# arp -v -n
Address HWtype HWaddress Flags
Mask Iface
172.16.110.112 ether 00:16:3e:21:c6:c5
C eth0
172.16.110.106 ether 00:16:3e:20:df:87
C eth0
Entries: 2 Skipped: 0 Found: 2
If I try to ping 172.16.110.113 (container B) from 172.16.110.106
(container A), I get a "Host Unreachable":
# ping 172.16.110.113
PING 172.16.110.113 (172.16.110.113) 56(84) bytes of data.
From 172.16.110.106 icmp_seq=1 Destination Host Unreachable
From 172.16.110.106 icmp_seq=2 Destination Host Unreachable
From 172.16.110.106 icmp_seq=3 Destination Host Unreachable
From 172.16.110.106 icmp_seq=4 Destination Host Unreachable
From 172.16.110.106 icmp_seq=5 Destination Host Unreachable
...
If while this ping is running I connect to container B and ping the
gateway (172.16.0.1), the ping running on container A suddenly starts
working:
...
From 172.16.110.106 icmp_seq=6 Destination Host Unreachable
From 172.16.110.106 icmp_seq=7 Destination Host Unreachable
From 172.16.110.106 icmp_seq=8 Destination Host Unreachable
64 bytes from 172.16.110.113: icmp_seq=57 ttl=64 time=993 ms
64 bytes from 172.16.110.113: icmp_seq=58 ttl=64 time=0.283 ms
64 bytes from 172.16.110.113: icmp_seq=59 ttl=64 time=0.274 ms
...
The arp table on container A now has an entry for container B:
# arp -v -n
Address HWtype HWaddress Flags
Mask Iface
172.16.110.112 ether 00:16:3e:21:c6:c5
C eth0
172.16.110.111 ether 00:16:3e:31:2a:f0
C eth0
172.16.110.113 ether 00:16:3e:65:2a:c5
C eth0
172.16.110.110 ether 00:16:3e:5c:62:a9
C eth0
Entries: 5 Skipped: 0 Found: 5
The arp table on container B of course now has an entry for the gateway:
# arp -v -n
Address HWtype HWaddress Flags
Mask Iface
172.16.110.112 ether 00:16:3e:21:c6:c5
C eth0
172.16.110.106 ether 00:16:3e:20:df:87
C eth0
172.16.0.1 ether 64:12:25:e3:d4:4c
C eth0
Entries: 3 Skipped: 0 Found: 3
We've been running VMs (KVM) with this identical bridged/bonded
configuration for years and have never had an issue with a VM on one
host being unable to communicate with a VM on another host.We've never
had to force an arp table update of any kind. Why do containers behave
in this manner?
Peter
On 09/10/2015 01:01 PM, Bostjan Skufca wrote:
Hi Peter,
since you mentioned you are using bridged interfaces, is my assumption
correct that your containers's network connection is joined directly
to this bridge and containers talk to the world direcly (L2) and not
via routed (L3) network over host OS?
Did you try using routed setup (using bond0 directly and creating
dedicated br1just for containers) and taking bridging functionality
of the linux kernel out of the picture?
I am very interested in your findings, as I have similar setup planned
for deployment in the coming weeks (vlan trunk -> bond -> bridge ->
containers).
(I had similar issues on bonded interfaces on VMware, where tap-based
OpenVPN server would not work at all. It had to do something with how
vmware handles bonds and this not being compatible with L2 stuff
coming out of VM. The second thing I remember very vaguely is that I
tried using bridged interface inside container too, and it did not
cooperate well with host's bridge, so I stopped using bridges inside
containers altogether.)
b.
_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
