Re: [lxc-users] Intermittent network issue with containers

2020-07-01 Thread Joshua Schaeffer 
Thanks Fajar, I'll look into the workaround.

On 7/1/20 12:40 AM, Fajar A. Nugraha wrote:
> On Wed, Jul 1, 2020 at 1:05 PM Joshua Schaeffer
>  wrote:
>> And the really odd part is that if I try to actually ping *from* the 
>> container *to* my local box it works AND afterwards my original ping *from* 
>> my local box *to* the container starts to work.
> I had a similar problem on a vmware system some time ago. Gave up
> trying to fix it (I don't manage the vmware system), implement a
> workaround instead.
>
> Its either:
> - duplicate IP somewhere on your network
> - your router or switch somehow can't manage arp cache for the container hosts
>
> My workaround is to install iputils-arping (on ubuntu), and (for your
> case) do something like this (preferably on a systemd service)
>
> arping -I veth-mgmt -i 10 -b 10.2.28.1
>
> Or you could replace it with ping, whatever works for you.
>

-- 
Thanks,
Joshua Schaeffer

___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] Intermittent network issue with containers

2020-07-01 Thread Fajar A. Nugraha 
On Wed, Jul 1, 2020 at 1:05 PM Joshua Schaeffer
 wrote:
> And the really odd part is that if I try to actually ping *from* the 
> container *to* my local box it works AND afterwards my original ping *from* 
> my local box *to* the container starts to work.

I had a similar problem on a vmware system some time ago. Gave up
trying to fix it (I don't manage the vmware system), implement a
workaround instead.

Its either:
- duplicate IP somewhere on your network
- your router or switch somehow can't manage arp cache for the container hosts

My workaround is to install iputils-arping (on ubuntu), and (for your
case) do something like this (preferably on a systemd service)

arping -I veth-mgmt -i 10 -b 10.2.28.1

Or you could replace it with ping, whatever works for you.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] Intermittent network issue with containers

2020-07-01 Thread Joshua Schaeffer 
I'm not sure this is actually an issue with LXD but I've been scratching my 
head on this for a while and unable to figure out what is going on so reaching 
out to many different sources. I'm intermittently losing connection to all of 
my container's second interfaces. If I ping out *from* the container *to* an 
external address then the network connection is restored temporarily. Anywhere 
between 5 to 60 minutes later the problem reappears. On the surface it looks 
like a routing or reverse path filtering issue, but (I believe) I've setup 
those parameters properly.

For example I'm trying to ping from my local box (172.16.44.18) to the 
container's second interface called "veth-int-core" (10.2.80.129). Note that 
all general traffic is supposed to go out the first interface called 
"veth-mgmt" (10.2.28.65) and that the default gateway is set on this interface. 
I've set rp_filter on veth-int-core to 2 so the system should not drop the 
packet because of reverse path filtering.

    root@container1:~# cat /proc/sys/net/ipv4/conf/veth-int-core/rp_filter
    2


>From my local box I try ping the veth-int-core interface on the container and 
>receive no response:
    root@client:~$ date -u; ping -c 10 10.2.80.129; date -u
    Tue 30 Jun 2020 22:09:34 PM UTC
    PING 10.2.80.129 (10.2.80.129) 56(84) bytes of data.

    --- 10.2.80.129 ping statistics ---
    10 packets transmitted, 0 received, 100% packet loss, time 9200ms

    Tue 30 Jun 2020 22:09:53 PM UTC

If I sniff the wire on the container at the same time we can see the packet 
arrive with the ICMP request. We can also see an ICMP type 3 code 1 
(destination unreachable) response which includes the ICMP reply in the packet.

    root@container1:~# tcpdump -nevi any icmp
    tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 
262144 bytes
    16:09:34.626373  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 7638, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 1, length 
64
    16:09:35.633862  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 7689, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 2, length 
64
    16:09:36.657897  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 7882, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 3, length 
64
    16:09:37.682063  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 7901, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 4, length 
64
    16:09:37.695263  In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 
(tos 0xc0, ttl 64, id 59700, offset 0, flags [none], proto ICMP (1), length 112)
    10.2.80.129 > 10.2.80.129: ICMP host 172.16.44.18 unreachable, length 92
        (tos 0x0, ttl 64, id 9378, offset 0, flags [none], proto ICMP (1), 
length 84)
    10.2.80.129 > 172.16.44.18: ICMP echo reply, id 18986, seq 1, length 64
    16:09:37.695271  In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 
(tos 0xc0, ttl 64, id 59701, offset 0, flags [none], proto ICMP (1), length 112)
    10.2.80.129 > 10.2.80.129: ICMP host 172.16.44.18 unreachable, length 92
        (tos 0x0, ttl 64, id 9430, offset 0, flags [none], proto ICMP (1), 
length 84)
    10.2.80.129 > 172.16.44.18: ICMP echo reply, id 18986, seq 2, length 64
    16:09:37.695276  In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 
(tos 0xc0, ttl 64, id 59702, offset 0, flags [none], proto ICMP (1), length 112)
    10.2.80.129 > 10.2.80.129: ICMP host 172.16.44.18 unreachable, length 92
        (tos 0x0, ttl 64, id 9612, offset 0, flags [none], proto ICMP (1), 
length 84)
    10.2.80.129 > 172.16.44.18: ICMP echo reply, id 18986, seq 3, length 64
    16:09:38.705661  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 8081, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 5, length 
64
    16:09:39.729581  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 8101, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 6, length 
64
    16:09:40.753507  In e4:aa:5d:99:88:4a ethertype IPv4 (0x0800), length 100: 
(tos 0x0, ttl 62, id 8299, offset 0, flags [DF], proto ICMP (1), length 84)
    172.16.44.18 > 10.2.80.129: ICMP echo request, id 18986, seq 7, length 
64
    16:09:41.759252  In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 
(tos 0xc0, ttl 64, id 60134, offset 0, flags [none], proto ICMP (1), length 112)
    10.2.80.129 > 10.2.80.129: ICMP host 172.16.44.18 unreachable, length 92