Bug#865975: docker.io breaks (bridged) network for VMs

[Clint Adams]

On Sun, Oct 22, 2017 at 02:00:58AM +0200, Alban Browaeys wrote:
> or a config item in /etc/docker/daemon.json:
> {
>   "iptables": false
> }

Seems like the docker.io package should ship this as a conffile then.

Bug#865975: docker.io breaks (bridged) network for VMs

[Alban Browaeys]

Package: docker.io
Followup-For: Bug #865975


The FORWARD chain policy is set to DROP by docker since 1.13.

The verbose (-V) iptables output (which gives interfaces and packet counters) 
is:
# iptables -L -v -n
Chain INPUT (policy ACCEPT 281 packets, 14176 bytes)
 pkts bytes target prot opt in out source   destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target prot opt in out source   destination
0 0 DOCKER-ISOLATION  all  --  *  *   0.0.0.0/0
0.0.0.0/0
0 0 DOCKER all  --  *  docker0  0.0.0.0/00.0.0.0/0
0 0 ACCEPT all  --  *  docker0  0.0.0.0/00.0.0.0/0  
  ctstate RELATED,ESTABLISHED
0 0 ACCEPT all  --  docker0 !docker0  0.0.0.0/00.0.0.0/0
0 0 ACCEPT all  --  docker0 docker0  0.0.0.0/00.0.0.0/0

Chain OUTPUT (policy ACCEPT 225 packets, 27980 bytes)
 pkts bytes target prot opt in out source   destination

Chain DOCKER (1 references)
 pkts bytes target prot opt in out source   destination

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target prot opt in out source   destination
0 0 RETURN all  --  *  *   0.0.0.0/00.0.0.0/0


I reproduced the network setup but not the KVM one.I cannot confirm
that forwarding is broken.

Upstream provides:

- a command line switch to docker daemon "--iptables=false"
or a config item in /etc/docker/daemon.json:
{
  "iptables": false
}

- upstream also tell to revert the FORWARD policy to ACCEPT byhand ...
but I tested and it stay so on docker restart (even stop and start).
If the box is rebooted the change is lost
as confirmed by 
https://docs.docker.com/engine/userguide/networking/default_network/container-communication/
"The iptables settings are lost when the system reboots. If you want the change 
to be permanent,
refer to your Linux distribution’s documentation."
Mind we cannot apply it from /etc/rc.local or anything boot related as it has 
to be applied
after docker is started ... 
with socket activation we activate docker daemon long after boot.



references:

- 
https://docs.docker.com/engine/userguide/networking/default_network/container-communication/
Container communication between hosts
For security reasons, Docker configures the iptables rules to prevent 
containers from forwarding traffic
from outside the host machine, on Linux hosts. Docker sets the default policy 
of the FORWARD chain to DROP.
(...)
Note: In Docker 1.12 and earlier, the default FORWARD chain policy was ACCEPT. 
When you upgrade
to Docker 1.13 or higher, this default is automatically changed for you.



- Also from 
https://docs.docker.com/engine/userguide/networking/default_network/container-communication/
Communication between containers
(...)
Docker will never make changes to your system iptables rules if you set 
--iptables=false when the daemon starts.
Otherwise the Docker server will add a default rule to the FORWARD chain with a 
blanket ACCEPT policy
if you retain the default --icc=true, or else will set the policy to DROP if 
--icc=false.



Best regards
Alban

-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 
'unstable'), (500, 'stable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.13.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), 
LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages docker.io depends on:
ii  adduser 3.116
ii  docker-containerd   0.2.3+git+docker1.13.1~ds1-1
ii  docker-runc 1.0.0~rc2+git+docker1.13.1~ds1-2
ii  golang-libnetwork   0.8.0-dev.2+git20170202.599.45b4086-3
ii  iptables1.6.1-2+b1
ii  libapparmor12.11.0-11
ii  libc6   2.24-17
ii  libdevmapper1.02.1  2:1.02.142-1
ii  libsqlite3-03.20.1-2
ii  libsystemd0 235-2
ii  lsb-base9.20170808

Versions of packages docker.io recommends:
ii  ca-certificates  20170717
ii  cgroupfs-mount   1.4
ii  git  1:2.15.0~rc1-1
ii  xz-utils 5.2.2-1.3

Versions of packages docker.io suggests:
ii  aufs-tools   1:4.1+20161219-1
ii  btrfs-progs  4.13.3-1
ii  debootstrap  1.0.91
pn  docker-doc   
ii  rinse3.2
pn  zfs-fuse | zfsutils  

-- no debconf information

Bug#865975: docker.io breaks (bridged) network for VMs

[Roland Kammerer]

Hi,

docker is not configured in any special way, the only change to the
config file was attached in the original report.

Here are the relevant outputs of a full session after a fresh boot:

 after boot 

$ ip a s
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
2: eth2:  mtu 1500 qdisc pfifo_fast master br0 
state UP group default qlen 1000
link/ether c0:3f:d5:61:1a:0f brd ff:ff:ff:ff:ff:ff
3: br0:  mtu 1500 qdisc noqueue state UP group 
default qlen 1000
link/ether c0:3f:d5:61:1a:0f brd ff:ff:ff:ff:ff:ff
inet 10.43.70.1/16 brd 10.43.255.255 scope global br0
   valid_lft forever preferred_lft forever
inet6 2001:858:107:1:c23f:d5ff:fe61:1a0f/64 scope global mngtmpaddr dynamic 
   valid_lft 86398sec preferred_lft 14398sec
inet6 fe80::c23f:d5ff:fe61:1a0f/64 scope link 
   valid_lft forever preferred_lft forever

$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source   destination 

Chain FORWARD (policy ACCEPT)
target prot opt source   destination 

Chain OUTPUT (policy ACCEPT)
target prot opt source   destination 

$ brcrl show
bridge name bridge id STP enabled interfaces
br0 8002.c03fd5611a0f no  eth2

 after VM start (which is a KVM based VM, with its NIC on br0, device model 
virtio) 

The output is from the host, the VM pings fine to public hostnames
(e.g., debian.org)

$ ip a s
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
2: eth2:  mtu 1500 qdisc pfifo_fast master br0 
state UP group default qlen 1000
link/ether c0:3f:d5:61:1a:0f brd ff:ff:ff:ff:ff:ff
3: br0:  mtu 1500 qdisc noqueue state UP group 
default qlen 1000
link/ether c0:3f:d5:61:1a:0f brd ff:ff:ff:ff:ff:ff
inet 10.43.70.1/16 brd 10.43.255.255 scope global br0
   valid_lft forever preferred_lft forever
inet6 2001:858:107:1:c23f:d5ff:fe61:1a0f/64 scope global mngtmpaddr dynamic 
   valid_lft 86394sec preferred_lft 14394sec
inet6 fe80::c23f:d5ff:fe61:1a0f/64 scope link 
   valid_lft forever preferred_lft forever
4: vnet0:  mtu 1500 qdisc pfifo_fast master 
br0 state UNKNOWN group default qlen 1000
link/ether fe:54:00:67:36:c8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe67:36c8/64 scope link 
   valid_lft forever preferred_lft forever

$ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source   destination 

Chain FORWARD (policy ACCEPT)
target prot opt source   destination 

Chain OUTPUT (policy ACCEPT)
target prot opt source   destination 

brctl show
bridge name bridge id STP enabled interfaces
br0 8000.c03fd5611a0f no  eth2
  vnet0

 after "docker images" (yes, the only docker command I ran), and from this 
point on networking in the VM is dead: 

$ ip a s
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
2: eth2:  mtu 1500 qdisc pfifo_fast master br0 
state UP group default qlen 1000
link/ether c0:3f:d5:61:1a:0f brd ff:ff:ff:ff:ff:ff
3: br0:  mtu 1500 qdisc noqueue state UP group 
default qlen 1000
link/ether c0:3f:d5:61:1a:0f brd ff:ff:ff:ff:ff:ff
inet 10.43.70.1/16 brd 10.43.255.255 scope global br0
   valid_lft forever preferred_lft forever
inet6 2001:858:107:1:c23f:d5ff:fe61:1a0f/64 scope global mngtmpaddr dynamic 
   valid_lft 86397sec preferred_lft 14397sec
inet6 fe80::c23f:d5ff:fe61:1a0f/64 scope link 
   valid_lft forever preferred_lft forever
4: vnet0:  mtu 1500 qdisc pfifo_fast master 
br0 state UNKNOWN group default qlen 1000
link/ether fe:54:00:67:36:c8 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe67:36c8/64 scope link 
   valid_lft forever preferred_lft forever
5: docker0:  mtu 1500 qdisc noqueue state 
DOWN group default 
link/ether 02:42:52:94:20:6b brd ff:ff:ff:ff:ff:ff
inet 

Bug#865975: docker.io breaks (bridged) network for VMs

[Antoine Beaupre]

Control: tags -1 +moreinfo

On Mon, Jun 26, 2017 at 11:16:56AM +0200, Roland Kammerer wrote:
> Package: docker.io
> Version: 1.13.1~ds1-2
> Severity: critical
> Tags: upstream
> Justification: breaks unrelated software
> 
> Dear Maintainer,
> 
> * What led up to the situation?
> Any docker command like "docker images"
> 
> * What was the outcome of this action?
> Network breaks for all libvirt VMs (i.e., they are not able to ping each
> other, or public domains, and after a reboot they do not get an IP via
> dhcp). The VMs are connected to a bridge (br0):
> 
> # The loopback network interface
> auto lo
> iface lo inet loopback
> 
> # The primary network interface
> #allow-hotplug eth2
> #iface eth2 inet dhcp
> # This is an autoconfigured IPv6 interface
> #iface eth2 inet6 auto
> 
> auto br0
> iface br0 inet dhcp
>   bridge_ports eth2
>   # bridge_stp on
>   bridge_maxwait 0
>   bridge_fd 0
> 
> I don't have any firewall/iptables rules on my machine.
> 
> * What outcome did you expect instead?
> That networking still works (as it did with older docker versions).

Which versions were working?

> The situation can be fixed via "iptables -I FORWARD -i br0 -o br0 -j ACCEPT".
> Before that I saw "Chain FORWARD (policy drop 3493 packests, 829K bytes)". 
> Therefore, I assume that docker messages with the chains.

It would be useful if you could provide a step by step set of
instructions on how to reproduce this. For example, what type of VMs
does libvirt use? KVM? How is docker configured? What commands are ran
exactly?

Thanks,

A.


signature.asc
Description: PGP signature

Bug#865975: docker.io breaks (bridged) network for VMs

[Roland Kammerer]

Package: docker.io
Version: 1.13.1~ds1-2
Severity: critical
Tags: upstream
Justification: breaks unrelated software

Dear Maintainer,

* What led up to the situation?
Any docker command like "docker images"

* What was the outcome of this action?
Network breaks for all libvirt VMs (i.e., they are not able to ping each
other, or public domains, and after a reboot they do not get an IP via
dhcp). The VMs are connected to a bridge (br0):

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug eth2
#iface eth2 inet dhcp
# This is an autoconfigured IPv6 interface
#iface eth2 inet6 auto

auto br0
iface br0 inet dhcp
bridge_ports eth2
# bridge_stp on
bridge_maxwait 0
bridge_fd 0

I don't have any firewall/iptables rules on my machine.

* What outcome did you expect instead?
That networking still works (as it did with older docker versions).

The situation can be fixed via "iptables -I FORWARD -i br0 -o br0 -j ACCEPT".
Before that I saw "Chain FORWARD (policy drop 3493 packests, 829K bytes)". 
Therefore, I assume that docker messages with the chains.


-- System Information:
Debian Release: 9.0
  APT prefers stable
  APT policy: (700, 'stable'), (600, 'unstable'), (550, 'experimental'), (500, 
'stable-debug')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages docker.io depends on:
ii  adduser  3.115
ii  containerd   0.2.3+git20170126.85.aa8187d~ds1-1
ii  golang-libnetwork0.8.0-dev.2+git20170202.599.45b4086-1
ii  init-system-helpers  1.48
ii  iptables 1.6.0+snapshot20161117-6
ii  libapparmor1 2.11.0-3
ii  libc62.24-11
ii  libdevmapper1.02.1   2:1.02.137-2
ii  libsqlite3-0 3.16.2-4
ii  libsystemd0  232-24
ii  lsb-base 9.20161125
ii  runc 1.0.0~rc2+git20170201.133.9df8b30-1

Versions of packages docker.io recommends:
ii  ca-certificates  20161130+nmu1
ii  cgroupfs-mount   1.3
ii  git  1:2.11.0-3
ii  xz-utils 5.2.2-1.2+b1

Versions of packages docker.io suggests:
pn  aufs-tools   
pn  btrfs-progs  
ii  debootstrap  1.0.89
pn  docker-doc   
pn  rinse
pn  zfs-fuse | zfsutils  

-- Configuration Files:
/etc/default/docker changed:
DOCKER_OPTS="--storage-driver=devicemapper -H unix:///var/run/docker.sock"


-- no debconf information