Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Tianon Gravi]

severity 843530 important
thanks

On 8 November 2016 at 09:52, Tianon Gravi  wrote:
> Ouch, looks like we're now hitting
> https://github.com/opencontainers/runc/issues/1175, which doesn't
> appear to have a Docker or runc workaround yet (although adding
> "systemd.legacy_systemd_cgroup_controller=yes" to your system boot
> parameters should do the trick for now). :(

So, following that thread now, it looks like systemd upstream has
reverted that particular change (which likely isn't quite in Debian
yet, at least not stretch), but they do plan to reintroduce it again
later.

Given that there's not much more we (Debian) can directly do to
fix/overcome this beyond documenting it in README.Debian (which I plan
to commit in Git shortly), I've decreased the severity of this report.
Hopefully the runc maintainers will figure out a reasonable way to
support the unified hierarchy sometime in the near future. :(


♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Stef Walter]

On 08.11.2016 18:52, Tianon Gravi wrote:
> Ouch, looks like we're now hitting
> https://github.com/opencontainers/runc/issues/1175, which doesn't
> appear to have a Docker or runc workaround yet (although adding
> "systemd.legacy_systemd_cgroup_controller=yes" to your system boot
> parameters should do the trick for now). :(

Thanks. Good to know.

In case you're interested: In the Cockpit project we actively integrate
Linux (including Debian) and here's a page that will track when this
specific issue occurs during integration testing:

https://github.com/cockpit-project/cockpit/issues/5340

Stef

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Tianon Gravi]

On 8 November 2016 at 01:09, Stef Walter  wrote:
> Nov 08 04:04:29 unassigned-hostname docker[5826]:
> time="2016-11-08T04:04:29-05:00" level=error msg="containerd: start
> container" error="oci runtime error: could not synchronise with
> container process: no subsystem for mount"
> id=4be1274a79c35a25c0ef70a866f4d20b03e5a7bf3cf60131ae49ef0ef11bfb59
> Nov 08 04:04:29 unassigned-hostname docker[5826]:
> time="2016-11-08T04:04:29.430453214-05:00" level=error msg="Handler for
> POST
> /v1.23/containers/4be1274a79c35a25c0ef70a866f4d20b03e5a7bf3cf60131ae49ef0ef11bfb59/start
> returned error: rpc error: code = 2 desc = \"oci runtime error: could
> not synchronise with container process: no subsystem for mount\""

Ouch, looks like we're now hitting
https://github.com/opencontainers/runc/issues/1175, which doesn't
appear to have a Docker or runc workaround yet (although adding
"systemd.legacy_systemd_cgroup_controller=yes" to your system boot
parameters should do the trick for now). :(


♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Stef Walter]

On 07.11.2016 16:44, Tianon Gravi wrote:
> On 7 November 2016 at 05:34, Stef Walter  wrote:
>> The docker package is unfortunately currently broken. It fails to run
>> containers and instead produces the following message:
>>
>> rpc error: code = 2 desc = "oci runtime error: could not synchronise with 
>> container process: no subsystem for mount"
>>
>> This can be reproduced by running something like:
>>
>> docker run -ti busybox /bin/sh
>>
>> Or any similar command.
> 
> Can you please provide the relevant log lines from the daemon?
> 
> (Either "/var/log/docker.log" or "journalctl -u docker.service")

Sure thing. Here you go. The full file is attached. The relevant lines are:

Nov 08 04:04:29 unassigned-hostname docker[5826]:
time="2016-11-08T04:04:29-05:00" level=error msg="containerd: start
container" error="oci runtime error: could not synchronise with
container process: no subsystem for mount"
id=4be1274a79c35a25c0ef70a866f4d20b03e5a7bf3cf60131ae49ef0ef11bfb59
Nov 08 04:04:29 unassigned-hostname docker[5826]:
time="2016-11-08T04:04:29.430453214-05:00" level=error msg="Handler for
POST
/v1.23/containers/4be1274a79c35a25c0ef70a866f4d20b03e5a7bf3cf60131ae49ef0ef11bfb59/start
returned error: rpc error: code = 2 desc = \"oci runtime error: could
not synchronise with container process: no subsystem for mount\""

In case it helps, here is a compressed qemu/libvirt qcow2 image which
replicates this issue:

https://fedorapeople.org/groups/cockpit/images/debian-unstable-e20afebbfad06c2ba3d4573c71ec6ece14ead4a6.qcow2.xz

Stef
-- Logs begin at Tue 2016-11-08 02:16:38 EST, end at Tue 2016-11-08 04:04:29 
EST. --
Nov 08 04:04:27 unassigned-hostname systemd[1]: Starting Docker Application 
Container Engine...
Nov 08 04:04:27 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:27.737449987-05:00" level=info msg="New containerd 
process, pid: 5829\n"
Nov 08 04:04:28 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:28.767716036-05:00" level=info msg="[graphdriver] using 
prior storage driver \"overlay\""
Nov 08 04:04:28 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:28.797969887-05:00" level=info msg="Graph migration to 
content-addressability took 0.00 seconds"
Nov 08 04:04:28 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:28.808881857-05:00" level=info msg="Firewalld running: 
false"
Nov 08 04:04:28 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:28.924174351-05:00" level=info msg="Default bridge 
(docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can 
be used to set a preferred IP address"
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.013499004-05:00" level=warning msg="Your kernel does 
not support swap memory limit."
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.017007389-05:00" level=info msg="Loading containers: 
start."
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.018738290-05:00" level=info msg="Loading containers: 
done."
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.019102560-05:00" level=info msg="Daemon has completed 
initialization"
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.019153274-05:00" level=info msg="Docker daemon" 
commit=b9f10c9 graphdriver=overlay version=1.11.2
Nov 08 04:04:29 unassigned-hostname systemd[1]: Started Docker Application 
Container Engine.
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.100046437-05:00" level=info msg="API listen on 
/var/run/docker.sock"
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29-05:00" level=error msg="containerd: start container" 
error="oci runtime error: could not synchronise with container process: no 
subsystem for mount" 
id=4be1274a79c35a25c0ef70a866f4d20b03e5a7bf3cf60131ae49ef0ef11bfb59
Nov 08 04:04:29 unassigned-hostname docker[5826]: 
time="2016-11-08T04:04:29.430453214-05:00" level=error msg="Handler for POST 
/v1.23/containers/4be1274a79c35a25c0ef70a866f4d20b03e5a7bf3cf60131ae49ef0ef11bfb59/start
 returned error: rpc error: code = 2 desc = \"oci runtime error: could not 
synchronise with container process: no subsystem for mount\""

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[dan]

For anyone needing a workaround. This bug also hit me this morning. I
looked through a bit of the code in runc/libcontainer and it looks like it
is caused by a cgroup issue.

I took an educated guess and downgraded systemd from 232-2 to 231-9 (which
is still available in stretch) and now docker works again.

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Tianon Gravi]

On 7 November 2016 at 05:34, Stef Walter  wrote:
> The docker package is unfortunately currently broken. It fails to run
> containers and instead produces the following message:
>
> rpc error: code = 2 desc = "oci runtime error: could not synchronise with 
> container process: no subsystem for mount"
>
> This can be reproduced by running something like:
>
> docker run -ti busybox /bin/sh
>
> Or any similar command.

Can you please provide the relevant log lines from the daemon?

(Either "/var/log/docker.log" or "journalctl -u docker.service")

♥,
- Tianon
  4096R / B42F 6819 007F 00F8 8E36  4FD4 036A 9C25 BF35 7DD4

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Tom Marble]

I can confirm I am hitting this exact same bug (same system
information).

--Tom

Bug#843530: docker.io: docker broken: oci runtime error: could not synchronize with container process

[Stef Walter]

Package: docker.io
Version: 1.11.2~ds1-6
Severity: grave
Justification: renders package unusable

The docker package is unfortunately currently broken. It fails to run
containers and instead produces the following message:

rpc error: code = 2 desc = "oci runtime error: could not synchronise with 
container process: no subsystem for mount"

This can be reproduced by running something like:

docker run -ti busybox /bin/sh

Or any similar command.

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 4.8.0-1-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages docker.io depends on:
ii  adduser  3.115
ii  containerd   0.2.1~ds1-3
ii  init-system-helpers  1.46
ii  iptables 1.6.0-4
ii  libapparmor1 2.10.95-5
ii  libc62.24-5
ii  libdevmapper1.02.1   2:1.02.133-1
ii  libsqlite3-0 3.15.1-1
ii  libsystemd0  232-2
ii  runc 0.1.1+dfsg1-1

Versions of packages docker.io recommends:
ii  ca-certificates  20161102
ii  cgroupfs-mount   1.3
ii  git  1:2.10.2-2
ii  xz-utils 5.2.2-1.2

Versions of packages docker.io suggests:
pn  aufs-tools   
pn  btrfs-progs  
ii  debootstrap  1.0.86
pn  docker-doc   
pn  rinse
pn  zfs-fuse | zfsutils  

-- no debconf information