bug#34333: Docker daemon failing to start on boot
Hi, I am having an issue with the docker daemon as a service in guixsd.
It seems that dockerd will not start on boot, but it will successfully
start after running "guix system reconfigure".
Here is my config.scm:
(use-modules (gnu)
(gnu system nss)
(gnu services)
(gnu services docker))
(use-service-modules desktop)
(use-package-modules certs gnome)
(operating-system
(host-name "guixsd")
(timezone "Europe/Oslo")
(locale "en_US.utf8")
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")))
(file-systems (cons (file-system
(device (file-system-label "my-root"))
(mount-point "/")
(type "ext4"))
%base-file-systems))
(users (cons (user-account
(name "allana")
(group "users")
(supplementary-groups '("wheel" "docker" "netdev"
"audio" "video"))
(home-directory "/home/allana"))
%base-user-accounts))
;; This is where we specify system-wide packages.
(packages (cons* nss-certs ;for HTTPS access
gvfs ;for user mounts
%base-packages))
(services (cons* (console-keymap-service "no-latin1")
(gnome-desktop-service)
(service docker-service-type)
%desktop-services))
;; Allow resolution of '.local' host names with mDNS.
(name-service-switch %mdns-host-lookup-nss))
After booting:
allana@guixsd ~$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is
the docker daemon running?
allana@guixsd ~$ sudo herd status dockerd
Password:
Status of dockerd:
It is stopped.
It is enabled.
Provides (dockerd).
Requires (containerd file-system-/sys/fs/cgroup/blkio
file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset
file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory).
Conflicts with ().
Will be respawned.
allana@guixsd ~$ sudo herd start dockerd
Service dockerd could not be started.
herd: failed to start service dockerd
But if I run:
allana@guixsd ~$ sudo guix reconfigure config.scm
...
allana@guixsd ~$ docker ps
CONTAINER IDIMAGE COMMAND CREATED
STATUS PORTS NAMES
allana@guixsd ~$ sudo herd status dockerd
Password:
Status of dockerd:
It is started.
Running value is 2123.
It is enabled.
Provides (dockerd).
Requires (containerd file-system-/sys/fs/cgroup/blkio
file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset
file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory).
Conflicts with ().
Will be respawned.
Dump from /var/log/messages:
Feb 5 10:06:37 localhost -- MARK --
Feb 5 10:15:47 localhost nscd: 319 monitored file `/etc/hosts` was
deleted, removing watch
Feb 5 10:15:47 localhost nscd: 319 monitored file `/etc/hosts` was
created, adding watch
Feb 5 10:15:47 localhost nscd: 319 monitored file `/etc/hosts` was
written to
Feb 5 10:15:47 localhost nscd: 319 monitored file `/etc/services` was
deleted, removing watch
Feb 5 10:15:47 localhost nscd: 319 monitored file `/etc/services` was
created, adding watch
Feb 5 10:15:47 localhost nscd: 319 monitored file `/etc/services` was
written to
Feb 5 10:15:47 localhost NetworkManager[355]: [1549358147.5315]
settings: hostname changed from (none) to "guixsd"
Feb 5 10:15:48 localhost shepherd[1]: Evaluating user expression (let*
((services (map primitive-load (?))) # ?) ?).
Feb 5 10:15:48 localhost shepherd[1]: Service user-homes could not be
started.
Feb 5 10:15:48 localhost shepherd[1]: Service term-auto could not be
started.
Feb 5 10:15:49 localhost vmunix: [ 2965.192083] bridge: filtering via
arp/ip/ip6tables is no longer available by default. Update your scripts
to load br_netfilter if you need this.
Feb 5 10:15:49 localhost vmunix: [ 2965.194289] Bridge firewalling
registered
Feb 5 10:15:49 localhost vmunix: [ 2965.405743] Initializing XFRM
netlink socket
Feb 5 10:15:49 localhost NetworkManager[355]: [1549358149.2361]
manager: (docker0): new Bridge device
(/org/freedesktop/NetworkManager/Devices/3)
Feb 5 10:15:49 localhost avahi-daemon[361]: Joining mDNS multicast
group on interface docker0.IPv4 with address 172.17.0.1.
Feb 5 10:15:49 localhost NetworkManager[355]: [1549358149.3174]
device (docker0): state change: unmanaged -> unavailable (reason
'connection-assumed', internal state 'external')
Feb 5 10:15:49 localhost vmunix: [ 2965.500493] IPv6:
ADDRCONF(NETDEV_UP): docker0: link is not ready
Feb 5 10:15:49 localhost avahi-daemon[361]: New relevant interface
docker0.IPv4 for mDNS.
Feb 5 10:15:49 localhost NetworkManager[355]: [1549358149.3195]
keyfile: add connection in-memory
(33e2c9e2-
bug#34333: Docker daemon failing to start on boot
Hello, al...@adair.io skribis: > Hi, I am having an issue with the docker daemon as a service in > guixsd. It seems that dockerd will not start on boot, but it will > successfully start after running "guix system reconfigure". [...] > After booting: > > > > allana@guixsd ~$ docker ps > Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is > the docker daemon running? > allana@guixsd ~$ sudo herd status dockerd > Password: > Status of dockerd: > It is stopped. > It is enabled. This is what happens at boot time: > Feb 5 10:16:51 localhost shepherd[1]: Service containerd has been [...] > Feb 5 10:16:51 localhost NetworkManager[353]: > [1549358211.3005] NetworkManager (version 1.8.4) is starting... (for > the first time) [...] > [1549358211.9031] manager: NetworkManager state is now > CONNECTED_GLOBAL > Feb 5 10:16:52 localhost dbus-daemon[350]: [system] Successfully > activated service 'org.freedesktop.PolicyKit1' > Feb 5 10:16:54 localhost ntpd[354]: Listen normally on 4 enp0s3 > 10.0.2.15:123 > Feb 5 10:16:54 localhost ntpd[354]: Listen normally on 5 enp0s3 > [fe80::e882:9a4a:3a68:9d76%2]:123 > Feb 5 10:16:56 localhost shepherd[1]: Service dockerd could not be > started. Danny, could it be that the ‘docker’ service should depend on ‘networking’? FWIW, “make check-system TESTS=docker” passes for me. Thanks, Ludo’.
bug#34333: Docker daemon failing to start on boot
On Fri, 08 Feb 2019 22:55:48 +0100 Ludovic Courtès wrote: > Danny, could it be that the ‘docker’ service should depend on > ‘networking’? Might be the case. They certainly muck around a lot with networking, so it can't hurt to wait until host networking is set up. I've added it--let's see. Allan, can you guix pull and guix reconfigure and then try again? pgpo1iiTVTeR4.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
On 2019-02-11 10:46, Danny Milosavljevic wrote: On Fri, 08 Feb 2019 22:55:48 +0100 Ludovic Courtès wrote: Danny, could it be that the ‘docker’ service should depend on ‘networking’? Might be the case. They certainly muck around a lot with networking, so it can't hurt to wait until host networking is set up. I've added it--let's see. Allan, can you guix pull and guix reconfigure and then try again? I guix pulled and guix system reconfigured. After a successful reconfigure and reboot, the service was still not started. -- Allan Adair +47 468 12 135 al...@adair.io http://allan.adair.io
bug#34333: Docker daemon failing to start on boot
On Mon, 11 Feb 2019 13:11:33 + Allan Adair wrote: > I guix pulled and guix system reconfigured. After a successful > reconfigure and reboot, the service was still not started. Hmm, is the "containerd" process running? (pidof containerd) pgpD8FksabhhE.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Hi Allan, I've added some more requirements--let's see. Can you guix pull and guix reconfigure and then try again once more? pgpdLPEyf4j05.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
On 2019-02-11 17:31, Danny Milosavljevic wrote: Hi Allan, I've added some more requirements--let's see. Can you guix pull and guix reconfigure and then try again once more? Still no luck. Do you face the same issue? -- Allan Adair +47 468 12 135 al...@adair.io http://allan.adair.io
bug#34333: Docker daemon failing to start on boot
Hi Allan, On Tue, 12 Feb 2019 09:05:10 + Allan Adair wrote: > On 2019-02-11 17:31, Danny Milosavljevic wrote: > > Hi Allan, > > > > I've added some more requirements--let's see. > > > > Can you guix pull and guix reconfigure and then try again once more? > > Still no luck. Do you face the same issue? I don't know since my machine has not finished updating after the recent staging to master merge. pgpiWmSuiTG4M.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Hey Danny.
I'm curious to know if the dockerd service type works for you. It does
not work for me, but maybe I am doing something wrong. Below is my
current config.scm:
(use-modules (gnu)
(gnu system nss)
(gnu services))
(use-service-modules desktop docker)
(use-package-modules certs gnome)
(operating-system
(host-name "guixsd")
(timezone "Europe/Oslo")
(locale "en_US.utf8")
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")))
(file-systems (cons (file-system
(device (file-system-label "my-root"))
(mount-point "/")
(type "ext4"))
%base-file-systems))
(users (cons (user-account
(name "allana")
(group "users")
(supplementary-groups '("wheel"
"docker"
"netdev"
"audio"
"video"))
(home-directory "/home/allana"))
%base-user-accounts))
;; This is where we specify system-wide packages.
(packages (cons* nss-certs ;for HTTPS access
gvfs ;for user mounts
%base-packages))
(services (cons* (console-keymap-service "no-latin1")
(gnome-desktop-service)
(service docker-service-type)
%desktop-services))
;; Allow resolution of '.local' host names with mDNS.
(name-service-switch %mdns-host-lookup-nss))
After a "guix system reconfigure", it works -- meaning that the dockerd
service starts. But when booting in the future, the dockerd
daemon never starts. I am however able to execute "sudo herd start
dockerd" after booting because I have included dockerd in my user
profile. I can also "guix system reconfigure" at this point to start
the dockerd service, but then at the next boot it will still not start.
Danny Milosavljevic writes:
> Hi Allan,
>
> On Tue, 12 Feb 2019 09:05:10 +
> Allan Adair wrote:
>
>> On 2019-02-11 17:31, Danny Milosavljevic wrote:
>> > Hi Allan,
>> >
>> > I've added some more requirements--let's see.
>> >
>> > Can you guix pull and guix reconfigure and then try again once more?
>>
>> Still no luck. Do you face the same issue?
>
> I don't know since my machine has not finished updating after the
> recent staging to master merge.
--
Allan Adair
http://allan.adair.io
bug#34333: Docker daemon failing to start on boot
On Wed, 27 Feb 2019 15:17:14 +0100 Allan Adair wrote: > After a "guix system reconfigure", it works -- meaning that the > dockerd service starts. But when booting in the future, the dockerd > daemon never starts. I am however able to execute "sudo herd start > dockerd" after booting because I have included dockerd in my user > profile. I can also "guix system reconfigure" at this point to start > the dockerd service, but then at the next boot it will still not > start. I can confirm this behaviour, though I haven't yet investigated it further. Björn pgpHy1h0kMyOp.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
On Wed, Feb 27, 2019 at 04:53:27PM +0100, Björn Höfling wrote: > > After a "guix system reconfigure", it works -- meaning that the > > dockerd service starts. But when booting in the future, the dockerd > > daemon never starts. I am however able to execute "sudo herd start > > dockerd" after booting because I have included dockerd in my user > > profile. I can also "guix system reconfigure" at this point to start > > the dockerd service, but then at the next boot it will still not > > start. > > I can confirm this behaviour, though I haven't yet investigated it > further. Could this be yet another manifestation of https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34580 ? Services not started on boot? Andreas
bug#34333: Docker daemon failing to start on boot
Andreas Enge writes: > On Wed, Feb 27, 2019 at 04:53:27PM +0100, Björn Höfling wrote: >> > After a "guix system reconfigure", it works -- meaning that the >> > dockerd service starts. But when booting in the future, the dockerd >> > daemon never starts. I am however able to execute "sudo herd start >> > dockerd" after booting because I have included dockerd in my user >> > profile. I can also "guix system reconfigure" at this point to start >> > the dockerd service, but then at the next boot it will still not >> > start. >> >> I can confirm this behaviour, though I haven't yet investigated it >> further. > > Could this be yet another manifestation of >https://debbugs.gnu.org/cgi/bugreport.cgi?bug=34580 ? > Services not started on boot? > > Andreas I'm not so sure. One thing that I am unable to do is "herd start dockerd". -- Allan Adair http://allan.adair.io
bug#34333: Docker daemon failing to start on boot
On Fri, Mar 01, 2019 at 09:58:20AM +0100, Allan Adair wrote: > >> > I am however able to execute "sudo herd start > >> > dockerd" after booting > > I'm not so sure. One thing that I am unable to do is "herd start > dockerd". You mean, you are not able to start it via sudo su - herd start dockerd ? It is normal that you cannot start services as a normal user. Andreas
bug#34333: Docker daemon failing to start on boot
On Fri, 1 Mar 2019 14:09:32 +0100 Andreas Enge wrote: > On Fri, Mar 01, 2019 at 09:58:20AM +0100, Allan Adair wrote: > > >> > I am however able to execute "sudo herd start > > >> > dockerd" after booting > > > > I'm not so sure. One thing that I am unable to do is "herd start > > dockerd". > > You mean, you are not able to start it via > sudo su - > herd start dockerd > ? > > It is normal that you cannot start services as a normal user. I wondered about that too. For me, a "sudo herd start dockerd" works. After that, I can use the docker-cli tools to work with docker images and containers. They deploy and run fine. Björn pgpummMjXGKBA.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Björn Höfling writes: > On Fri, 1 Mar 2019 14:09:32 +0100 > Andreas Enge wrote: > >> On Fri, Mar 01, 2019 at 09:58:20AM +0100, Allan Adair wrote: >> > >> > I am however able to execute "sudo herd start >> > >> > dockerd" after booting >> > >> > I'm not so sure. One thing that I am unable to do is "herd start >> > dockerd". >> >> You mean, you are not able to start it via >> sudo su - >> herd start dockerd >> ? >> >> It is normal that you cannot start services as a normal user. > > I wondered about that too. > > For me, a "sudo herd start dockerd" works. After that, I can use the > docker-cli tools to work with docker images and containers. They deploy > and run fine. > > Björn After a fresh boot: allana@guixsd ~$ docker ps Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? allana@guixsd ~$ sudo herd status dockerd Password: Status of dockerd: It is stopped. It is enabled. Provides (dockerd). Requires (containerd dbus-system elogind file-system-/sys/fs/cgroup/blkio file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory networking udev). Conflicts with (). Will be respawned. allana@guixsd ~$ sudo herd start dockerd Service dockerd could not be started. herd: failed to start service dockerd -- Allan Adair http://allan.adair.io
bug#34333: Docker daemon failing to start on boot
Hi, On Fri, 01 Mar 2019 14:50:39 +0100 Allan Adair wrote: > After a fresh boot: > > allana@guixsd ~$ docker ps > Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the > docker daemon running? > allana@guixsd ~$ sudo herd status dockerd > Password: > Status of dockerd: > It is stopped. > It is enabled. > Provides (dockerd). > Requires (containerd dbus-system elogind file-system-/sys/fs/cgroup/blkio > file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset > file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory > networking udev). > Conflicts with (). > Will be respawned. > allana@guixsd ~$ sudo herd start dockerd > Service dockerd could not be started. > herd: failed to start service dockerd In a way that's good that this failure is reproducible so well. "herd start dockerd" would do: $(guix build docker)/bin/dockerd -p /var/run/docker.pid For debugging, can you please try invoking the latter instead of "herd start dockerd" ? After that, please try $(guix build docker)/bin/dockerd -p /var/run/docker.pid -D pgpwHg1iO57oU.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Hi Danny.
Sorry for the late response. I was offline for the last week or so.
I ended up having to repeat the first command with sudo
privileges. Please see below.
Danny Milosavljevic writes:
> Hi,
>
> On Fri, 01 Mar 2019 14:50:39 +0100
> Allan Adair wrote:
>
>> After a fresh boot:
>>
>> allana@guixsd ~$ docker ps
>> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
>> docker daemon running?
>> allana@guixsd ~$ sudo herd status dockerd
>> Password:
>> Status of dockerd:
>> It is stopped.
>> It is enabled.
>> Provides (dockerd).
>> Requires (containerd dbus-system elogind file-system-/sys/fs/cgroup/blkio
>> file-system-/sys/fs/cgroup/cpu file-system-/sys/fs/cgroup/cpuset
>> file-system-/sys/fs/cgroup/devices file-system-/sys/fs/cgroup/memory
>> networking udev).
>> Conflicts with ().
>> Will be respawned.
>> allana@guixsd ~$ sudo herd start dockerd
>> Service dockerd could not be started.
>> herd: failed to start service dockerd
>
> In a way that's good that this failure is reproducible so well.
>
> "herd start dockerd" would do:
>
> $(guix build docker)/bin/dockerd -p /var/run/docker.pid
>
> For debugging, can you please try invoking the latter instead
> of "herd start dockerd" ?
>
> After that, please try
>
> $(guix build docker)/bin/dockerd -p /var/run/docker.pid -D
allana@guixsd ~$ $(guix build docker)/bin/dockerd -p /var/run/docker.pid
substitute: updating substitutes from 'https://ci.guix.info'... 100.0%
substitute: updating substitutes from 'https://ci.guix.info'... 100.0%
substitute: updating substitutes from 'https://ci.guix.info'... 100.0%
substitute: updating substitutes from 'https://ci.guix.info'... 100.0%
93.1 MB will be downloaded:
/gnu/store/fqzdfac28a7h0wsxz6wdgjym491klbqh-docker-18.09.2
/gnu/store/5g8j76dqjafmah82kd30fc1d1bf74zib-iptables-1.6.2
/gnu/store/cqd3yimzpj0r29l3dyh8xcqbx70cbanc-xfsprogs-4.20.0
/gnu/store/hz2mq9wpddaifn2sij5msswhzzf8136b-iproute2-4.20.0
/gnu/store/pg3jsf0sp6p66lr6r0bfyz1lx2n6jb2b-containerd-1.2.4
The following graft will be made:
/gnu/store/a9frz994s0qiq6iay6yhfii8swvz6yyp-docker-18.09.2.drv
substituting /gnu/store/pg3jsf0sp6p66lr6r0bfyz1lx2n6jb2b-containerd-1.2.4...
downloading from
https://ci.guix.info/nar/gzip/pg3jsf0sp6p66lr6r0bfyz1lx2n6jb2b-containerd-1.2.4...
containerd-1.2.4 35.5MiB1.5MiB/s 00:23 [##] 100.0%
substituting /gnu/store/y3zh0rhc25vakfl4q49sxs8kzcqm8bhi-libnftnl-1.1.2...
downloading from
https://ci.guix.info/nar/gzip/y3zh0rhc25vakfl4q49sxs8kzcqm8bhi-libnftnl-1.1.2...
libnftnl-1.1.2 85KiB1.8MiB/s 00:00 [##] 100.0%
substituting /gnu/store/cqd3yimzpj0r29l3dyh8xcqbx70cbanc-xfsprogs-4.20.0...
downloading from
https://ci.guix.info/nar/gzip/cqd3yimzpj0r29l3dyh8xcqbx70cbanc-xfsprogs-4.20.0...
xfsprogs-4.20.0 2.3MiB 1.4MiB/s 00:02 [##] 100.0%
substituting /gnu/store/5g8j76dqjafmah82kd30fc1d1bf74zib-iptables-1.6.2...
downloading from
https://ci.guix.info/nar/gzip/5g8j76dqjafmah82kd30fc1d1bf74zib-iptables-1.6.2...
iptables-1.6.2 555KiB 2.4MiB/s 00:00 [##] 100.0%
substituting /gnu/store/hz2mq9wpddaifn2sij5msswhzzf8136b-iproute2-4.20.0...
downloading from
https://ci.guix.info/nar/gzip/hz2mq9wpddaifn2sij5msswhzzf8136b-iproute2-4.20.0...
iproute2-4.20.0 1.0MiB 3.3MiB/s 00:00 [##] 100.0%
substituting /gnu/store/fqzdfac28a7h0wsxz6wdgjym491klbqh-docker-18.09.2...
downloading from
https://ci.guix.info/nar/gzip/fqzdfac28a7h0wsxz6wdgjym491klbqh-docker-18.09.2...
docker-18.09.2 49.5MiB 1.5MiB/s 00:33 [##] 100.0%
applying 1 graft for
/gnu/store/a9frz994s0qiq6iay6yhfii8swvz6yyp-docker-18.09.2.drv...
grafting '/gnu/store/fqzdfac28a7h0wsxz6wdgjym491klbqh-docker-18.09.2' ->
'/gnu/store/2vf8f8bky5jlifghqgl92n7a3vx6icid-docker-18.09.2'...
successfully built
/gnu/store/a9frz994s0qiq6iay6yhfii8swvz6yyp-docker-18.09.2.drv
chmod /var/lib/docker: operation not permitted
allana@guixsd ~$ sudo $(guix build docker)/bin/dockerd -p /var/run/docker.pid
Password:
INFO[2019-03-11T09:55:18.870049018+01:00] parsed scheme: "unix"
module=grpc
INFO[2019-03-11T09:55:18.870157978+01:00] scheme "unix" not registered,
fallback to default scheme module=grpc
INFO[2019-03-11T09:55:18.870220789+01:00] parsed scheme: "unix"
module=grpc
INFO[2019-03-11T09:55:18.870263818+01:00] scheme "unix" not registered,
fallback to default scheme module=grpc
INFO[2019-03-11T09:55:18.870988104+01:00] ccResolverWrapper: sending new
addresses to cc: [{unix:///run/containerd/containerd.sock 0 }]
module=grpc
INFO[2019-03-11T09:55:18.871011336+01:00] ClientConn switching balancer to
"pick_first" module=grpc
INFO[2019-03-11T09:55:18.871121368+01:00] ccResolverWrapper: sending new
addresses to cc: [{unix:///run/containerd/containerd.sock 0 }]
module=grpc
INFO[2019-03-11T09:55:18.871135562+01
bug#34333: Docker daemon failing to start on boot
Hi Allan, On Mon, 11 Mar 2019 09:59:19 +0100 Allan Adair wrote: > Sorry for the late response. I was offline for the last week or so. No problem! > > I ended up having to repeat the first command with sudo > privileges. Please see below. Yes, so that looks as if it works fine. What's the difference to a failed start by herd (log file in /var/log/docker.log) ? pgpwNBZlYI8D2.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Hi Danny.
Danny Milosavljevic writes:
> Hi Allan,
>
> On Mon, 11 Mar 2019 09:59:19 +0100
> Allan Adair wrote:
>
>> Sorry for the late response. I was offline for the last week or so.
>
> No problem!
>
>>
>> I ended up having to repeat the first command with sudo
>> privileges. Please see below.
>
> Yes, so that looks as if it works fine. What's the difference to a failed
> start by herd (log file in /var/log/docker.log) ?
I have never actually been able to start the dockerd service using herd
explicitly. After booting:
allana@guixsd ~$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
allana@guixsd ~$ cat /var/log/docker.log
time="2019-03-18T10:23:30.462181353+01:00" level=warning msg="Error while
setting daemon root propagation, this is not generally critical but may cause
some functionality to not work or fallback to less desirable behavior"
dir=/var/lib/docker error="error writing file to signal mount cleanup on
shutdown: open /var/run/docker/unmount-on-shutdown: no such file or directory"
time="2019-03-18T10:23:30.46519+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T10:23:30.466019010+01:00" level=info msg="scheme \"unix\" not
registered, fallback to default scheme" module=grpc
time="2019-03-18T10:23:30.466291192+01:00" level=info msg="ccResolverWrapper:
sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0
}]" module=grpc
time="2019-03-18T10:23:30.466315303+01:00" level=info msg="ClientConn switching
balancer to \"pick_first\"" module=grpc
time="2019-03-18T10:23:30.466349982+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc00012d090, CONNECTING" module=grpc
time="2019-03-18T10:23:30.46736+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc00012d090, READY" module=grpc
time="2019-03-18T10:23:30.467531354+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T10:23:30.467544289+01:00" level=info msg="scheme \"unix\" not
registered, fallback to default scheme" module=grpc
time="2019-03-18T10:23:30.467972429+01:00" level=info msg="ccResolverWrapper:
sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0
}]" module=grpc
time="2019-03-18T10:23:30.467991848+01:00" level=info msg="ClientConn switching
balancer to \"pick_first\"" module=grpc
time="2019-03-18T10:23:30.468161326+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc00012d380, CONNECTING" module=grpc
time="2019-03-18T10:23:30.468444097+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc00012d380, READY" module=grpc
time="2019-03-18T10:23:30.471722313+01:00" level=error msg="'overlay' not found
as a supported filesystem on this host. Please ensure kernel is new enough and
has overlay support loaded." storage-driver=overlay2
time="2019-03-18T10:23:30.471762928+01:00" level=error msg="[graphdriver] prior
storage driver overlay2 failed: driver not supported"
Error starting daemon: error initializing graphdriver: driver not supported
The service does start after a guix system reconfigure:
allana@guixsd ~$ sudo guix system reconfigure /etc/config.scm > /dev/null 2>&1
Password:
allana@guixsd ~$ docker ps
CONTAINER IDIMAGE COMMAND CREATED
STATUS PORTS NAMES
allana@guixsd ~$ cat /var/log/docker.log
time="2019-03-18T11:04:08.548958068+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T11:04:08.549060661+01:00" level=info msg="scheme \"unix\" not
registered, fallback to default scheme" module=grpc
time="2019-03-18T11:04:08.549129691+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T11:04:08.549145165+01:00" level=info msg="scheme \"unix\" not
registered, fallback to default scheme" module=grpc
time="2019-03-18T11:04:08.549194625+01:00" level=info msg="ccResolverWrapper:
sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0
}]" module=grpc
time="2019-03-18T11:04:08.549225327+01:00" level=info msg="ClientConn switching
balancer to \"pick_first\"" module=grpc
time="2019-03-18T11:04:08.549295334+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc0007c8730, CONNECTING" module=grpc
time="2019-03-18T11:04:08.549428581+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc0007c8730, READY" module=grpc
time="2019-03-18T11:04:08.549823791+01:00" level=info msg="ccResolverWrapper:
sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0
}]" module=grpc
time="2019-03-18T11:04:08.549852586+01:00" level=info msg="ClientConn switching
balancer to \"pick_first\"" module=grpc
time="2019-03-18T11:04:08.549895079+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc00048c190, CONNECTING" module=grpc
time="2019-03-18T11:04:08.550230781+01:00" level=info msg="pickfirstBalancer:
HandleSub
bug#34333: Docker daemon failing to start on boot
Hi Allan,
thanks for the logs!
I've found the problem now.
daemon/graphdriver/overlay2/overlay.go:
func supportsOverlay() error {
// We can try to modprobe overlay first before looking at
// proc/filesystems for when overlay is supported
exec.Command("modprobe", "overlay").Run()
f, err := os.Open("/proc/filesystems")
if err != nil {
return err
}
defer f.Close()
s := bufio.NewScanner(f)
for s.Scan() {
if s.Text() == "nodev\toverlay" {
return nil
}
}
logrus.WithField("storage-driver", "overlay2").Error("'overlay' not
found as a supported filesystem on this host. Please ensure kernel is new
enough and has overlay support loaded.")
return graphdriver.ErrNotSupported
}
We don't load "overlay" explicitly. The above is some weird
contraption--loading kernel modules from random user space programs. Seriously?
And I suspect that modprobe is not found in your system profile.
As a workaround, try adding "kmod" to the list of packages in your
operating-system in your system configuration and reconfigure.
But the real fix is for Docker to stop doing this weird thing in the first
place. Nowadays, modules are autoloaded when someone is accessing the thing
(by udev, or just by using it etc).
In this case, they do
if err := mount("overlay", mountTarget, "overlay", 0, mountData); err
!= nil {
later on. And that's how it should have been detecting it, too.
pgp8j008cpYps.pgp
Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
For our own reference: # lsmod |grep overlay # mkdir -p /b # mount -t overlay none /b mount: /b: wrong fs type, bad option, bad superblock on /a, missing codepage or helper program, or other error. # lsmod |grep overlay overlay 110592 0 pgpaqHWkkR0Wf.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Hi Danny. With great excitement I edited my config.scm to include kmod,
ran guix system reconfigure, and rebooted my machine. Unfortunately my
changes did not seem to fix the issue. I hope the session below can help
us further. Thanks so much for working on this issue.
allana@guixsd ~$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
allana@guixsd ~$ cat /var/log/docker.log
time="2019-03-18T14:39:59.788932321+01:00" level=warning msg="Error while
setting daemon root propagation, this is not generally critical but may cause
some functionality to not work or fallback to less desirable behavior"
dir=/var/lib/docker error="error writing file to signal mount cleanup on
shutdown: open /var/run/docker/unmount-on-shutdown: no such file or directory"
time="2019-03-18T14:39:59.797964377+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T14:39:59.797982675+01:00" level=info msg="scheme \"unix\" not
registered, fallback to default scheme" module=grpc
time="2019-03-18T14:39:59.798127164+01:00" level=info msg="ccResolverWrapper:
sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0
}]" module=grpc
time="2019-03-18T14:39:59.798220831+01:00" level=info msg="ClientConn switching
balancer to \"pick_first\"" module=grpc
time="2019-03-18T14:39:59.798291248+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc000771980, CONNECTING" module=grpc
time="2019-03-18T14:39:59.800603937+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xc000771980, READY" module=grpc
time="2019-03-18T14:39:59.801234292+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T14:39:59.801254794+01:00" level=info msg="scheme \"unix\" not
registered, fallback to default scheme" module=grpc
time="2019-03-18T14:39:59.801329244+01:00" level=info msg="ccResolverWrapper:
sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0
}]" module=grpc
time="2019-03-18T14:39:59.801366954+01:00" level=info msg="ClientConn switching
balancer to \"pick_first\"" module=grpc
time="2019-03-18T14:39:59.801507445+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xcd79d0, CONNECTING" module=grpc
time="2019-03-18T14:39:59.802331100+01:00" level=info msg="pickfirstBalancer:
HandleSubConnStateChange: 0xcd79d0, READY" module=grpc
time="2019-03-18T14:39:59.815614194+01:00" level=error msg="'overlay' not found
as a supported filesystem on this host. Please ensure kernel is new enough and
has overlay support loaded." storage-driver=overlay2
time="2019-03-18T14:39:59.815664314+01:00" level=error msg="[graphdriver] prior
storage driver overlay2 failed: driver not supported"
Error starting daemon: error initializing graphdriver: driver not supported
allana@guixsd ~$ cat /etc/config.scm
(use-modules (gnu)
(gnu system nss)
(gnu services))
(use-service-modules desktop docker)
(use-package-modules certs gnome linux)
(operating-system
(host-name "guixsd")
(timezone "Europe/Oslo")
(locale "en_US.utf8")
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")))
(file-systems (cons (file-system
(device (file-system-label "my-root"))
(mount-point "/")
(type "ext4"))
%base-file-systems))
(users (cons (user-account
(name "allana")
(group "users")
(supplementary-groups '("wheel"
"docker"
"netdev"
"audio"
"video"))
(home-directory "/home/allana"))
%base-user-accounts))
;; This is where we specify system-wide packages.
(packages (cons* nss-certs ;for HTTPS access
gvfs ;for user mounts
kmod ;for modprobe/dockerd
%base-packages))
(services (cons* (console-keymap-service "no-latin1")
(gnome-desktop-service)
(service docker-service-type)
%desktop-services))
;; Allow resolution of '.local' host names with mDNS.
(name-service-switch %mdns-host-lookup-nss))
allana@guixsd ~$ sudo herd start dockerd
Password:
Service dockerd could not be started.
herd: failed to start service dockerd
allana@guixsd ~$ sudo guix system reconfigure /etc/config.scm > /dev/null 2>&1
allana@guixsd ~$ docker ps
CONTAINER IDIMAGE COMMAND CREATED
STATUS PORTS NAMES
allana@guixsd ~$ cat /var/log/docker.log
time="2019-03-18T14:43:00.850449641+01:00" level=info msg="parsed scheme:
\"unix\"" module=grpc
time="2019-03-18T14:43:00.850524161+01:00" level=info msg="
bug#34333: Docker daemon failing to start on boot
Hi Allan, I've pushed a new way to get rid of this bug to guix master. Could you try guix pull and then guix reconfigure and report back? pgpGYs2hXy1fG.pgp Description: OpenPGP digital signature
bug#34333: Docker daemon failing to start on boot
Danny Milosavljevic writes: > Hi Allan, > > I've pushed a new way to get rid of this bug to guix master. > > Could you try > > guix pull > > and then > > guix reconfigure > > and report back? Success! Thank you very much. -- Allan Adair http://allan.adair.io
