It just broke for me on two servers again, more or less at the same
time:
root@backup01 ~ # ls -l /data/lxd
total 0
drwx------ 1 root root 198 Jan 24 03:34 containers
(...)
Both servers are running Ubuntu 18.04 with LXD from snap:
lxd 3.9 9919 stable canonical✓ -
And storage on a btrfs device:
root@lxd05 ~ # lxc storage list
+---------+-------------+--------+-----------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+---------+-------------+--------+-----------+---------+
| default | | btrfs | /data/lxd | 16 |
+---------+-------------+--------+-----------+---------+
root@backup01 ~ # lxc storage list
+---------+-------------+--------+-----------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+---------+-------------+--------+-----------+---------+
| default | | btrfs | /data/lxd | 44 |
+---------+-------------+--------+-----------+---------+
Not sure what's causing, but it's yet another time I'm seeing it.
Tomasz
On 2018-09-24 22:43, Christian Brauner wrote:
On Mon, Sep 24, 2018 at 03:40:57PM +0200, Tomasz Chmielewski wrote:
Turns out something changed the permissions on "containers" directory:
Odd, the new storage snapshot api performs an on-disk upgrade but it
shouldn't touch the containers directory...
//cc Stéphane
Christian
# lxc storage list
+---------+-------------+--------+-----------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+---------+-------------+--------+-----------+---------+
| default | | btrfs | /data/lxd | 12 |
+---------+-------------+--------+-----------+---------+
# ls -l /data/lxd
total 0
drwxr-xr-x 1 root root 90 Sep 24 13:05 archives
drwx------ 1 root root 518 Sep 24 13:12 containers <--------- here
drwx--x--x 1 root root 0 Mar 28 16:14 custom
drwx------ 1 root root 0 Sep 21 06:05 images
drwx------ 1 root root 0 Sep 24 05:48 snapshots
This fixed it:
chmod 711 /data/lxd/containers/
I'm 99% sure we did not change the permissions on that directory...
Tomasz
On 2018-09-24 15:32, Tomasz Chmielewski wrote:
> I'm not able to start any container today.
>
> # lxc start preprod-app
> Error: Failed to run: /snap/lxd/current/bin/lxd forkstart preprod-app
> /var/snap/lxd/common/lxd/containers
> /var/snap/lxd/common/lxd/logs/preprod-app/lxc.conf:
> Try `lxc info --show-log preprod-app` for more info
>
>
> # lxc info --show-log preprod-app
> Name: preprod-app
> Remote: unix://
> Architecture: x86_64
> Created: 2018/09/05 15:01 UTC
> Status: Stopped
> Type: persistent
> Profiles: default
>
> Log:
>
> lxc preprod-app 20180924132438.883 WARN conf -
> conf.c:lxc_map_ids:2917 - newuidmap binary is missing
> lxc preprod-app 20180924132438.883 WARN conf -
> conf.c:lxc_map_ids:2923 - newgidmap binary is missing
> lxc preprod-app 20180924132438.887 WARN conf -
> conf.c:lxc_map_ids:2917 - newuidmap binary is missing
> lxc preprod-app 20180924132438.887 WARN conf -
> conf.c:lxc_map_ids:2923 - newgidmap binary is missing
> lxc preprod-app 20180924132438.917 ERROR dir -
> storage/dir.c:dir_mount:195 - Permission denied - Failed to mount
> "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" on
> "/var/snap/lxd/common/lxc/"
> lxc preprod-app 20180924132438.917 ERROR conf -
> conf.c:lxc_mount_rootfs:1337 - Failed to mount rootfs
> "/var/snap/lxd/common/lxd/containers/preprod-app/rootfs" onto
> "/var/snap/lxd/common/lxc/" with options "(null)"
> lxc preprod-app 20180924132438.917 ERROR conf -
> conf.c:lxc_setup_rootfs_prepare_root:3446 - Failed to setup rootfs for
> lxc preprod-app 20180924132438.917 ERROR conf -
> conf.c:lxc_setup:3510 - Failed to setup rootfs
> lxc preprod-app 20180924132438.917 ERROR start -
> start.c:do_start:1234 - Failed to setup container "preprod-app"
> lxc preprod-app 20180924132438.918 ERROR sync -
> sync.c:__sync_wait:59 - An error occurred in another process (expected
> sequence number 5)
> lxc preprod-app 20180924132439.235 ERROR start -
> start.c:__lxc_start:1910 - Failed to spawn container "preprod-app"
> lxc preprod-app 20180924132439.235 ERROR lxccontainer -
> lxccontainer.c:wait_on_daemonized_start:840 - Received container state
> "ABORTING" instead of "RUNNING"
> lxc preprod-app 20180924132439.963 WARN conf -
> conf.c:lxc_map_ids:2917 - newuidmap binary is missing
> lxc preprod-app 20180924132439.101 WARN conf -
> conf.c:lxc_map_ids:2923 - newgidmap binary is missing
> lxc 20180924132439.380 WARN commands -
> commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to
> receive response for command "get_state"
>
>
> # snap list
> Name Version Rev Tracking Publisher Notes
> core 16-2.35 5328 stable canonical✓ core
> lxd 3.5 8774 stable canonical✓ -
>
>
> This is on Ubuntu 18.04.
_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users
