With systemd version 226 systemd apparently runs in a different cgroup (init.scope) which causes problems: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1497420 https://bugzilla.proxmox.com/show_bug.cgi?id=755
Apparently it keeps trying to restart systemd-journald, which is accompanied by very informative log messages: Oct 12 08:49:36 myhost lxcfs[16970]: get_next_cgroup_dir: I was fed bad input (repeated a bunch of times) I put the foreground lxc-start output at the bottom. Since this is such a very informative error message I changed it a bit to print the two strings it's comparing: taskcg = lxc/103/init.scope querycg = lxc/103/system.slice/systemd-journald.service Now, this 'init.scope' seems to be new in systemd. I'm not quite sure if this is all there's to it, but since that was the first thing I saw it's what I went with, and I'm having some questions regarding my understanding of the lxcfs code regarding caller_is_in_ancestor() and get_next_cgroup_dir(). For one, it seems overly restrictive - more than a usual system would be if I'm not mistaken? And on the other hand the strlen() comparison seems a bit off as does the creation of the 'next' string. I took the liberty of running the code in a standalone .c file with a few test-input lines. In particular, caller_is_in_ancestor contains the following comment: * if the answer is false and nextcg is not NULL, then *nextcg will point * to a nih_alloc'd string containing the next cgroup directory under cg ~ ~~~~~~~~ But here's what I get: $ ./a.out cg = /a/b (1) Test: '1:foo:/a': true (2) Test: '1:foo:/a/b': true (3) Test: '1:foo:/a/b/c': false, got next: 'c' get_next_cgroup_dir: I was fed bad input : '/x' and '/a/b' (4) Test: '1:foo:/x': false get_next_cgroup_dir: I was fed bad input : '/x/b' and '/a/b' (5) Test: '1:foo:/x/b': false (6) Test: '1:foo:/x/b/z': false, got next: 'z' (7) Test: '1:foo:/foobar': false, got next: 'ar' The 7th test shows a rather obvious issue there. The returned 'next' string when the return value is false is taken from the wrong string. 1..5 seem fine, 6 returns 'z', which is not the directory under the 'cg' variable mentioned in the comment, but from the caller's path. 7 returns 'ar', which is &"/foobar"[strlen("/a/b")] which is basically how the string is created. So now I'm wondering if the current issue is an lxc-only issue, a systemd-issue, or both, as I'm pretty sure that I don't give a damn about whether systemd-journald manages to run when my system is broken. In a broken system I'd rather get a freaking bash emergency console than a system that refuses to do anything just because it has no freaking *logger*... grrr... (Well, I can attach to the namespaces anyway, so in containers that part is a non-issue, but really... come on...) --- lxc-start foreground output --- # lxc-start -lDEBUG -o ~/lxc.log -F -n 103 systemd 226 running in system mode. (+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN) Detected virtualization lxc. Detected architecture x86-64. Welcome to Arch Linux! Set hostname to <carc>. Failed to install release agent, ignoring: No such file or directory display-manager.service: Cannot add dependency job, ignoring: Unit display-manager.service failed to load: No such file or directory. [ OK ] Started Forward Password Requests to Wall Directory Watch. Failed to reset devices.list on /lxc/103: Permission denied Failed to reset devices.list on /lxc/103/user.slice: Permission denied [ OK ] Created slice User and Session Slice. [ OK ] Reached target Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Reached target Remote File Systems. [ OK ] Listening on Journal Audit Socket. [ OK ] Listening on /dev/initctl Compatibility Named Pipe. Failed to reset devices.list on /lxc/103/system.slice: Permission denied [ OK ] Created slice System Slice. Failed to reset devices.list on /lxc/103/system.slice/dev-mqueue.mount: Permission denied Mounting POSIX Message Queue File System... Failed to reset devices.list on /lxc/103/system.slice/system-container\x2dgetty.slice: Permission denied [ OK ] Created slice system-container\x2dgetty.slice. Failed to reset devices.list on /lxc/103/system.slice/dev-hugepages.mount: Permission denied Mounting Huge Pages File System... Failed to reset devices.list on /lxc/103/system.slice/systemd-remount-fs.service: Permission denied Starting Remount Root and Kernel File Systems... Failed to reset devices.list on /lxc/103/system.slice/tmp.mount: Permission denied Mounting Temporary Directory... Failed to reset devices.list on /lxc/103/system.slice/system-getty.slice: Permission denied [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Device-mapper event daemon FIFOs. [ OK ] Listening on Journal Socket (/dev/log). Failed to reset devices.list on /lxc/103/system.slice/systemd-journald.service: Permission denied Starting Journal Service... [ OK ] Reached target Swap. Failed to reset devices.list on /lxc/103/system.slice/proc-uptime.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/proc-meminfo.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/proc-stat.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-tty1.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/proc-diskstats.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-console.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/proc-sysrq\x2dtrigger.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/sys-devices-virtual-net.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/sys-fs-fuse-connections.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/proc-sys-net.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/something.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/dev-lxc-tty2.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/proc-cpuinfo.mount: Permission denied Failed to reset devices.list on /lxc/103/system.slice/-.mount: Permission denied Failed to reset devices.list on /lxc/103/init.scope: Permission denied dev-mqueue.mount: Mount process exited, code=exited status=219 [FAILED] Failed to mount POSIX Message Queue File System. See 'systemctl status dev-mqueue.mount' for details. dev-mqueue.mount: Unit entered failed state. dev-hugepages.mount: Mount process exited, code=exited status=219 [FAILED] Failed to mount Huge Pages File System. See 'systemctl status dev-hugepages.mount' for details. dev-hugepages.mount: Unit entered failed state. systemd-remount-fs.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. systemd-remount-fs.service: Unit entered failed state. systemd-remount-fs.service: Failed with result 'exit-code'. tmp.mount: Mount process exited, code=exited status=219 [FAILED] Failed to mount Temporary Directory. See 'systemctl status tmp.mount' for details. [DEPEND] Dependency failed for Basic System. [DEPEND] Dependency failed for D-Bus System Message Bus. dbus.service: Job dbus.service/start failed with result 'dependency'. [DEPEND] Dependency failed for Getty on tty1. getty@tty1.service: Job getty@tty1.service/start failed with result 'dependency'. [DEPEND] Dependency failed for Login Service. systemd-logind.service: Job systemd-logind.service/start failed with result 'dependency'. [DEPEND] Dependency failed for Console Getty. console-getty.service: Job console-getty.service/start failed with result 'dependency'. [DEPEND] Dependency failed for Multi-User System. [DEPEND] Dependency failed for Graphical Interface. graphical.target: Job graphical.target/start failed with result 'dependency'. multi-user.target: Job multi-user.target/start failed with result 'dependency'. [DEPEND] Dependency failed for Permit User Sessions. systemd-user-sessions.service: Job systemd-user-sessions.service/start failed with result 'dependency'. [DEPEND] Dependency failed for Container Getty on /dev/pts/0. container-getty@0.service: Job container-getty@0.service/start failed with result 'dependency'. [DEPEND] Dependency failed for Container Getty on /dev/pts/1. container-getty@1.service: Job container-getty@1.service/start failed with result 'dependency'. basic.target: Job basic.target/start failed with result 'dependency'. tmp.mount: Unit entered failed state. systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. [DEPEND] Dependency failed for Flush Journal to Persistent Storage. systemd-journal-flush.service: Job systemd-journal-flush.service/start failed with result 'dependency'. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. Starting Journal Service... [ OK ] Reached target Login Prompts. Starting Create System Users... Starting Load/Save Random Seed... Starting Rebuild Hardware Database... Starting Rebuild Dynamic Linker Cache... systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. systemd-sysusers.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Create System Users. See 'systemctl status systemd-sysusers.service' for details. systemd-sysusers.service: Unit entered failed state. systemd-sysusers.service: Failed with result 'exit-code'. systemd-random-seed.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Load/Save Random Seed. See 'systemctl status systemd-random-seed.service' for details. systemd-random-seed.service: Unit entered failed state. systemd-random-seed.service: Failed with result 'exit-code'. systemd-hwdb-update.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Rebuild Hardware Database. See 'systemctl status systemd-hwdb-update.service' for details. systemd-hwdb-update.service: Unit entered failed state. systemd-hwdb-update.service: Failed with result 'exit-code'. ldconfig.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Rebuild Dynamic Linker Cache. See 'systemctl status ldconfig.service' for details. ldconfig.service: Unit entered failed state. ldconfig.service: Failed with result 'exit-code'. Starting Journal Service... [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Rebuild Journal Catalog... Starting Create Volatile Files and Directories... systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. Starting Journal Service... [ ***] (1 of 3) A start job is running for Journal Service (9s / 1min 30s)systemd-journal-catalog-update.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Rebuild Journal Catalog. See 'systemctl status systemd-journal-catalog-update.service' for details. systemd-journal-catalog-update.service: Unit entered failed state. systemd-journal-catalog-update.service: Failed with result 'exit-code'. systemd-tmpfiles-setup.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Create Volatile Files and Directories. See 'systemctl status systemd-tmpfiles-setup.service' for details. systemd-tmpfiles-setup.service: Unit entered failed state. systemd-tmpfiles-setup.service: Failed with result 'exit-code'. Starting Update UTMP about System Boot/Shutdown... Starting Update is Completed... systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. Starting Journal Service... [ ***] (1 of 3) A start job is running for Update UTMP about System Boot/Shutdown (19s / no limit)systemd-update-utmp.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Update UTMP about System Boot/Shutdown. See 'systemctl status systemd-update-utmp.service' for details. systemd-update-utmp.service: Unit entered failed state. systemd-update-utmp.service: Failed with result 'exit-code'. systemd-update-done.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Update is Completed. See 'systemctl status systemd-update-done.service' for details. systemd-update-done.service: Unit entered failed state. systemd-update-done.service: Failed with result 'exit-code'. systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. Starting Journal Service... [** ] A start job is running for Journal Service (9s / 1min 30s)systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. Starting Journal Service... [ *** ] A start job is running for Journal Service (9s / 1min 30s)systemd-journald.service: Main process exited, code=exited, status=219/CGROUP [FAILED] Failed to start Journal Service. See 'systemctl status systemd-journald.service' for details. systemd-journald.service: Unit entered failed state. systemd-journald.service: Failed with result 'exit-code'. Starting Journal Service... [ *** ] A start job is running for Journal Service (7s / 1min 30s)Received SIGRTMIN+14. _______________________________________________ lxc-devel mailing list lxc-devel@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-devel