Bug#1038315: lxd: AppArmor profile violations cause PrivateNetwork=yes services in the container fail to start
On 6/24/23 22:19, Mathias Gibbens wrote: >> systemd-hostnamed.service is probably also affected, but in my case I >> paved over the issue by setting PrivateNetwork=no in an override. > > For the moment setting PrivateNetwork=no for affected services is > probably a better approach than totally removing the container's > apparmor profile. > Hi Mathias, sorry for late reply. Indeed, overriding affected services with PrivateNetwork=no works. Thanks for the better workaround! -- An old man doll... just what I always wanted! - Clara
Bug#1038315: lxd: AppArmor profile violations cause PrivateNetwork=yes services in the container fail to start
Control: tags -1 + confirmed Control: forwarded https://discuss.linuxcontainers.org/t/systemd-hostnamed-unable-to-start-on-lxd-5-0-containers/17454 Hi Bagas, Michael, At first glance, this looks like an apparmor problem. Unfortunately all the issues that have been mentioned previously are about five years old. Ubuntu has carried custom patches for the kernel and apparmor in the past, but at least the relevant kernel changes seem to have been merged quite a while back[1]. I will see what I can figure out on my end and update this when I know more. > systemd-hostnamed.service is probably also affected, but in my case I > paved over the issue by setting PrivateNetwork=no in an override. For the moment setting PrivateNetwork=no for affected services is probably a better approach than totally removing the container's apparmor profile. Mathias [1] -- https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80a17a5f501ea048d86f81d629c94062b76610d4 signature.asc Description: This is a digitally signed message part
Bug#1038315: lxd: AppArmor profile violations cause PrivateNetwork=yes services in the container fail to start
Package: lxd Version: 5.0.2-5 Followup-For: Bug #1038315 I'm unsure if current breakage is due to apparmor itself (the bug reported there was apparently fixed a while ago), lxd's apparmor profile, or somewhere else (as satisfying as blaming systemd would be...). See linked bugs listed below. Is it possible that the AppArmor socket mediation patches have not made it into upstream and/or Debian kernels? :thinking_face: If so, then this bug needs to be reassigned to the kernel. I'm writing here because this also breaks the plocate-updatedb service inside LXD containers. I don't think it's a bug with network namespacing inside containers, as unshare -u ifconfig works fine, for example. In the container: root@pat:~# systemctl status plocate-updatedb.service × plocate-updatedb.service - Update the plocate database Loaded: loaded (/lib/systemd/system/plocate-updatedb.service; static) Drop-In: /run/systemd/system/service.d └─zzz-lxc-service.conf Active: failed (Result: exit-code) since Fri 2023-06-23 09:53:56 AWST; 5h 31min ago TriggeredBy: ● plocate-updatedb.timer Process: 33437 ExecStart=/usr/sbin/updatedb.plocate (code=exited, status=225/NETWORK) Main PID: 33437 (code=exited, status=225/NETWORK) CPU: 584us Jun 23 09:53:56 pat systemd[1]: Starting plocate-updatedb.service - Update the plocate database... Jun 23 09:53:56 pat systemd[1]: plocate-updatedb.service: Main process exited, code=exited, status=225/NETWORK Jun 23 09:53:56 pat systemd[1]: plocate-updatedb.service: Failed with result 'exit-code'. Jun 23 09:53:56 pat systemd[1]: Failed to start plocate-updatedb.service - Update the plocate database. On the host after attempting to start the service inside the guest: 2023-06-23T09:53:56.040427+08:00 grook kernel: [772843.931461] audit: type=1400 audit(1687485236.036:118): apparmor="DENIED" operation="file_lock" profile="lxd-pat_" pid=3334600 comm="(.plocate)" family="unix" sock_type=ram" protocol=0 requested_mask="send" 2023-06-23T09:53:56.040437+08:00 grook kernel: [772843.931469] audit: type=1400 audit(1687485236.036:119): apparmor="DENIED" operation="file_lock" profile="lxd-pat_" pid=3334600 comm="(.plocate)" family="unix" sock_type=ram" protocol=0 requested_mask="send" Host information (both host and guest are running bookworm BTW): Kernel: Linux 6.1.0-9-amd64 (SMP w/8 CPU threads; PREEMPT) Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages lxd depends on: ii adduser 3.134 ii attr 1:2.5.1-4 ii ca-certificates 20230311 ii init-system-helpers 1.65.2 ii libacl1 2.3.1-3 ii libc62.36-9 ii libcap2 1:2.66-4 ii libdqlite0 1.11.1-1 ii libgcc-s112.2.0-14 ii liblxc-common1:5.0.2-1 ii liblxc1 1:5.0.2-1 ii libsqlite3-0 3.40.1-2 ii libudev1 252.6-1 ii lxcfs5.0.3-1 ii lxd-client 5.0.2-5 ii rsync3.2.7-1 ii squashfs-tools 1:4.5.1-1 ii uidmap 1:4.13+dfsg1-1+b1 ii xz-utils 5.4.1-0.2 Versions of packages lxd recommends: ii apparmor 3.0.8-3 ii dnsmasq-base [dnsmasq-base] 2.89-1 ii lxd-agent5.0.2-5 Versions of packages lxd suggests: pn btrfs-progs pn ceph-common ii gdisk 1.0.9-2.1 ii lvm22.03.16-2 ii lxd-tools 5.0.2-5 ii zfsutils-linux 2.1.11-1 The container itself does not have apparmour installed. systemd-hostnamed.service is probably also affected, but in my case I paved over the issue by setting PrivateNetwork=no in an override. Related: - https://bugs.launchpad.net/bugs/1575779 and https://bugs.launchpad.net/bugs/1780227 - https://bugs.launchpad.net/bugs/1635382 - https://github.com/lxc/lxc/issues/820 and https://github.com/lxc/lxd/issues/1603 -MD
Bug#1038315: lxd: AppArmor profile violations cause PrivateNetwork=yes services in the container fail to start
Package: lxd Version: 5.0.2-5 Severity: important Dear Maintainer, I spin up containers using LXD. I'm primarily doing this for learning purposes (for example installing and configuring web services). I launched my first container (test) to get my feet wet by: ``` lxc launch images:debian/12 test ``` When I tried to set hostname in the container via (`hostnamectl set-hostname test.test`), I got `Could not set pretty hostname: Connection timed out` error. journalctl on the container showed errors related to network namespacing: ``` Jun 17 03:03:12 test dbus-daemon[81]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.7' (uid=0 pid=104 comm="hostnamectl set-hostname test.test") Jun 17 03:03:12 test (ostnamed)[105]: systemd-hostnamed.service: Failed to set up network namespacing: Permission denied Jun 17 03:03:12 test systemd[1]: Starting systemd-hostnamed.service - Hostname Service... Jun 17 03:03:12 test (ostnamed)[105]: systemd-hostnamed.service: Failed at step NETWORK spawning /lib/systemd/systemd-hostnamed: Permission denied Jun 17 03:03:12 test systemd[1]: systemd-hostnamed.service: Main process exited, code=exited, status=225/NETWORK Jun 17 03:03:12 test systemd[1]: systemd-hostnamed.service: Failed with result 'exit-code'. Jun 17 03:03:12 test systemd[1]: Failed to start systemd-hostnamed.service - Hostname Service. Jun 17 03:03:37 test dbus-daemon[81]: [system] Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms) ``` dmesg on the host revealed that these errors above are due to AppArmor policy violations: ``` [10673.299973] audit: type=1400 audit(1686970915.519:84): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_" pid=12984 comm="(crub_all)" family="unix" sock_type="dgram" protocol=0 requested_mask="send" [10673.299988] audit: type=1400 audit(1686970915.519:85): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_" pid=12984 comm="(crub_all)" family="unix" sock_type="dgram" protocol=0 requested_mask="send" [10675.793944] audit: type=1400 audit(1686970918.015:86): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_" pid=12991 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send" [10675.793966] audit: type=1400 audit(1686970918.015:87): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_" pid=12991 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send" [10750.671804] audit: type=1400 audit(1686970992.896:88): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_" pid=13038 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send" [10750.671817] audit: type=1400 audit(1686970992.896:89): apparmor="DENIED" operation="file_lock" profile="lxd-myself_test_" pid=13038 comm="(ostnamed)" family="unix" sock_type="dgram" protocol=0 requested_mask="send" ``` >From the upstream discussion [1], the workaround is to put AppArmor profile for the container unconfined. To do this, I have to run: ``` lxc config set test raw.lxc "lxc.apparmor.profile=unconfined" ``` See lxc.container.conf(5) manpage [2] for explanation of lxc config key. The upstream discussion stated that this bug should have been already fixed in recent systemd, AppArmor, and Linux kernel versions (at least as shipped in Ubuntu Bionic release). However, I can (still) reproduce it on Debian testing. Thanks. [1]: https://discuss.linuxcontainers.org/t/bionic-containers-on-xenial-host-systemd-hostnamed-unable-to-start/1732 [2]: https://manpages.ubuntu.com/manpages/lunar/en/man5/lxc.container.conf.5.html -- System Information: Debian Release: 12.0 APT prefers testing APT policy: (500, 'testing') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-9-amd64 (SMP w/4 CPU threads; PREEMPT) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages lxd depends on: ii adduser 3.134 ii attr 1:2.5.1-4 ii ca-certificates 20230311 ii init-system-helpers 1.65.2 ii libacl1 2.3.1-3 ii libc62.36-9 ii libcap2 1:2.66-4 ii libdqlite0 1.11.1-1 ii libgcc-s112.2.0-14 ii liblxc-common1:5.0.2-1 ii liblxc1 1:5.0.2-1 ii libsqlite3-0 3.40.1-2 ii libudev1 252.6-1 ii lxcfs5.0.3-1 ii lxd-client 5.0.2-5 ii rsync3.2.7-1 ii squashfs-tools 1:4.5.1-1 ii uidmap 1:4.13+dfsg1-1+b1 ii xz-utils 5.4.1-0.2 Versions of packages lxd recommends: ii apparmor 3.0.8-3 ii dnsmasq-base [dnsmasq-base] 2.89-1 ii lxd-agent5.0.2-5 Versions of packages lxd
