[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
@maciek-borzecki the parser can change its behavior based on a few things. 1. the kernel its built against. This would not change behavior when run in a container vs at system level. 2. If a feature-file is specified, via --features-file, --policy- features, or --kernel-features. This allows overriding the normal policy and kernel examination that the parser does when compiling policy. 3. If /sys/kernel/security/apparmor/features is not available. The parser will fallback to an old set of features available in a kernel before the kernel module started exporting what the kernel module supports on the running kernel. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
""" Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '16' """ Does not cause any change in return codes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
The warning message: """ Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '16' """ Is caused by a LXCFS bug which we've since fixed. Systems still showing this message are in need of a snap refresh and host reboot. It's quite likely that the warning is somehow causing apparmor_parser to exit non-zero which would be the core of the issue here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
It appears that the same version of apparmor parser (2.13.3 as reported by Simon), behaves differently when running in the container. Specifically, the command that snapd also executes did not fail in the container, while it should have. Actually there's even an error message but the exit code is still 0. Since it did not fail, then snapd assumes that bpf is supported and generates a snippet for snap-confine apparmor profile. I think the first step here, is to figure out why apparmor_parser did not fail. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
I can easily reproduce with stgraber's instructions on an arm64 VM
running 20.04 with the 5.4 kernel.
Maciej asked me to provide the output of the following commands:
in the LXD container:
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser
--preprocess ; echo $?
Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to
0 * # of cpus == '16'AppArmor parser error, in stdin line 1: Invalid capability
bpf.
0
on the host:
ubuntu@anbox-streaming-stack-0:~$ echo 'profile snap-test { capability bpf, }'
| apparmor_parser --preprocess ; echo $?
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpf1
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1964636
Title:
Incorrect handling of apparmor `bpf` capability
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
We start seeing the same behavior in the Anbox Cloud CI for a few days now. What tests (via spread) primarily do: 1. ssh to an existing arm64 VM 2. Install a fresh LXD from latest/edge and configure it with the following preseed (setting security.nesting to true or false doesn't make a difference): ``` config: cluster.https_address: $addr:8443 core.https_address: $addr:8443 cluster: enabled: false server_name: lxd0 networks: - name: lxdbr0 type: bridge config: ipv4.nat: true ipv4.dhcp.expiry: infinite ipv4.address: $LXD_SUBNET ipv6.address: none profiles: - name: default config: security.nesting: true devices: root: path: / pool: default type: disk eth0: type: nic nictype: bridged parent: lxdbr0 storage_pools: - name: default driver: zfs config: size: 20GB ``` 3. Now juju starts to bootstrap a controller on top of LXD and then we deploy our charms. 4. At some point the tests run the following: 11:08:00 timeout -s KILL 5m sudo -u root -H /snap/bin/juju ssh ams/0 -o 'ConnectionAttempts 30' -- /snap/bin/amc image add bionic:android10:arm64 /home/ubuntu/anbox-lxd-image.tar.xz 11:08:00 snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks This seems to be consis11:08:00 timeout -s KILL 5m sudo -u root -H /snap/bin/juju ssh ams/0 -o 'ConnectionAttempts 30' -- /snap/bin/amc image add bionic:android10:arm64 /home/ubuntu/anbox-lxd-image.tar.xz 11:08:00 snap-confine has elevated permissions and is not confined but should be. Refusing to continue to avoid permission escalation attacks This doesn't always happen but I haven't yet checked if it's only happening on one particular machine. The VMs are all running 20.04 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
@cascardo that sysctl does indeed change when/if bpf_capable() is called, so a possibility to explore. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
@jjohansen nope, we use the apparmor_parser that ships in core20. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
Does changing the sysctl kernel.unprivileged_bpf_disabled to 0 fix the issue? This has been recently changed in all kernels supporting ebpf to default to 2, even on those kernels that do not support CAP_BPF. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
@stgraber while I know snapd currently isn't vendoring the apparmor_parser, is the LXD snap vendoring apparmor? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
Yeah, that's because you're using LXD 4.23 which has the older LXCFS and doesn't trigger the apparmor warning message which then confuses everything else (per previous comment). If you `snap install lxd --channel=latest/edge`, you should start getting the broken behavior. Note however that we did fix LXCFS so that broken behavior on an apparmor warning will soon go away as the warning in question will no longer be issued by apparmor. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
I pulled a clean 20.04 cloud image VM from https://cloud-
images.ubuntu.com/focal/current/
root@ubuntu:/home/guest# grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 20.04.4 LTS"
root@ubuntu:/home/guest# uname -a
Linux ubuntu 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022
x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:/home/guest# echo 'profile snap-test { capability bpf, }' |
apparmor_parser --preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
as expected:
root@ubuntu:/home/guest# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory
root@ubuntu:/home/guest# snap list lxd
Name Version RevTracking Publisher Notes
lxd 4.0.922526 4.0/stable/… canonical✓ -
root@ubuntu:/home/guest# lxd init --auto
root@ubuntu:/home/guest# lxc launch images:ubuntu/20.04 c1
Creating c1
Starting c1
root@ubuntu:/home/guest# lxc exec c1 -- apt install snapd -y
..
root@ubuntu:/home/guest# lxc exec c1 -- snap list
No snaps are installed yet. Try 'snap install hello-world'.
As expected bpf isn't supported by apparmor_parser:
root@c1:~# echo 'profile snap-test { capability bpf, }' |
apparmor_parser --preprocess
restarted the guest:
root@ubuntu:/home/guest# lxc restart c1
and it's still the same:
root@ubuntu:/home/guest# lxc exec c1 -t /bin/bash
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser
--preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpfroot@c1:~#
root@c1:~#
root@c1:~# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory
The only difference is that I didn't install or run distrobuilder. So I
proceeded to do it.
root@c1:~# snap install distrobuilder --edge --classic
2022-03-12T09:17:52Z INFO Waiting for automatic snapd restart...
distrobuilder (edge) git-f883431 from Stéphane Graber (stgraber) installed
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser
--preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpf
root@c1:~# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory
and restart:
root@c1:~# exit
root@ubuntu:/home/guest# lxc restart c1
root@ubuntu:/home/guest# lxc exec c1 -t /bin/bash
root@c1:~# echo 'profile snap-test { capability bpf, }' | apparmor_parser
--preprocess
AppArmor parser error, in stdin line 1: Invalid capability bpf.
profile snap-test { capability bpfroot@c1:~#
root@c1:~#
root@c1:~# cat /var/lib/snapd/apparmor/snap-confine/cap-bpf
cat: /var/lib/snapd/apparmor/snap-confine/cap-bpf: No such file or directory
root@c1:~# systemctl status snapd.apparmor
● snapd.apparmor.service - Load AppArmor profiles managed internally by snapd
Loaded: loaded (/lib/systemd/system/snapd.apparmor.service; enabled;
vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (exited) since Sat 2022-03-12 09:18:46 UTC; 47s ago
Process: 134 ExecStart=/usr/lib/snapd/snapd-apparmor start (code=exited,
status=0/SUCCESS)
Main PID: 134 (code=exited, status=0/SUCCESS)
Mar 12 09:18:46 c1 systemd[1]: Starting Load AppArmor profiles managed
internally by snapd...
Mar 12 09:18:46 c1 snapd-apparmor[134]: /usr/lib/snapd/snapd-apparmor: 47:
ns_stacked: not found
Mar 12 09:18:46 c1 snapd-apparmor[134]: /usr/lib/snapd/snapd-apparmor: 48:
ns_name: not found
Mar 12 09:18:46 c1 systemd[1]: Finished Load AppArmor profiles managed
internally by snapd.
root@c1:~# exit
root@ubuntu:/home/guest# lxc exec c1 -- distrobuilder
System container image builder for LXC and LXD
Usage:
distrobuilder [command]
Available Commands:
build-dir Build plain rootfs
build-lxc Build LXC image from scratch
build-lxd Build LXD image from scratch
help Help about any command
pack-lxc Create LXC image from existing rootfs
pack-lxd Create LXD image from existing rootfs
repack-windows Repack Windows ISO with drivers included
Flags:
--cache-dir Cache directory
--cleanup Clean up cache directory (default true)
--debug Enable debug output
--disable-overlay Disable the use of filesystem
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
As part of digging into this issue, I found this error: ``` Warning from stdin (line 1): apparmor_parser: Warning capping number of jobs to 0 * # of cpus == '4' ``` Which started appearing since we released LXCFS 5.0.0 (available in LXD edge and candidate). The root cause is https://github.com/lxc/lxcfs/issues/522 This only causes an apparmor warning and the parser is otherwise functional, but this may also be interacting with snapd somehow, causing it to mis-detect available or missing features somehow? ** Bug watch added: LXCFS bug tracker #522 https://github.com/lxc/lxcfs/issues/522 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
The steps in the description reproduce this issue for me on a clean Ubuntu 20.04 server install on either 5.4.0 (GA) or 5.13.0 (HWE) kernels. The container doesn't need to be privileged. The main steps really seem to be the installation of a classic snap followed by restart of the container. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
Snapd does not check for the bpf feature through the kernel, it very explicitly only checks the apparmor_parser support, see https://github.com/snapcore/snapd/blob/master/sandbox/apparmor/apparmor.go#L331-L372 for full details of how we check for features -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
This is failing because the apparmor_parser before the 3.0 release would build its capability list from the installed kernel headers. The apparmor_parser here was built against a kernel without support for cap 'bpf' root@priv:~# apparmor_parser -V AppArmor parser version 2.13.3 Copyright (C) 1999-2008 Novell Inc. Copyright 2009-2018 Canonical Ltd. This was fixed in 3.0 by having a static caps list (with full mapping info) and the dynamic auto-generated list (against the kernel headers) that is used to check that the static list has not become stale. In addition the parser can pull kernel supported caps straight from the apparmor kernel module (it will however be missing the mapping info). To fix similar issues upstream a limited version of the 3.0 fix was backported to 2.13. And release in 2.13.5. Ideally we would pull 2.13.6 in but at a minimum we need to pick 726c3fc1 parser: Make sure apparmor can build on old kernels 3f8cfac3 parser/Makefile: fix generated cap comparison against known list ad45b807 parser: add CAP_BPF/PERFMON; convert to pregenerated cap list cd4a1613 Add CAP_BPF and CAP_PERFMON to severity.db 60007d3f parser: Add warning to capability_table about the need to update the Makefile ef8d5141 parser/Makefile: use LC_ALL=C when invoking sed 4e194b2f parser: unify capability name handling ed61e482 parser: cleanup capability_table generation by dropping cap sys_log efb6952e parser: Move to a pre-generated cap_names.h Now for the bits I didn't quite figure out: - Why does snapd think that the parser supports `bpf` when it in fact doesn't? My guess is that its checking the kernel for support of bpf and not the parser. - Why does this only seem to hit with `distrobuilder`, testing with `hello-world` doesn't hit this issue though we've seen similar behavior from the `go` snap, is it a bug that only triggers on classic snaps? my guess is that it will depend on what connections (plugs) are used. Classic snaps in general shouldn't trigger this because they are run unconfined, but LXD triggers it because it has its own profile. Only snaps that plug a connection that requires bpf would trigger this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1964636] Re: Incorrect handling of apparmor `bpf` capability
Thanks for the report, I just tried to reproduce this on both updated impish and updated jammy and couldn't reproduce. I notice on impish specifically that my focal container does not get the /var/lib/snapd/apparmor/snap-confine/cap-bpf file which is expected because the apparmor_parser for focal does not know about the bpf capability. What's confusing is how/why your container saw this, because again snapd actually tries to compile a program with apparmor_parser with "capability bpf," in it and only if that succeeds will it generate that snippet to include in snap-confine's policy. So for this to have happened to you, the apparmor_parser that snapd sees inside the focal container must have been able to successfully compile with that snippet. You mentioned on IRC that this was a privileged container, is there anyway that in addition to being a privileged container somehow it had a newer apparmor_parser in the container too? ** Changed in: snapd (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1964636 Title: Incorrect handling of apparmor `bpf` capability To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
