Public bug reported:
After upgrading our images from 5.15.0-1052-aws to 5.15.0-1053-aws we've
noticed increased CPU usage caused by lock contention in the AppArmor
code paths. We've since continued upgraded past that, but the problem
persisted.
Digging more, we've found that rapid (parallel) invocation of
getaddrinfo was the cause of the contention (via openat, stat, etc
syscalls) and thereby spinning on that lock, which in turn maxed the CPU
of those instances. This is on c6i.24xlarge instances, so 96 CPUs. I'm
attaching a perf flamegraph of the exact stacktraces we're seeing.
We did not expect a patch-level kernel upgrade to cause this increased
CPU usage, however we managed to work around this issue via caching, so
this report is more informational/checking if this was intended. Looking
at the upstream kernel it seems like this eventually got fixed with per-
CPU buffers.
The only related changes we've found in the diffs between these two
versions were the addition of AppArmor checking for mqueues, which I
don't believe we're using explicitly anywhere. I'm not sure if that
change had other effects though.
This is all on Ubuntu 20.04 with the AWS-specific kernel package,
although I'm assuming it applies to the Ubuntu kernel generally. I was
unable to do the mapping to the "real" release versions though, so
reporting in this package since I got the exact version numbers.
** Affects: linux-aws (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "Flamegraph of stacktraces with contention"
https://bugs.launchpad.net/bugs/2069478/+attachment/5789701/+files/Perf%20Flamegraph.png
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2069478
Title:
Increased lock contention in aa_{get,set}_buffer
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2069478/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
