Public bug reported: The kernel keeps track of the time spent in userspace, kernel space and for quite a while (since 2.6.24) also guest_time.
This is accessible to userspace a) globally via /proc/stat [1] b) per process via /proc/<PID>/stat But since ~5.9 this accounting is broken by [3] #An example of a busy guest reporting all zeros in a common userspace tool ubuntu@node-horsea:~$ pidstat -p $(pidof qemu-system-x86_64) 5 Linux 5.11.0-13-generic (node-horsea) 04/07/21 _x86_64_ (12 CPU) 05:13:13 UID PID %usr %system %guest %wait %CPU CPU Command 05:13:18 64055 8186 10.20 96.80 0.00 0.00 107.00 4 qemu-system-x86 05:13:23 64055 8186 5.00 51.00 0.00 0.00 56.00 5 qemu-system-x86 05:13:28 64055 8186 3.60 109.80 0.00 0.00 113.40 5 qemu-system-x86 05:13:33 64055 8186 1.80 14.20 0.00 0.00 16.00 5 qemu-system-x86 This is due to these stats staying zero: $ cat /proc/stat | grep "cpu " cpu 11345 154 10342 71506151 549 0 161 0 0 0 The Testcase for this is simple, run any KVM guest content for example even the following without any prep is enough $ sudo qemu-system-x86_64 -enable-kvm -nographic That should execute at least a bit of time in the guest for startup code. Surely, if you want anything for long term debugging you can run a full Guest and do any busy loop in the guest. This was reported upstream to the kernel in [4] and Debian [5] and qemu [6]. AFAIK there are two suggested [8][9], but no accepted solution yet. If it would be "only" the global counter (a) this might be prio low-to- mid, because while still inhibiting monitoring you'd at least still have the per guest counters (b). But since both are broken one has no easy means anymore to check if there is massive host-kernel (%sys) or host- userspace (%usr) overhead over the guest (%guest) anymore. Therefore (maybe due to my old perf-engineer heart) IMHO this is rather important to be fixed soon. Best in the next possible update for Hirsute, but that might be hard. But at lest for an Hirsute based HWE kernel to Focal I'd consider this a hard blocker. [1]: https://www.kernel.org/doc/Documentation/filesystems/proc.txt see "1.8 Miscellaneous kernel statistics in /proc/stat" [2]: http://manpages.ubuntu.com/manpages/focal/man5/proc.5.html see "(43) guest_time %lu" [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87fa7f3e98a1310ef1ac1900e7ee7f9610a038bc [4]: https://bugzilla.kernel.org/show_bug.cgi?id=209831 [5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371 [6]: https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg00732.html [7]: https://lore.kernel.org/kvm/1617011036-11734-1-git-send-email-wanpen...@tencent.com/ [8]: https://lore.kernel.org/kvm/20210206004218.312023-1-sea...@google.com/ ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: linux-image-5.11.0-13-generic 5.11.0-13.14 ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7 Uname: Linux 5.11.0-13-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu61 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CasperMD5CheckResult: unknown Date: Wed Apr 7 05:02:03 2021 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: HP ProLiant DL360 Gen9 PciMultimedia: ProcFB: 0 mgag200drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-13-generic root=UUID=c941b173-e6b5-485a-a02b-8d966b8d3c73 ro --- console=ttyS1,115200 RelatedPackageVersions: linux-restricted-modules-5.11.0-13-generic N/A linux-backports-modules-5.11.0-13-generic N/A linux-firmware 1.196 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/22/2018 dmi.bios.release: 2.56 dmi.bios.vendor: HP dmi.bios.version: P89 dmi.board.name: ProLiant DL360 Gen9 dmi.board.vendor: HP dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.ec.firmware.release: 2.60 dmi.modalias: dmi:bvnHP:bvrP89:bd01/22/2018:br2.56:efr2.60:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr: dmi.product.family: ProLiant dmi.product.name: ProLiant DL360 Gen9 dmi.product.sku: 780018-S01 dmi.sys.vendor: HP ** Affects: linux Importance: Unknown Status: Unknown ** Affects: linux (Ubuntu) Importance: High Status: New ** Affects: linux-hwe (Ubuntu) Importance: Critical Status: New ** Affects: linux (Debian) Importance: Unknown Status: Unknown ** Tags: amd64 apport-bug hirsute package-from-proposed uec-images ** Bug watch added: Debian Bug tracker #986371 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371 ** Also affects: linux (Debian) via https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371 Importance: Unknown Status: Unknown ** Bug watch added: Linux Kernel Bug Tracker #209831 https://bugzilla.kernel.org/show_bug.cgi?id=209831 ** Also affects: linux via https://bugzilla.kernel.org/show_bug.cgi?id=209831 Importance: Unknown Status: Unknown ** Also affects: linux-hwe (Ubuntu) Importance: Undecided Status: New ** Changed in: linux-hwe (Ubuntu) Importance: Undecided => Critical ** Changed in: linux (Ubuntu) Importance: Undecided => High ** Description changed: The kernel keeps track of the time spent in userspace, kernel space and for quite a while (since 2.6.24) also guest_time. This is accessible to userspace a) globally via /proc/stat [1] b) per process via /proc/<PID>/stat But since ~5.9 this accounting is broken by [3] + #An example of a busy guest reporting all zeros in a common userspace tool + ubuntu@node-horsea:~$ pidstat -p $(pidof qemu-system-x86_64) 5 + Linux 5.11.0-13-generic (node-horsea) 04/07/21 _x86_64_ (12 CPU) + 05:13:13 UID PID %usr %system %guest %wait %CPU CPU Command + 05:13:18 64055 8186 10.20 96.80 0.00 0.00 107.00 4 qemu-system-x86 + 05:13:23 64055 8186 5.00 51.00 0.00 0.00 56.00 5 qemu-system-x86 + 05:13:28 64055 8186 3.60 109.80 0.00 0.00 113.40 5 qemu-system-x86 + 05:13:33 64055 8186 1.80 14.20 0.00 0.00 16.00 5 qemu-system-x86 + + This is due to these stats staying zero: + $ cat /proc/stat | grep "cpu " + cpu 11345 154 10342 71506151 549 0 161 0 0 0 + The Testcase for this is simple, run any KVM guest content for example even the following without any prep is enough - $ sudo qemu-system-x86_64 -enable-kvm -nographic + $ sudo qemu-system-x86_64 -enable-kvm -nographic That should execute at least a bit of time in the guest for startup code. Surely, if you want anything for long term debugging you can run a full Guest and do any busy loop in the guest. This was reported upstream to the kernel in [4] and Debian [5] and qemu [6]. AFAIK there are two suggested [8][9], but no accepted solution yet. If it would be "only" the global counter (a) this might be prio low-to- mid, because while still inhibiting monitoring you'd at least still have the per guest counters (b). But since both are broken one has no easy means anymore to check if there is massive host-kernel (%sys) or host- userspace (%usr) overhead over the guest (%guest) anymore. Therefore (maybe due to my old perf-engineer heart) IMHO this is rather important to be fixed soon. Best in the next possible update for Hirsute, but that might be hard. But at lest for an Hirsute based HWE kernel to Focal I'd consider this a hard blocker. [1]: https://www.kernel.org/doc/Documentation/filesystems/proc.txt see "1.8 Miscellaneous kernel statistics in /proc/stat" [2]: http://manpages.ubuntu.com/manpages/focal/man5/proc.5.html see "(43) guest_time %lu" [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87fa7f3e98a1310ef1ac1900e7ee7f9610a038bc [4]: https://bugzilla.kernel.org/show_bug.cgi?id=209831 [5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371 [6]: https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg00732.html [7]: https://lore.kernel.org/kvm/1617011036-11734-1-git-send-email-wanpen...@tencent.com/ [8]: https://lore.kernel.org/kvm/20210206004218.312023-1-sea...@google.com/ ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: linux-image-5.11.0-13-generic 5.11.0-13.14 ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7 Uname: Linux 5.11.0-13-generic x86_64 AlsaDevices: - total 0 - crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq - crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer + total 0 + crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq + crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu61 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CasperMD5CheckResult: unknown Date: Wed Apr 7 05:02:03 2021 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: HP ProLiant DL360 Gen9 PciMultimedia: - + ProcFB: 0 mgag200drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-13-generic root=UUID=c941b173-e6b5-485a-a02b-8d966b8d3c73 ro --- console=ttyS1,115200 RelatedPackageVersions: - linux-restricted-modules-5.11.0-13-generic N/A - linux-backports-modules-5.11.0-13-generic N/A - linux-firmware 1.196 + linux-restricted-modules-5.11.0-13-generic N/A + linux-backports-modules-5.11.0-13-generic N/A + linux-firmware 1.196 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/22/2018 dmi.bios.release: 2.56 dmi.bios.vendor: HP dmi.bios.version: P89 dmi.board.name: ProLiant DL360 Gen9 dmi.board.vendor: HP dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.ec.firmware.release: 2.60 dmi.modalias: dmi:bvnHP:bvrP89:bd01/22/2018:br2.56:efr2.60:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr: dmi.product.family: ProLiant dmi.product.name: ProLiant DL360 Gen9 dmi.product.sku: 780018-S01 dmi.sys.vendor: HP -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1922846 Title: KVM guest time accounting is broken Status in Linux: Unknown Status in linux package in Ubuntu: New Status in linux-hwe package in Ubuntu: New Status in linux package in Debian: Unknown Bug description: The kernel keeps track of the time spent in userspace, kernel space and for quite a while (since 2.6.24) also guest_time. This is accessible to userspace a) globally via /proc/stat [1] b) per process via /proc/<PID>/stat But since ~5.9 this accounting is broken by [3] #An example of a busy guest reporting all zeros in a common userspace tool ubuntu@node-horsea:~$ pidstat -p $(pidof qemu-system-x86_64) 5 Linux 5.11.0-13-generic (node-horsea) 04/07/21 _x86_64_ (12 CPU) 05:13:13 UID PID %usr %system %guest %wait %CPU CPU Command 05:13:18 64055 8186 10.20 96.80 0.00 0.00 107.00 4 qemu-system-x86 05:13:23 64055 8186 5.00 51.00 0.00 0.00 56.00 5 qemu-system-x86 05:13:28 64055 8186 3.60 109.80 0.00 0.00 113.40 5 qemu-system-x86 05:13:33 64055 8186 1.80 14.20 0.00 0.00 16.00 5 qemu-system-x86 This is due to these stats staying zero: $ cat /proc/stat | grep "cpu " cpu 11345 154 10342 71506151 549 0 161 0 0 0 The Testcase for this is simple, run any KVM guest content for example even the following without any prep is enough $ sudo qemu-system-x86_64 -enable-kvm -nographic That should execute at least a bit of time in the guest for startup code. Surely, if you want anything for long term debugging you can run a full Guest and do any busy loop in the guest. This was reported upstream to the kernel in [4] and Debian [5] and qemu [6]. AFAIK there are two suggested [8][9], but no accepted solution yet. If it would be "only" the global counter (a) this might be prio low- to-mid, because while still inhibiting monitoring you'd at least still have the per guest counters (b). But since both are broken one has no easy means anymore to check if there is massive host-kernel (%sys) or host-userspace (%usr) overhead over the guest (%guest) anymore. Therefore (maybe due to my old perf-engineer heart) IMHO this is rather important to be fixed soon. Best in the next possible update for Hirsute, but that might be hard. But at lest for an Hirsute based HWE kernel to Focal I'd consider this a hard blocker. [1]: https://www.kernel.org/doc/Documentation/filesystems/proc.txt see "1.8 Miscellaneous kernel statistics in /proc/stat" [2]: http://manpages.ubuntu.com/manpages/focal/man5/proc.5.html see "(43) guest_time %lu" [3]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=87fa7f3e98a1310ef1ac1900e7ee7f9610a038bc [4]: https://bugzilla.kernel.org/show_bug.cgi?id=209831 [5]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986371 [6]: https://lists.gnu.org/archive/html/qemu-devel/2021-04/msg00732.html [7]: https://lore.kernel.org/kvm/1617011036-11734-1-git-send-email-wanpen...@tencent.com/ [8]: https://lore.kernel.org/kvm/20210206004218.312023-1-sea...@google.com/ ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: linux-image-5.11.0-13-generic 5.11.0-13.14 ProcVersionSignature: Ubuntu 5.11.0-13.14-generic 5.11.7 Uname: Linux 5.11.0-13-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Apr 6 12:56 seq crw-rw---- 1 root audio 116, 33 Apr 6 12:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu61 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CasperMD5CheckResult: unknown Date: Wed Apr 7 05:02:03 2021 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: HP ProLiant DL360 Gen9 PciMultimedia: ProcFB: 0 mgag200drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-13-generic root=UUID=c941b173-e6b5-485a-a02b-8d966b8d3c73 ro --- console=ttyS1,115200 RelatedPackageVersions: linux-restricted-modules-5.11.0-13-generic N/A linux-backports-modules-5.11.0-13-generic N/A linux-firmware 1.196 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 01/22/2018 dmi.bios.release: 2.56 dmi.bios.vendor: HP dmi.bios.version: P89 dmi.board.name: ProLiant DL360 Gen9 dmi.board.vendor: HP dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.ec.firmware.release: 2.60 dmi.modalias: dmi:bvnHP:bvrP89:bd01/22/2018:br2.56:efr2.60:svnHP:pnProLiantDL360Gen9:pvr:rvnHP:rnProLiantDL360Gen9:rvr:cvnHP:ct23:cvr: dmi.product.family: ProLiant dmi.product.name: ProLiant DL360 Gen9 dmi.product.sku: 780018-S01 dmi.sys.vendor: HP To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1922846/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
