Re: [Kernel-packages] [Bug 1861359] Re: swap storms kills interactive use
On Sat, Feb 29, 2020 at 10:06:42AM -, Andrea Righi wrote: > Seth, can you try to see if you can reproduce the problem with the > latest unstable kernel (5.4.0-17.21)? https://launchpad.net/~canonical- > kernel-team/+archive/ubuntu/unstable > > I can't reproduce the problem with it. I have not verified yet, but I > suspect it might be related to this commit: > > d92ff805127f mm: move_pages: report the number of non-attempted pages I've installed this kernel, but haven't rebooted into it. I noticed while installing it, I got a zillion messages: depmod: ERROR: ../libkmod/libkmod.c:515 lookup_builtin_file() could not open builtin file '/lib/modules/5.4.0-17-generic/modules.builtin.bin' Both linux-modules-5.4.0-17-generic and linux-modules-extra-5.4.0-17-generic were installed, so I'm not sure what's gone wrong here. -- Steve Beattie http://NxNW.org/~steve/ -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1861359 Title: swap storms kills interactive use Status in linux package in Ubuntu: Confirmed Bug description: Hello, several times since upgrading to focal from 19.04 I've found my computer entirely unresponsive for periods of twenty or thirty seconds. No mouse movement, no keyboard input, the screen output does not change. My computer was using swap space and despite very slow writeout speeds well below what the NVME drive can handle, the computer was unusable. I've captured some vmstat 1 output and top output that I started collecting during the event. (Normally one very long painful period is followed by several shorter periods of uselessness.) Thanks ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.4.0-12-generic 5.4.0-12.15 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 Uname: Linux 5.4.0-12-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu15 Architecture: amd64 Date: Wed Jan 29 23:44:05 2020 ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-signed-5.4 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-12-generic. ApportVersion: 2.20.11-0ubuntu16 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: sarnold2734 F pulseaudio /dev/snd/controlC1: sarnold2734 F pulseaudio Card0.Amixer.info: Card hw:0 'PCH'/'HDA Intel PCH at 0x2fe1028000 irq 145' Mixer name : 'Realtek ALC285' Components : 'HDA:10ec0285,17aa225c,0012 HDA:8086280b,80860101,0010' Controls : 53 Simple ctrls : 15 Card1.Amixer.info: Card hw:1 'Audio'/'Generic ThinkPad Dock USB Audio at usb-:00:14.0-4.2.4, high speed' Mixer name : 'USB Mixer' Components : 'USB17ef:306f' Controls : 9 Simple ctrls : 4 DistroRelease: Ubuntu 20.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: LENOVO 20KHCTO1WW NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu@/vmlinuz-5.4.0-12-generic root=ZFS=rpool/ROOT/ubuntu ro root=ZFS=rpool/ROOT/ubuntu quiet splash acpi_osi=! "acpi_osi=Windows 2015" vt.handoff=1 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 RelatedPackageVersions: linux-restricted-modules-5.4.0-12-generic N/A linux-backports-modules-5.4.0-12-generic N/A linux-firmware1.185 Tags: focal Uname: Linux 5.4.0-12-generic x86_64 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) UserGroups: adm cdrom libvirt lpadmin plugdev sambashare sbuild sudo _MarkForUpload: True dmi.bios.date: 11/25/2019 dmi.bios.vendor: LENOVO dmi.bios.version: N23ET69W (1.44 ) dmi.board.asset.tag: Not Available dmi.board.name: 20KHCTO1WW dmi.board.vendor: LENOVO dmi.board.version: SDK0J40709 WIN dmi.chassis.asset.tag: No Asset Information dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: None dmi.modalias: dmi:bvnLENOVO:bvrN23ET69W(1.44):bd11/25/2019:svnLENOVO:pn20KHCTO1WW:pvrThinkPadX1Carbon6th:rvnLENOVO:rn20KHCTO1WW:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrNone: dmi.product.family: ThinkPad X1 Carbon 6th dmi.product.name: 20KHCTO1WW dmi.product.sku: LENOVO_MT_20KH_BU_Think_FM_ThinkPad X1 Carbon 6th dmi.product.version: ThinkPad X1 Carbon 6th dmi.sys.vendor: LENOVO --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version
Re: [Kernel-packages] [Bug 1861359] Re: swap storms kills interactive use
> What do you have in /proc/sys/vm/swappiness? Could you try to set that > to 0 (kernel prefers to drop file-backed pages instead of swapping out > anonymous pages) and see if the swap out activity is still happening? Unfortunately, this did not solve the problem. Setting swappiness to 0 and re-enabling my swap nearly immediately caused the whole computer to lag whenever firefox had focus -- and swap use jumped to 380M, despite having roughly five gigs free space. Turning swap back off again helped immensely. Thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1861359 Title: swap storms kills interactive use Status in linux package in Ubuntu: Confirmed Bug description: Hello, several times since upgrading to focal from 19.04 I've found my computer entirely unresponsive for periods of twenty or thirty seconds. No mouse movement, no keyboard input, the screen output does not change. My computer was using swap space and despite very slow writeout speeds well below what the NVME drive can handle, the computer was unusable. I've captured some vmstat 1 output and top output that I started collecting during the event. (Normally one very long painful period is followed by several shorter periods of uselessness.) Thanks ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.4.0-12-generic 5.4.0-12.15 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 Uname: Linux 5.4.0-12-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu15 Architecture: amd64 Date: Wed Jan 29 23:44:05 2020 ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-signed-5.4 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-12-generic. ApportVersion: 2.20.11-0ubuntu16 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: sarnold2734 F pulseaudio /dev/snd/controlC1: sarnold2734 F pulseaudio Card0.Amixer.info: Card hw:0 'PCH'/'HDA Intel PCH at 0x2fe1028000 irq 145' Mixer name : 'Realtek ALC285' Components : 'HDA:10ec0285,17aa225c,0012 HDA:8086280b,80860101,0010' Controls : 53 Simple ctrls : 15 Card1.Amixer.info: Card hw:1 'Audio'/'Generic ThinkPad Dock USB Audio at usb-:00:14.0-4.2.4, high speed' Mixer name : 'USB Mixer' Components : 'USB17ef:306f' Controls : 9 Simple ctrls : 4 DistroRelease: Ubuntu 20.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: LENOVO 20KHCTO1WW NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu@/vmlinuz-5.4.0-12-generic root=ZFS=rpool/ROOT/ubuntu ro root=ZFS=rpool/ROOT/ubuntu quiet splash acpi_osi=! "acpi_osi=Windows 2015" vt.handoff=1 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 RelatedPackageVersions: linux-restricted-modules-5.4.0-12-generic N/A linux-backports-modules-5.4.0-12-generic N/A linux-firmware1.185 Tags: focal Uname: Linux 5.4.0-12-generic x86_64 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) UserGroups: adm cdrom libvirt lpadmin plugdev sambashare sbuild sudo _MarkForUpload: True dmi.bios.date: 11/25/2019 dmi.bios.vendor: LENOVO dmi.bios.version: N23ET69W (1.44 ) dmi.board.asset.tag: Not Available dmi.board.name: 20KHCTO1WW dmi.board.vendor: LENOVO dmi.board.version: SDK0J40709 WIN dmi.chassis.asset.tag: No Asset Information dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: None dmi.modalias: dmi:bvnLENOVO:bvrN23ET69W(1.44):bd11/25/2019:svnLENOVO:pn20KHCTO1WW:pvrThinkPadX1Carbon6th:rvnLENOVO:rn20KHCTO1WW:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrNone: dmi.product.family: ThinkPad X1 Carbon 6th dmi.product.name: 20KHCTO1WW dmi.product.sku: LENOVO_MT_20KH_BU_Think_FM_ThinkPad X1 Carbon 6th dmi.product.version: ThinkPad X1 Carbon 6th dmi.sys.vendor: LENOVO --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-12-generic. ApportVersion: 2.20.11-0ubuntu16 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: sarnold2734 F pulseaudio /dev/snd/controlC1: sarnold2734 F pulseaudio Card0.Amixer.info: Card hw:0 'PCH'/'HDA Intel PCH at 0x2fe1028000 irq 145' Mixer name : 'Realtek ALC285'
Re: [Kernel-packages] [Bug 1861359] Re: swap storms kills interactive use
On Mon, Feb 10, 2020 at 07:53:20AM -, Andrea Righi wrote: > OK, so we know that it's not related to the memory cgroup subsystem. But this is a good instinct. It does seem to happen when eg firefox or git is in heavy memory use, not the system as a whole. > Another reason of such unexpected swapping activity could be due to > memory compaction code that is triggering some direct memory reclaim and > forcing to swap out pages. > > What do you have in /proc/sys/vm/swappiness? Could you try to set that > to 0 (kernel prefers to drop file-backed pages instead of swapping out > anonymous pages) and see if the swap out activity is still happening? $ cat /proc/sys/vm/swappiness 60 I'll try 0. > Moreover, we can do some more advanced tracing of the direct memory reclaim > code, for example: > $ sudo kprobe-perf -s 'p:shrink_node' > This doesn't work: # kprobe-perf -s 'p:shrink_node' ERROR: func shrink_node not in /sys/kernel/debug/tracing/available_filter_functions. Either it doesn't exist, or, it might be unsafe to kprobe. Exiting. Use -F to override. # ls -l /sys/kernel/debug/tracing/available_filter_functions ls: cannot access '/sys/kernel/debug/tracing/available_filter_functions': No such file or directory # find /sys/kernel/debug/tracing /sys/kernel/debug/tracing /sys/kernel/debug/tracing/instances /sys/kernel/debug/tracing/trace_stat /sys/kernel/debug/tracing/per_cpu /sys/kernel/debug/tracing/per_cpu/cpu7 /sys/kernel/debug/tracing/per_cpu/cpu6 /sys/kernel/debug/tracing/per_cpu/cpu5 /sys/kernel/debug/tracing/per_cpu/cpu4 /sys/kernel/debug/tracing/per_cpu/cpu3 /sys/kernel/debug/tracing/per_cpu/cpu2 /sys/kernel/debug/tracing/per_cpu/cpu1 /sys/kernel/debug/tracing/per_cpu/cpu0 /sys/kernel/debug/tracing/options # Thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1861359 Title: swap storms kills interactive use Status in linux package in Ubuntu: Confirmed Bug description: Hello, several times since upgrading to focal from 19.04 I've found my computer entirely unresponsive for periods of twenty or thirty seconds. No mouse movement, no keyboard input, the screen output does not change. My computer was using swap space and despite very slow writeout speeds well below what the NVME drive can handle, the computer was unusable. I've captured some vmstat 1 output and top output that I started collecting during the event. (Normally one very long painful period is followed by several shorter periods of uselessness.) Thanks ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.4.0-12-generic 5.4.0-12.15 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 Uname: Linux 5.4.0-12-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu15 Architecture: amd64 Date: Wed Jan 29 23:44:05 2020 ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-signed-5.4 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-12-generic. ApportVersion: 2.20.11-0ubuntu16 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC0: sarnold2734 F pulseaudio /dev/snd/controlC1: sarnold2734 F pulseaudio Card0.Amixer.info: Card hw:0 'PCH'/'HDA Intel PCH at 0x2fe1028000 irq 145' Mixer name : 'Realtek ALC285' Components : 'HDA:10ec0285,17aa225c,0012 HDA:8086280b,80860101,0010' Controls : 53 Simple ctrls : 15 Card1.Amixer.info: Card hw:1 'Audio'/'Generic ThinkPad Dock USB Audio at usb-:00:14.0-4.2.4, high speed' Mixer name : 'USB Mixer' Components : 'USB17ef:306f' Controls : 9 Simple ctrls : 4 DistroRelease: Ubuntu 20.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: LENOVO 20KHCTO1WW NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu@/vmlinuz-5.4.0-12-generic root=ZFS=rpool/ROOT/ubuntu ro root=ZFS=rpool/ROOT/ubuntu quiet splash acpi_osi=! "acpi_osi=Windows 2015" vt.handoff=1 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 RelatedPackageVersions: linux-restricted-modules-5.4.0-12-generic N/A linux-backports-modules-5.4.0-12-generic N/A linux-firmware1.185 Tags: focal Uname: Linux 5.4.0-12-generic x86_64 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) UserGroups: adm cdrom libvirt lpadmin
