Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 15:08:27 -0500 Hector Gonzalez Jaime via Dng wrote: > > Issueing root@bobby:~# cat /sys/block/vda/queue/scheduler gives: > > > > [mq-deadline] none > > > > Is it wrong? > > It's as it should be. Did you check this on the hypervisor? The use of > vda suggests this was checked on a VM, please check the physical host, > which is the one doing the I/O for your VM. Yes, the phisical host says the very same: root@archimede:~# cat /sys/block/sda/queue/scheduler [mq-deadline] none > The physical host is also > the one that needs to have a few dedicated processors to perform I/O for > the VMs. Yes it has free processors. Luciano. -- /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 02485781 FAX: +39 0248028247 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 21:56:43 +0200, Luciano wrote in message <4mswlm0ldwz1f...@baobab.bilink.it>: > On Wed, 14 Sep 2022 21:16:27 +0200 > Arnt Karlsen wrote: > > > > echo "deadline" >/sys/block/sda/queue/scheduler > > > > > > > ..a possibly simpler idea: Give the vm another cpu thread to > > help do the i/o work? > It already has twelve cpu theads. ..ok, wrong diagnosis on my part, I assumed you only gave it one thread, with 12, your i/o problems are somewhere else. > Is there a way to reserve some of them to I/O? ..anyone? I have read somewhere(?) that you can assign (or somesuch) a program to run on a specific cpu, if that helps your search for ideas. > Luciano. -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On 9/14/22 14:54, Luciano Mannucci wrote: On Wed, 14 Sep 2022 12:37:41 -0500 Hector Gonzalez Jaime via Dng wrote: kernel:[ 7336.007287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] if I write to the disk via dd nothing wrong happens... Luciano. Check which scheduler you are using, for virtual machine loads you might want to use "deadline", assuming your disk is sda, the first command checks your scheduler, the second changes to deadline. cat /sys/block/sda/queue/scheduler echo "deadline" >/sys/block/sda/queue/schedule Well, the disk seems to be "vda". Issueing root@bobby:~# cat /sys/block/vda/queue/scheduler gives: [mq-deadline] none Is it wrong? It's as it should be. Did you check this on the hypervisor? The use of vda suggests this was checked on a VM, please check the physical host, which is the one doing the I/O for your VM. The physical host is also the one that needs to have a few dedicated processors to perform I/O for the VMs. Luciano. -- Hector Gonzalez ca...@genac.org ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 21:16:27 +0200 Arnt Karlsen wrote: > > echo "deadline" >/sys/block/sda/queue/scheduler > > > > ..a possibly simpler idea: Give the vm another cpu thread to > help do the i/o work? It already has twelve cpu theads. Is there a way to reserve some of them to I/O? Luciano. -- /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 02485781 FAX: +39 0248028247 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 12:37:41 -0500 Hector Gonzalez Jaime via Dng wrote: > >kernel:[ 7336.007287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! > > [swapper/0:0] > > > > if I write to the disk via dd nothing wrong happens... > > > > Luciano. > > Check which scheduler you are using, for virtual machine loads you might > want to use "deadline", assuming your disk is sda, the first command > checks your scheduler, the second changes to deadline. > > cat /sys/block/sda/queue/scheduler > > echo "deadline" >/sys/block/sda/queue/schedule Well, the disk seems to be "vda". Issueing root@bobby:~# cat /sys/block/vda/queue/scheduler gives: [mq-deadline] none Is it wrong? Luciano. -- /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 02485781 FAX: +39 0248028247 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 12:37:41 -0500, Hector wrote in message <36d765d0-e9a7-ccaa-8b36-098e1f275...@genac.org>: > On 9/14/22 10:02, Luciano Mannucci wrote: > > On Wed, 14 Sep 2022 12:49:19 +0200 > > Luciano Mannucci wrote: > > > >>> vm.dirty_background_bytes=67108864 > >>> vm.dirty_bytes=268435456 > >>> > >>> Maybe this additional information is helpful: > >>> > >>> https://forum.proxmox.com/threads/io-performance-tuning.15893/ > >>> https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ > >>> > >>> Hope that helps, > >> Yes, it does! > >> Works like a charm! > > I've been to quick... > > Now only if the data comes from the local LAN (not drossing routers > > or firewalls) I still get > > > >kernel:[ 7336.007287] watchdog: BUG: soft lockup - CPU#0 stuck > > for 22s! [swapper/0:0] > > > > if I write to the disk via dd nothing wrong happens... > > > > Luciano. > > Check which scheduler you are using, for virtual machine loads you > might want to use "deadline", assuming your disk is sda, the first > command checks your scheduler, the second changes to deadline. > > cat /sys/block/sda/queue/scheduler > > echo "deadline" >/sys/block/sda/queue/scheduler > ..a possibly simpler idea: Give the vm another cpu thread to help do the i/o work? -- ..med vennlig hilsen = with Kind Regards from Arnt Karlsen ...with a number of polar bear hunters in his ancestry... Scenarios always come in sets of three: best case, worst case, and just in case. ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On 9/14/22 10:02, Luciano Mannucci wrote: On Wed, 14 Sep 2022 12:49:19 +0200 Luciano Mannucci wrote: vm.dirty_background_bytes=67108864 vm.dirty_bytes=268435456 Maybe this additional information is helpful: https://forum.proxmox.com/threads/io-performance-tuning.15893/ https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ Hope that helps, Yes, it does! Works like a charm! I've been to quick... Now only if the data comes from the local LAN (not drossing routers or firewalls) I still get kernel:[ 7336.007287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] if I write to the disk via dd nothing wrong happens... Luciano. Check which scheduler you are using, for virtual machine loads you might want to use "deadline", assuming your disk is sda, the first command checks your scheduler, the second changes to deadline. cat /sys/block/sda/queue/scheduler echo "deadline" >/sys/block/sda/queue/scheduler -- Hector Gonzalez ca...@genac.org ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 12:49:19 +0200 Luciano Mannucci wrote: > > vm.dirty_background_bytes=67108864 > > vm.dirty_bytes=268435456 > > > > Maybe this additional information is helpful: > > > > https://forum.proxmox.com/threads/io-performance-tuning.15893/ > > https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ > > > > Hope that helps, > Yes, it does! > Works like a charm! I've been to quick... Now only if the data comes from the local LAN (not drossing routers or firewalls) I still get kernel:[ 7336.007287] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] if I write to the disk via dd nothing wrong happens... Luciano. -- /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 02485781 FAX: +39 0248028247 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
On Wed, 14 Sep 2022 11:04:39 +0200 Andreas Messer wrote: > # Start background writing when more than 128MB data are in write cache > # This value is tuned regarding write performance of HDD ~ 100MB > vm.dirty_background_bytes=67108864 > vm.dirty_bytes=268435456 > > Maybe this additional information is helpful: > > https://forum.proxmox.com/threads/io-performance-tuning.15893/ > https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ > > Hope that helps, Yes, it does! Works like a charm! Many thanks, Luciano. -- /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 02485781 FAX: +39 0248028247 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
Hi Luciano, Am Wed, Sep 14, 2022 at 07:24:07AM +0200 schrieb Luciano Mannucci: > hello all! > > I have a virtual machine running under kvm who started hanging giving > this message just before it dies: > > kernel:[ 296.013011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! > [swapper/0:0] > > This happens only on high i/o load. > The other virtual machines are all running with no problems. > What should I do? The message actually means, that moving memory pages to/from swap space took much longer than the kernel expects. This can happen when another process is using the entire I/O bandwidth to the disk. I had similar issues with my Desktop PC. It turned out this was somehow related to the 32GB RAM of my machine. When a process writes files, the kernel will cache the data first and executes the actual disk writes later depending on cache fill and time. When a process produces data very fast, the cache will grow more and more even while the kernel is already writing data out to disk and at some point an internal threshold in the kernel is hit. (/proc/sys/vm/dirty_ratio) At this time, the kernel will block all processes writing to disks and flush the entire cache content to the disk. If you have a lot of RAM, this flushing can take a lot of time (seconds till minutes). Large RAM machines are affected by this since the threshold is by default set as ratio from f RAM memory. I mitigated this by reconfigure the so called background write threshold cat /etc/sysctl.d/tuning.conf # The following settings are to avoid long application stalls when # writing large files to disk. They lower the amount of write # cached data in RAM until actual writing occurs. This will prevent # the system from writing data in large chunks while everything # else blocks. So this improves the latency of the desktop # The values are by defaulted computed as fraction of the main memory # which results in fairly large cached unwritten data on high memory # systems # Start background writing when more than 128MB data are in write cache # This value is tuned regarding write performance of HDD ~ 100MB vm.dirty_background_bytes=67108864 vm.dirty_bytes=268435456 Maybe this additional information is helpful: https://forum.proxmox.com/threads/io-performance-tuning.15893/ https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/ Hope that helps, cheers, Andreas -- gnuPG keyid: 8C2BAF51 fingerprint: 28EE 8438 E688 D992 3661 C753 90B3 BAAA 8C2B AF51 signature.asc Description: PGP signature ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
Re: [DNG] Chimaera CPU stuck
> On 14 Sep 2022, at 15:24, Luciano Mannucci wrote: > > hello all! > > I have a virtual machine running under kvm who started hanging giving > this message just before it dies: > > kernel:[ 296.013011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! > [swapper/0:0] > > This happens only on high i/o load. > The other virtual machines are all running with no problems. > What should I do? > > luciano. Have you tried upgrading to the kernel from chimaera-backports? Add sources for backports repository from https://www.devuan.org/os/packages and then `apt-get update && apt-get install -t chimaera-backports linux-image-amd64` should do it.___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
[DNG] Chimaera CPU stuck
hello all! I have a virtual machine running under kvm who started hanging giving this message just before it dies: kernel:[ 296.013011] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] This happens only on high i/o load. The other virtual machines are all running with no problems. What should I do? luciano. -- /"\ /Via A. Salaino, 7 - 20144 Milano (Italy) \ / ASCII RIBBON CAMPAIGN / PHONE : +39 02485781 FAX: +39 0248028247 X AGAINST HTML MAIL/ E-MAIL: posthams...@sublink.sublink.org / \ AND POSTINGS/ WWW: http://www.lesassaie.IT/ ___ Dng mailing list Dng@lists.dyne.org https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng
