[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
The morning copy ran without incident but the evening copy triggered a fault between 1944 and 1952 CST (UTC-5). I'm attaching a TAR which contains three screenshots (before, during, and just after the fault) as well as the time-relevant output of 'journalctl', and an sosreport. The sosreport contains 'sar' data which can be nicely graphed by 'ksar' (https://github.com/vlsi/ksar) if you, like me, would rather see a graph of the values instead of reading them line-by-line. The relevant systemd-oomd text today was this: =-=-=-=-=- Nov 13 19:44:41 Boromir systemd-oomd[1084]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-d21ca39b-f66d-4b6c-aa6b-e087b2171268.scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 59.43% > 50.00% for > 20s with reclaim activity Nov 13 19:44:41 Boromir systemd[2457]: vte-spawn-d21ca39b-f66d-4b6c-aa6b-e087b2171268.scope: systemd-oomd killed 2 process(es) in this unit. Nov 13 19:44:57 Boromir systemd-oomd[1084]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-552a2d8e-2b19-483b-824a-dd4a5138012b.scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 60.58% > 50.00% for > 20s with reclaim activity Nov 13 19:51:55 Boromir systemd-oomd[1084]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-48215c41-bdeb-4430-a3f1-eae7f6af30d3.scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 67.81% > 50.00% for > 20s with reclaim activity =-=-=-=-=- The immediately obvious symptom before oomd kicked in was the load average climbing and the system responsiveness falling. I tried taking two other screenshots during that time (load avg 30-40) but the clipboard didn't update or Gimp hung. ** Attachment added: "OOM_Fault_20221113_1952.tar" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1985887/+attachment/5630682/+files/OOM_Fault_20221113_1952.tar -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Incomplete Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
Starting today I will be running the copy next to an htop window ready to capture screenshots (before, during, and after failure). The first run today did not fail. I used a variation of your script to copy the source file multiple times to different file names, in the hope that taking my main storage to nearly-full might have an impact but it still did not fail. There will be another copy tonight / tomorrow. I will keep you updated. Thank you for your aid in trying to reproduce this on your system, too! -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Incomplete Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
Yesterday's copies did not trigger systemd-oomd, but one of them did cause systemd-journald to timeout and restart. Oct 16 14:00:07 Boromir systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)! Oct 16 14:00:07 Boromir systemd[1]: systemd-journald.service: Killing process 578 (systemd-journal) with signal SIGABRT. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Incomplete Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
1. "ionice -c3 nice cp -a SRC DST" is mandatory? Does it reproducible through a normal "cp -a"? A: Unknown. I've been using that out of habit for years to retain as much responsiveness as possible. For future copies I'll use just 'cp -a'. 2. What's the fail rate? A: Over the last two weeks I had four such files to copy and two failed on their first attempt. So I'd say 2/6 total failure rate. 3. Would you please share the data type from "SRC"? a single large 29,613MB file? or a lot of small files? or 300MB * 100 files? A: Sorry, I should have said that earlier. It's a single MP4 file, the size for the past three months has averaged 27,283MB (20 to 36GB). The two associated with the most recent OOMs were 30,710MB (stopped around 10.7 GB transferred) and 29,458MB. The additional load of extraneous processes varies (Firefox, LibreOffice, Chrome, Thunderbird, etc). My own failed attempts to reproduce this on-demand were initially done with no other programs running. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Incomplete Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
I've had two more instances, one on the 18th and one (multiple, actually) today. Both were triggered by copying a file as I'd described (from CLI, "ionice -c3 nice cp -a SRC DST"). Today's OOM events also killed a Firefox window for some reason. I captured an sosreport on both days, but last week's event showed I'd neglected to activate sysstat/sar. I am only uploading today's sosreport, but if the older one may be useful I can add it as well. All of the recent OOMs had this reason, with a varying initial percentage: "due to memory pressure for /user.slice/user-1000.slice/user@1000.service being 99.00% > 50.00% for > 20s with reclaim activity" ** Attachment added: "SOS report and systemd-oomd status" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1985887/+attachment/5619231/+files/sosreport_20220926.tar -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Incomplete Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
Embarrassingly it seems that this week I'm getting more proper behavior. I will keep trying to replicate here as time permits, but the system applied some updates automatically and I also installed a few things since the first incident (sane, wireshark, etc). Rolling everything back isn't much of an option for me right now. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Incomplete Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1985887] Re: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s)
I have a 'normal use case' that triggers this: Copying a large file. My system has 32GB of RAM and 4GB of swap. Trying to copy a 29,613MB file from a reasonably fast SD card to my local (cacheless) NVMe SSD is triggering systemd-oomd. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1985887 Title: systemd kills gnome-shell or gnome-terminal if gnome-terminal uses much memory (50% over 20s) Status in OEM Priority Project: Invalid Status in systemd package in Ubuntu: Invalid Bug description: [Steps to reproduce] 0. Install Jammy image 1. open gnome terminal 2. issue stress_ng or Canonical certification tool checkbox as "checkbox-cli run com.canonical.certification::memory/memory_stress_ng" or "stress-ng --stack 0 --timeout 300" 3. Terminal or Gnome-shell will be killed by systemd-oomd It's because all stressors are under same cgroup belongs to terminal. Both Wayland and Xorg can reproduce. over ssh and in multi-user.target work good. To manage notifications about this bug go to: https://bugs.launchpad.net/oem-priority/+bug/1985887/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
