Re: systemd-oomd blows away the gnome-shell desktop session
On 7/27/21 10:04 PM, Aleksei Bavshin wrote: IIRC, gnome-terminal supposed to put each tab into a new cgroup. which would address David's problem of losing his desktop session---but is still a bad experience, because you'd be running the compile (or some other memory-intensive process) in a tab, and the tab will simply disappear if OOM happened. This is confusing if you're not paying close attention, and unfriendly because the entire context of that shell session is lost --- I think even the shell command history is not saved. What should happen of course is the memory hog should get killed if it must, back to the shell prompt. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
related: systemd-oomd kills the whole session if it's started from console https://bugzilla.redhat.com/show_bug.cgi?id=1933494 ср, 28 июл. 2021 г. в 10:28, David Airlie : > > Hi, > > I've been using f34 with gnome/wayland session on a 16MB machine which > has 8GB zram swap configured. > > If I build anything large inside my desktop environment (llvm/clang) > from a gnome-terminal, systemd-oomd blows away my whole desktop slice. > This seems overly hostile. > > Now in theory I can use systemd-run to run a shell but my > understanding is the DE is meant to be smarter here. > > Is it at least launching firefox etc in new cgroups or are all > processes on my desktop in one big cgroup? > > Even if it launched gnome-terminal in another group I assume this > would result in it blowing away all my open terminals just to kill the > compiler/linker that is consuming all the RAM/swap. Again this seems > overly hostile. > > Any ideas other than campaigning for systemd-oomd to be turned off again? > > Dave. > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
I suspect that this may have been a swap-based kill and gnome-shell was using the most swap at the time. If you do `journalctl --unit systemd-oomd` do you see "systemd-oomd[]: Killed due to " ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
Hi, On Wed, 2021-07-28 at 13:33 +1000, David Airlie wrote: > > From your description, something obviously went wrong: either > > assignment of cgroups has failed and everything is in the same big > > group, or sd-oomd made a bad shot. systemd-cgls should show which it is. > > Thanks for the hint, systemd-cgls at least makes it appear as if > everything > is in different slice > > ─user.slice > │ └─user-1000.slice > │ ├─user@1000.service > │ │ ├─session.slice > > │ │ ├─app.slice > > │ │ │ ├─app-org.gnome.Terminal.slice > │ │ │ │ ├─vte-spawn-f4a41678-fa09-4ab7-b6d4-d89e18bdb5f4.scope > > I do find it strange it picks > Killed > /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service > > I might have to dig into systemd-oomd to see why it picks totally the > wrong option here. It is mostly going by the rate of "pgscan" in the memory statistics. So, if you are in a heavy swap situation and the kernel tries to evict pages from gnome-shell, then it is perfectly viable for it to become the victim. The main guard we have against that currently is the assigned uresourced memory protection (which is only a very partial guard). i.e. on Fedora Workstation, you should have "uresourced" running, which will assign a memory allocation of 250MiB to the current user and their session.slice. $ cat /sys/fs/cgroup/user.slice/memory.min 262144000 $ cat /sys/fs/cgroup/user.slice/user-1000.slice/memory.min 262144000 $ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/memory.min 262144000 $ cat /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/session.slice/memory.min 262144000 Basically, right now we Hope (TM), that this gives gnome-shell enough space to keep its working set in memory, so that it continues running smoothly and systemd-oomd does not consider it as a candidate. Note that you can configure the size of this memory allocation by editing /etc/uresourced.conf[1]. For now, we don't have another solution. A better solution requires reworking the systemd-oomd victim selection algorithm so that users can steer it away from important processes (i.e. exclude session.slice). I have some ideas for that, but that is pending on having some heuristic whether the killing a cgroup is going to improve the situation. Benjamin [1] To apply, you'll need to run systemctl restart uresourced.service and systemctl --user daemon-reload signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
On Wed, Jul 28, 2021 at 7:14 AM Chris Murphy wrote: > > Also there's below: > https://github.com/facebookincubator/below > > Hopefully it'll get packaged soon. > https://discussion.fedoraproject.org/t/article-proposal-below-an-interactive-resource-monitor-for-modern-linux-systems/31176/2 > This is packaged in Fedora already. "dnf install below" brings you the tool. I'm already using it. :) -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
Also there's below: https://github.com/facebookincubator/below Hopefully it'll get packaged soon. https://discussion.fedoraproject.org/t/article-proposal-below-an-interactive-resource-monitor-for-modern-linux-systems/31176/2 Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service >Is there a command to list the current memory usage for each cgroup? Yes, either in sysfs or you can use systemd-cgls -m, and you can pass a path to eliminate extraneous processes and reveal more detail $ systemd-cgtop -m /user.slice/user-1000.slice/user@1000.service/app.slice Or $ cd /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/app-cgroupify.slice $ grep -R . cgroupify@app-gnome-firefox-2532.scope.service/ We're expecting to get more detailed logging information in the next version of systemd slated for Fedora 35. But nevertheless what you're describing could be a bug so I suggest filing one against systemd, and ideally include the entire journal for this boot, e.g. journalctl -b -o short-monotonic --no-hostname > journal.log and attach it to the bug report. Or at least start 5 minutes of logging before the unexpected behavior. It is uresourced that includes the cgroupify utility that sticks browser tabbed processes into their own cgroup. While uresourced is enabled by default on Workstation, it's not on (yet) on other desktops, but is considered safe to run. So if you're not using Workstation edition you might want to install and enable uresourced, which is itself intended to be temporary. Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
On Wed, Jul 28, 2021 at 12:14 PM Aleksei Bavshin wrote: > > On 7/27/21 6:23 PM, David Airlie wrote: > > Hi, > > > > I've been using f34 with gnome/wayland session on a 16MB machine which > > has 8GB zram swap configured. > > > > If I build anything large inside my desktop environment (llvm/clang) > > from a gnome-terminal, systemd-oomd blows away my whole desktop slice. > > This seems overly hostile. > > > > Now in theory I can use systemd-run to run a shell but my > > understanding is the DE is meant to be smarter here. > > > > Is it at least launching firefox etc in new cgroups or are all > > processes on my desktop in one big cgroup? > > IIRC, gnome-terminal supposed to put each tab into a new cgroup. And > Gnome would create cgroup for each desktop application started via Gnome > shell or glib functions. > While it's still a bit fragile and there's a lot of ways to escape (i.e. > xdg-open, third-party application launchers, starting apps from the > terminal, etc.), normal use in Gnome should work just fine. > > --- > From your description, something obviously went wrong: either > assignment of cgroups has failed and everything is in the same big > group, or sd-oomd made a bad shot. systemd-cgls should show which it is. Thanks for the hint, systemd-cgls at least makes it appear as if everything is in different slice ─user.slice │ └─user-1000.slice │ ├─user@1000.service │ │ ├─session.slice │ │ ├─app.slice │ │ │ ├─app-org.gnome.Terminal.slice │ │ │ │ ├─vte-spawn-f4a41678-fa09-4ab7-b6d4-d89e18bdb5f4.scope I do find it strange it picks Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service I might have to dig into systemd-oomd to see why it picks totally the wrong option here. Is there a command to list the current memory usage for each cgroup? Dave. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: systemd-oomd blows away the gnome-shell desktop session
On 7/27/21 6:23 PM, David Airlie wrote: Hi, I've been using f34 with gnome/wayland session on a 16MB machine which has 8GB zram swap configured. If I build anything large inside my desktop environment (llvm/clang) from a gnome-terminal, systemd-oomd blows away my whole desktop slice. This seems overly hostile. Now in theory I can use systemd-run to run a shell but my understanding is the DE is meant to be smarter here. Is it at least launching firefox etc in new cgroups or are all processes on my desktop in one big cgroup? IIRC, gnome-terminal supposed to put each tab into a new cgroup. And Gnome would create cgroup for each desktop application started via Gnome shell or glib functions. While it's still a bit fragile and there's a lot of ways to escape (i.e. xdg-open, third-party application launchers, starting apps from the terminal, etc.), normal use in Gnome should work just fine. --- From your description, something obviously went wrong: either assignment of cgroups has failed and everything is in the same big group, or sd-oomd made a bad shot. systemd-cgls should show which it is. For the former, I happen to see at least one case where this outright fails - some SELinux user roles don't have access to the required systemd dbus interface and method call with cgroup assignment couldn't be sent. For the latter, I noticed that on my old hw (8GB ram) sd-oomd always preferred to kill firefox instead of gcc. Something made cgroup with firefox less likeable ¯\_(ツ)_/¯ Even if it launched gnome-terminal in another group I assume this would result in it blowing away all my open terminals just to kill the compiler/linker that is consuming all the RAM/swap. Again this seems overly hostile. Any ideas other than campaigning for systemd-oomd to be turned off again? Dave. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure -- With best regards, Aleksei Bavshin ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
systemd-oomd blows away the gnome-shell desktop session
Hi, I've been using f34 with gnome/wayland session on a 16MB machine which has 8GB zram swap configured. If I build anything large inside my desktop environment (llvm/clang) from a gnome-terminal, systemd-oomd blows away my whole desktop slice. This seems overly hostile. Now in theory I can use systemd-run to run a shell but my understanding is the DE is meant to be smarter here. Is it at least launching firefox etc in new cgroups or are all processes on my desktop in one big cgroup? Even if it launched gnome-terminal in another group I assume this would result in it blowing away all my open terminals just to kill the compiler/linker that is consuming all the RAM/swap. Again this seems overly hostile. Any ideas other than campaigning for systemd-oomd to be turned off again? Dave. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
