Re: systemd-oomd issues on desktop
On Tue, Jun 14, 2022 at 4:54 AM Lukas Märdian wrote: > > Am 14.06.22 um 01:22 schrieb Nick Rosbrook: > > On Mon, Jun 13, 2022 at 8:19 AM Lukas Märdian wrote: > >> I wonder if we could use a more selective approach, though, using > >> "OOMScoreAdjust=" in the systemd.exec environment (i.e. Gnome-Shell > >> launcher in Ubuntu's context, as sd-oomd is currently only enabled on > >> Ubuntu Desktop) [2], to reduce the probability of certain "important" > >> apps being killed first, e.g. by maintaining an allow-list of such apps. > > > > IIUC, OOMScoreAdjust only influences the behavior of the kernel OOM > > killer, not systemd-oomd. In any case, I do agree that a "selective > > approach" might improve the current situation (this is what I tried to > > convey in option (2)). > > Indeed. I misread the man page "This setting also applies to > systemd-oomd" => This only applies to "OOMPolicy=" and only as of v251 > so it won't help us here. [1] > > > The ManagedOOMPreference[1] option exists, but this property is > > ignored if the unit's cgroup is not owned by root. The rationale for > > the limitation is described in a link that Michel shared [2]. In > > short, this option is reserved for "critical services" and should be > > used sparingly. But, as Steve said earlier, it is likely that the > > browser is the most important process to many desktop users. Maybe > > this limitation on setting ManagedOOMPreference could be removed, or > > eased to at least allow setting `ManagedOOMPreference=avoid` on > > non-root owned cgroups? For this to be effective, there would need to > > be other viable kill candidates (i.e. cgroups with greater than 5% > > swap usage) when the SwapUsedLimit is met. > > Yes, I feel like allowing a "ManagedOOMPreference=avoid" for critical UI > applications, would be a sensible change to make. This should allow > starting applications like this (possibly integrated with the > gnome-shell launcher for allow-listed applications): > $ systemd-run --user -p ManagedOOMPreference=avoid firefox I have opened an upstream PR to implement this [1], and it seems upstream is OK with the idea in principle, but some more thinking needs to be done before it can be merged. Assuming we can push that change through upstream, service units will immediately benefit because .service files can configure the ManagedOOMPreference property. However, applications which are launched by gnome-shell or snapd run as transient scope units, which means the ManagedOOMPreference property needs to be set when e.g. systemd-run is invoked, as demonstrated in the example above. This means that a bit of integration work will be needed from snapd, gnome-shell, etc. to set ManagedOOMPreference=avoid on _some_ applications. This immediately raises new questions: 1. Which services and applications should be given a setting of ManagedOOMPreference=avoid by default? 2. What is the interface to designate such applications? It seems to me that we would want to have a "single source of truth" from which gnome-shell, snapd, etc. can determine when ManagedOOMPreference=avoid should be set. > Testing option (3) should be as easy as adding a new file in /etc to > override the "ManagedOOMSwap=" setting, using the current systemd build > available in Ubuntu Jammy: > > $ cat /etc/systemd/system/-.slice.d/10-oomd-root-slice-defaults.conf > > [Slice] > > ManagedOOMSwap=auto > I have been running this configuration on my machine for the last week or so, and I reported some of my findings on the LP [2]. For anyone else that has been testing this configuration, please report your findings on the same LP. In the longer term, I think that selectively setting ManagedOOMPreference=avoid on critical applications and services is desirable. But, given the need to improve the current situation ahead of 22.04.1, I think our next step should be to disable swap kill by default (i.e. set ManagedOOMSwap=auto on -.slice). I will continue to work with upstream on the ManagedOOMPreference changes so that we can investigate that option in future releases. Thanks, Nick [1] https://github.com/systemd/systemd/pull/23764 [2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159/comments/18 -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Mon, Jun 13 2022 at 14:18:53 +0200, Lukas Märdian wrote: Am 10.06.22 um 12:17 schrieb Sebastien Bacher: Le 10/06/2022 à 11:40, Julian Andres Klode a écrit : The bug reports we see show that systemd-oomd is working correctly: The browser gets killed, the system remains responsive without having become unresponsive as would be the usual case. It might be working 'correctly' but is not perceived as such by users. I've seen regular complains from users since the release stating that their browser or code editor got closed in front on them without warning, on a machine they had been using for years with the same configuration and software without issue. They might be getting short in resources but in practice they never experienced a sluggish system due to it and just see the feature as buggy. I agree with Julian in that systemd-oomd in general seems to be working as expected. Its purpose is all about jumping in _before_ a system reaches its point of no return and unresponsive swapping death. Therefore, I feel like we should not increase the recommended "SwapUsedLimit=90%" default much higher, i.e. option (1), as that could lead to situations where it's already too late to clear some memory and thus defeat the purpose of having sd-oomd. OTOH, receiving those bug reports shows that sd-oomd is not yet properly optimized either, killing people's "important" applications first (such as the browser). Especially, if the browser applies some memory monitoring on its own to discard/unload unused tabs and free up memory, as suggested by Olivier. The option (3) recommended by Nick, could be one viable option in the Ubuntu context (only 1G swap available) for the time being, until we can have a proper upstream solution (using notifications and hooks) [1]. Thanks for bringing this up with the upstream developers, Michel! I wonder if we could use a more selective approach, though, using "OOMScoreAdjust=" in the systemd.exec environment (i.e. Gnome-Shell launcher in Ubuntu's context, as sd-oomd is currently only enabled on Ubuntu Desktop) [2], to reduce the probability of certain "important" apps being killed first, e.g. by maintaining an allow-list of such apps. Of course we do not want to introduce different classes of apps randomly, but would need to come up with a proper policy of which apps would be eligible to have a lower "OOMScoreAdjust" value. I feel like having individual mechanisms on the application layer to keep memory consumption under control, such as a browser's tab unloading, could be a fair eligibility criteria. I'm not sure the extent to which this would be acceptably backported to 22.04, but I understand that we actually have all the infrastructure in place so that GNOME Shell could adjust the *foreground* application's OOM score. systemd-oomd has generally been a great improvement on my 16GiB RAM + 2GiB swap laptop, turning ~1/day hard lockups due to OOM conditions into something being relatively-cleanly terminated, but it would be nicer still if it could preferentially kill Evolution chugging along in the background rather than the Firefox window I'm currently using (or terminal window I'm currently compiling in). That, and a notification that an OOM condition was averted and was killed to avoid it would be a significant improvement in behaviour. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Tue, Jun 14, 2022 at 06:23:46PM +0200, Julian Andres Klode wrote: On Tue, Jun 14, 2022 at 03:47:58PM +0100, Dave Jones wrote: On Fri, Jun 10, 2022 at 08:16:15PM +0200, Sebastien Bacher wrote: > Le 09/06/2022 à 21:19, Dan Streetman a écrit : > > Personally, I think this is the correct option. 1GB is not a good > > default swap size. > > Did we ever consider doing the same than fedora is doing there? > > https://fedoraproject.org/wiki/Changes/SwapOnZRAM One minor data point here: yes, at least on the Pi images where zswap is activated by default. You actually mean no. I meant "yes, we considered using ZRAM (at least on the Pi images), or more broadly we considered using some form of memory compression, eventually settling on zswap as the solution for the Pi desktop images". -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Tue, Jun 14, 2022 at 03:47:58PM +0100, Dave Jones wrote: > On Fri, Jun 10, 2022 at 08:16:15PM +0200, Sebastien Bacher wrote: > > Le 09/06/2022 à 21:19, Dan Streetman a écrit : > > > Personally, I think this is the correct option. 1GB is not a good > > > default swap size. > > > > Did we ever consider doing the same than fedora is doing there? > > > > https://fedoraproject.org/wiki/Changes/SwapOnZRAM > > One minor data point here: yes, at least on the Pi images where zswap is > activated by default. > > Some background context: one goal on jammy was to make the desktop viable on > the Pi 4 with 2GB of RAM (though not the 1GB). Given that the Pi desktop > images ship with 1GB of swap (so we know there definitely *is* a swap-file You actually mean no. See what Fedora did is they disabled swapfiles/swap partitions, and instead only created a zram device and swap on that, so they only have compressed memory as swap. But also with that the case, one has to wonder how systemd-oomd ever worked for them without any (large) physical swap - they should have hit the same issues we did. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 08:16:15PM +0200, Sebastien Bacher wrote: Le 09/06/2022 à 21:19, Dan Streetman a écrit : Personally, I think this is the correct option. 1GB is not a good default swap size. Did we ever consider doing the same than fedora is doing there? https://fedoraproject.org/wiki/Changes/SwapOnZRAM One minor data point here: yes, at least on the Pi images where zswap is activated by default. Some background context: one goal on jammy was to make the desktop viable on the Pi 4 with 2GB of RAM (though not the 1GB). Given that the Pi desktop images ship with 1GB of swap (so we know there definitely *is* a swap-file that zswap can utilize), that the machine doesn't have upgradeable RAM (though I suppose moving one's storage to a bigger Pi is roughly equivalent), and that I knew oomd was being contemplated, I did a day of testing the 2GB Pi 4 with and without zswap active, and the difference was pretty stark (without zswap the system really didn't feel "useable" due to frequent oomd kills of ff). That said, I'm contemplating bumping the swap (which is allocated on first boot on the Pi images since jammy) up to 2GB for kinetic as that seems to reduce the number of oomd kills noticeably on the more memory constrained Pis (though just anecdotally; I haven't sat down and recorded/analyzed this yet). Dave. -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Mon, Jun 13, 2022 at 8:19 AM Lukas Märdian wrote: > I wonder if we could use a more selective approach, though, using > "OOMScoreAdjust=" in the systemd.exec environment (i.e. Gnome-Shell > launcher in Ubuntu's context, as sd-oomd is currently only enabled on > Ubuntu Desktop) [2], to reduce the probability of certain "important" > apps being killed first, e.g. by maintaining an allow-list of such apps. IIUC, OOMScoreAdjust only influences the behavior of the kernel OOM killer, not systemd-oomd. In any case, I do agree that a "selective approach" might improve the current situation (this is what I tried to convey in option (2)). The ManagedOOMPreference[1] option exists, but this property is ignored if the unit's cgroup is not owned by root. The rationale for the limitation is described in a link that Michel shared [2]. In short, this option is reserved for "critical services" and should be used sparingly. But, as Steve said earlier, it is likely that the browser is the most important process to many desktop users. Maybe this limitation on setting ManagedOOMPreference could be removed, or eased to at least allow setting `ManagedOOMPreference=avoid` on non-root owned cgroups? For this to be effective, there would need to be other viable kill candidates (i.e. cgroups with greater than 5% swap usage) when the SwapUsedLimit is met. Thanks, Nick [1] https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#ManagedOOMPreference=none%7Cavoid%7Comit [2] https://fedoraproject.org/wiki/Changes/EnableSystemdOomd#Can_we_exclude_certain_units_from_being_killed.3F -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Am 14.06.22 um 01:22 schrieb Nick Rosbrook: On Mon, Jun 13, 2022 at 8:19 AM Lukas Märdian wrote: I wonder if we could use a more selective approach, though, using "OOMScoreAdjust=" in the systemd.exec environment (i.e. Gnome-Shell launcher in Ubuntu's context, as sd-oomd is currently only enabled on Ubuntu Desktop) [2], to reduce the probability of certain "important" apps being killed first, e.g. by maintaining an allow-list of such apps. IIUC, OOMScoreAdjust only influences the behavior of the kernel OOM killer, not systemd-oomd. In any case, I do agree that a "selective approach" might improve the current situation (this is what I tried to convey in option (2)). Indeed. I misread the man page "This setting also applies to systemd-oomd" => This only applies to "OOMPolicy=" and only as of v251 so it won't help us here. [1] The ManagedOOMPreference[1] option exists, but this property is ignored if the unit's cgroup is not owned by root. The rationale for the limitation is described in a link that Michel shared [2]. In short, this option is reserved for "critical services" and should be used sparingly. But, as Steve said earlier, it is likely that the browser is the most important process to many desktop users. Maybe this limitation on setting ManagedOOMPreference could be removed, or eased to at least allow setting `ManagedOOMPreference=avoid` on non-root owned cgroups? For this to be effective, there would need to be other viable kill candidates (i.e. cgroups with greater than 5% swap usage) when the SwapUsedLimit is met. Yes, I feel like allowing a "ManagedOOMPreference=avoid" for critical UI applications, would be a sensible change to make. This should allow starting applications like this (possibly integrated with the gnome-shell launcher for allow-listed applications): $ systemd-run --user -p ManagedOOMPreference=avoid firefox Should we setup a PPA to apply this modification for testing it against the "Do not enable swap kill at all" (option 3) approach and ask the affected users [2] to test both methods for the best outcome? Testing option (3) should be as easy as adding a new file in /etc to override the "ManagedOOMSwap=" setting, using the current systemd build available in Ubuntu Jammy: $ cat /etc/systemd/system/-.slice.d/10-oomd-root-slice-defaults.conf [Slice] ManagedOOMSwap=auto -- Lukas Thanks, Nick [1] https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html#ManagedOOMPreference=none%7Cavoid%7Comit [2] https://fedoraproject.org/wiki/Changes/EnableSystemdOomd#Can_we_exclude_certain_units_from_being_killed.3F [1] https://github.com/systemd/systemd/pull/22629 [2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159 -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Am 10.06.22 um 12:17 schrieb Sebastien Bacher: Le 10/06/2022 à 11:40, Julian Andres Klode a écrit : The bug reports we see show that systemd-oomd is working correctly: The browser gets killed, the system remains responsive without having become unresponsive as would be the usual case. It might be working 'correctly' but is not perceived as such by users. I've seen regular complains from users since the release stating that their browser or code editor got closed in front on them without warning, on a machine they had been using for years with the same configuration and software without issue. They might be getting short in resources but in practice they never experienced a sluggish system due to it and just see the feature as buggy. I agree with Julian in that systemd-oomd in general seems to be working as expected. Its purpose is all about jumping in _before_ a system reaches its point of no return and unresponsive swapping death. Therefore, I feel like we should not increase the recommended "SwapUsedLimit=90%" default much higher, i.e. option (1), as that could lead to situations where it's already too late to clear some memory and thus defeat the purpose of having sd-oomd. OTOH, receiving those bug reports shows that sd-oomd is not yet properly optimized either, killing people's "important" applications first (such as the browser). Especially, if the browser applies some memory monitoring on its own to discard/unload unused tabs and free up memory, as suggested by Olivier. The option (3) recommended by Nick, could be one viable option in the Ubuntu context (only 1G swap available) for the time being, until we can have a proper upstream solution (using notifications and hooks) [1]. Thanks for bringing this up with the upstream developers, Michel! I wonder if we could use a more selective approach, though, using "OOMScoreAdjust=" in the systemd.exec environment (i.e. Gnome-Shell launcher in Ubuntu's context, as sd-oomd is currently only enabled on Ubuntu Desktop) [2], to reduce the probability of certain "important" apps being killed first, e.g. by maintaining an allow-list of such apps. Of course we do not want to introduce different classes of apps randomly, but would need to come up with a proper policy of which apps would be eligible to have a lower "OOMScoreAdjust" value. I feel like having individual mechanisms on the application layer to keep memory consumption under control, such as a browser's tab unloading, could be a fair eligibility criteria. Cheers, Lukas [1] https://github.com/systemd/systemd/issues/23606 [2] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#OOMScoreAdjust= -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 2:16 PM Sebastien Bacher wrote: > > Le 09/06/2022 à 21:19, Dan Streetman a écrit : > > Personally, I think this is the correct option. 1GB is not a good > > default swap size. > > Did we ever consider doing the same than fedora is doing there? > > https://fedoraproject.org/wiki/Changes/SwapOnZRAM compressed swap is generally good, when used lightly, and both zram and zswap are good options, however each has significant downsides. For zram, the downside is the unpredictability of pages that need to be swapped, meaning some pages might be very poorly compressible; it would be better to simply write out such pages to disk, but zram (when acting as a swap device) isn't able to 'reject' poorly compressible pages, which means the actual performance improvements of zram can be highly variable depending on workload. Alternately zswap is able to 'reject' poorly compressible pages (i.e. let them get written to disk), but the downside is you actually must have real disk swap configured (because of how zswap is designed, using the 'frontswap' hook); it can't do memory-only swap compression like zram. One downside of both approaches (assuming a zram system also has 'fallback' real disk swap) is zram/zswap will fill up with swapped pages (by design, obviously). If they get filled up (even partially) with pages that really don't need to be accessed for a long time, those compressed pages will stick around in memory, taking up space, when it would have been better to simply write them out to disk. This is particularly problematic once zram or zswap become 'full'. For systems with *only* zram swap, this isn't an issue, since the only options are 1) the page is in ram uncompressed or 2) the page is in zram compresssed, which is why zram is particularly well suited to embedded/iot systems without any disk swap. The fedoraproject page there does lay out the comparison quite well, but I'm not so sure enabling either by default for all users would be appropriate. Also, I moved on from being an active zswap maintainer years ago, so it's possible things have changed. > > Cheers, > Sebastien Bacher > > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 11:25 AM Steve Langasek wrote: > > On Fri, Jun 10, 2022 at 11:40:52AM +0200, Julian Andres Klode wrote: > > On Thu, Jun 09, 2022 at 03:19:36PM -0400, Dan Streetman wrote: > > > > > I think that either option (1) or (3) would be the most reasonable -- > > > > maybe trying (1) first and falling back to (3) if necessary. If anyone > > > > has an opinion on this, or can think of other options, I would > > > > appreciate the input. > > > > Was systemd-oomd enabled by default for a specific reason? The kernel > > > is quite able to handle oom situations itself, and has been for years, > > > so while I'm not trying to suggest systemd-oomd is without any use > > > case, I'm skeptical that systemd-oomd should be enabled *by default*. > > > I think it's more likely to behave better when enabled to address a > > > specific system use case, and leave the default behavior of handling > > > oom to the kernel. > > > No what the kernel does is it starts stuttering, the system becomes > > unresponsive and eventually needs a hard reset maybe. > > > The bug reports we see show that systemd-oomd is working correctly: > > The browser gets killed, the system remains responsive without having > > become unresponsive as would be the usual case. > > If systemd-oomd is killing in-use processes before the user is bothered by > the sluggishness, then it's not working correctly. > > It's difficult to ensure the oom killer is working "correctly" given such a > soft definition, but I agree that the increase in user complaints on 22.04 > indicate we haven't found the right balance yet. I haven't looked at the details of how systemd-oomd works, exactly, nor what the default config is, but I'd suggest the foundations team take a close look at it from a perspective of what issue you want to 'fix' - if that issue is avoiding 'swap hell', then look at systemd-oomd from the perspective of being able to detect high (and 'high' is a subjective term that will differ across systems), sustained (another subjective term) swap reads and writes (and even then, it's not always as simple as 'application X is causing heavy swapping'). If the issue is avoiding actual ENOMEM errors, look at systemd-oomd from the perspective of total free system memory (e.g. is the system in direct reclaim?). I have absolutely no doubt that systemd-oomd is better suited than the kernel to handling oom conditions, when systemd-oomd is configured properly for the system workload; I'm much more skeptical that systemd-oomd can be generically configured to handle detecting "out of memory" for any system workload if configured with generic defaults. > > -- > Steve Langasek Give me a lever long enough and a Free OS > Debian Developer to set it on, and I can move the world. > Ubuntu Developer https://www.debian.org/ > slanga...@ubuntu.com vor...@debian.org > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Thu, Jun 9, 2022 at 4:05 PM Steve Langasek wrote:
>
> On Thu, Jun 09, 2022 at 03:19:36PM -0400, Dan Streetman wrote:
> > > 4. Increase swap on Ubuntu. I am adding this for completeness, but I
> > > doubt this is a viable option.
>
> > Personally, I think this is the correct option. 1GB is not a good
> > default swap size.
>
> Could you elaborate why? This default was arrived at some time ago with
> input from the kernel team regarding the fact that the kernel behaves best
> when it has some swap space but shouldn't necessarily have a lot.
Because swap isn't only used when there is pressure on anonymous
pages, and over time, anonymous pages from applications that don't
access all their memory often can wind up swapped out. The amount of
'useful' swap on a system is absolutely a function of the specific
system's workload as well as overall memory, so coming up with a
default that is 'best' for all systems is no easy task; however IMHO
using an absolute value (e.g. 1GB) is definitely not the 'best'
default; a percentage of total memory, or stepped values based on
system memory ranges (similar to how kdump calculates the amount of
memory to reserve for the kdump kernel) is almost certainly a better
approach.
For some background that might help, when the system experiences
'memory pressure' (in the context of performing swap), it only means
the number of free pages has fallen below the 'high' watermark. At
this point, kswapd is engaged to (in the background) start finding
ways to increase the number of free pages. It (mostly) does this is by
looking through *both* the pagecache as well as anonymous pages, and
attempts to find pages that it can free. The value of 'swappiness'
controls how much it attempts to find pages of each type; a value of
100 means kswapd will look equally at anonymous and pagecache pages,
while by default swappiness is 60, meaing kswapd will try harder to
evict pagecache pages than swapping anonymous pages.
What that means is that swap can be used up over time, even on a
system where there is never any memory pressure from anonymous pages.
As a simple example, setup a system with 1GB swap, and 16GB of ram,
then run:
$ stress-ng -m 1 --vm-bytes 12G --vm-hang 3600 &
( wait 20-30 seconds for the pages to be allocated )
$ sudo dd if=/dev/zero of=/randomfile bs=1024k count=12k
$ for i in {1..10}; do cat /randomfile > /dev/null ; free -h ; done
That will 1) allocate 12G of anonymous pages that the application
isn't using, 2) create a file larger than the remaining free memory,
and 3) cause memory pressure on the pagecache, which (over time)
causes the anonymous pages to get swapped out. Note that at no time
was there any anonymous page memory pressure - meaning, the amount of
anonymous pages was always multiple G lower than the total amount of
memory.
A quick test for me shows I'm able to fill up the entire 1GB of swap
in just a few runs:
totalusedfree shared buff/cache available
Mem:15Gi12Gi 170Mi 2.0Mi 3.3Gi 3.1Gi
Swap: 1.0Gi35Mi 988Mi
totalusedfree shared buff/cache available
Mem:15Gi12Gi 161Mi 2.0Mi 3.3Gi 3.1Gi
Swap: 1.0Gi39Mi 984Mi
totalusedfree shared buff/cache available
Mem:15Gi11Gi 147Mi 0.0Ki 3.8Gi 3.6Gi
Swap: 1.0Gi 519Mi 504Mi
totalusedfree shared buff/cache available
Mem:15Gi11Gi 158Mi 0.0Ki 4.2Gi 4.1Gi
Swap: 1.0Gi 1.0Gi 0B
Also note this *in no way* should be taken as a suggestion that
'swappiness' should be reduced; that is *absolutely* not what I'm
saying and would be a mistake to lower it (by default).
Also I think the discussion of default swap size is (almost)
completely separate from the discussion of systemd-oomd; I think the
current default is set so low that it it's aggravating systemd-oomd's
behavior, but I don't think increasing the default swap size is a
singular 'fix' for any/all issues with systemd-oomd.
>
> If we think 1GB is a wrong value, we should change it from 22.04.1
> forward... but don't have a good way to automatically change the allocation
> for existing installs. Fortunately, our use of swap files mean it's
> possible for the end user to non-destructively increase their swap space,
> but I wouldn't be comfortable with us doing this automatically as part of a
> release upgrade or in an SRU!
>
> --
> Steve Langasek Give me a lever long enough and a Free OS
> Debian Developer to set it on, and I can move the world.
> Ubuntu Developer https://www.debian.org/
> slanga...@ubuntu.com vor...@debian.org
--
ubuntu-devel mailing
Re: systemd-oomd issues on desktop
Hi, I'm part of the team that worked on the Fedora Change Proposal that enabled systemd-oomd by default: https://fedoraproject.org/wiki/Changes/EnableSystemdOomd Joining this discussion late, apologies - I wasn't subscribed to ubuntu-devel prior to today, but will try to catch up on the thread on the archive online. I had a discussion about the lack of notification before a kill with pabs on the debian-mentors channel recently (while packaging psi-notify for Debian; Ubuntu PPA is here: https://launchpad.net/~michel-slm/+archive/ubuntu/psi-notify) (both psi-notify and systemd-oomd act on the same pressure information, but psi-notify just notifies you when certain thresholds while oomd kills the cgroup that causes most of the memory pressure to try and stop getting to a situation where the kernel does it, and likely pick a worse target). pabs and I agreed that having an actionable notification prior to the actual kill would be an improvement, even if the process likely to be killed might differ between the notification and the actual kill, and after discussing this with the team, Anita published this RFC last week: https://github.com/systemd/systemd/issues/23606 Given that we have two major distributions now shipping oomd (Fedora and Ubuntu -- strictly in chronological order, not implying any value judgement here), should we discuss any potential tweak / improvement either on the GitHub issue and/or on the systemd-devel mailing list? https://lists.freedesktop.org/mailman/listinfo/systemd-devel It will probably be good to share as much implementation details as possible and get them applied upstream. Best regards, -- Michel Alexandre Salim identities: https://keyoxide.org/5dce2e7e9c3b1cffd335c1d78b229d2f7ccc04f2 signature.asc Description: This is a digitally signed message part -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
hi, Am Freitag, dem 10.06.2022 um 20:16 +0200 schrieb Sebastien Bacher: > Le 09/06/2022 à 21:19, Dan Streetman a écrit : > > Personally, I think this is the correct option. 1GB is not a good > > default swap size. > > Did we ever consider doing the same than fedora is doing there? > > https://fedoraproject.org/wiki/Changes/SwapOnZRAM > i had implemented it in 11.04 in various arm images via PPAs ... in 12.04 adam (infinity) took the code and uploaded it as "zram-config" to the archive where it still sits in universe ... looking at the changelog, it looks like it was ported to systemd in hirsute, it'd just need a MIR and seeding ... ciao oli signature.asc Description: This is a digitally signed message part -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Le 09/06/2022 à 21:19, Dan Streetman a écrit : Personally, I think this is the correct option. 1GB is not a good default swap size. Did we ever consider doing the same than fedora is doing there? https://fedoraproject.org/wiki/Changes/SwapOnZRAM Cheers, Sebastien Bacher -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Le 10/06/2022 à 17:52, Julian Andres Klode a écrit : but I think the main problem is that there is no desktop integration telling the user why something was killed so they assume malfunction. If we had a popup after an oom event saying The application firefox was closed to maintain system performance we'd have a lot less complaints. I disagree with that. I think the main complain is that the users expect to be able to keep doing what they had started, especially when they had been doing the same type of work for a long time without experiencing any sluggishness. It would probably better to be warning before facts than after, like when the system is getting close from the point displaying a notification, the same way than you get a low battery warning before having the laptop suspending. Cheers, Sebastien Bacher -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
As a user with low RAM on my system (4GB) and a slow HDD (slow disk access means slow swap access), I believe Firefox waits way too long to unload its tabs. If only 5% of swap is free, it's too late. First, because the system has already chugged extremely badly from using any swap in the first place. Second, because at 5% swap left, there is no more room left to maneuver, swap is virtually entirely full at that point, and the system will be stuck for entire minutes waiting for it to get unstuck, juggling pages across extremely small amounts of free swap and free RAM. Which it may not successfully free itself from for hours if there is well and truly too much memory attempted to be used. *I would love it if Firefox/Chromium unloaded tabs sooner, as that would be a much less drastic intervention than killing programs (or even using any swap in the first place, for HDD users!!).* I personally find minimal data loss from unloading tabs. And I am willing to risk that limited/contained amount of data loss, rather than a hard system power-off (or sysrq REISUB) that risks data loss of ALL open work across ALL programs. And furthermore: I'd much rather gamble that *one tab *gets unloaded in a relatively orderly way, and I lose some data, than *all of Firefox *completely and abruptly quits without a chance to internally save data. Or some arbitrary other program, opaquely and seemingly at random from a user's perspective. Side note: Julian Andres Klode on June 10: > If the user is bothered by memory usage and swapping death, then we > might already be in the point of no return territory and it's too > late to be able to kill stuff. > This seems true to me. Running low on both available memory *and* running low on swap simultaneously means you are already painted into a corner. It is a dire spot to be in, performance-wise, as it is hard to find a clean way out without the user manually closing a program, and hoping they can do so before they run out of swap and closing programs becomes so slow that they give up in favor of a hard power-cycle (or sysrq REISUB). Firefox waits a really long time, too long IMO, to unload any tabs. It is deferring the problem until it is already out of hand, IMO. *But *regarding *HDD vs SSD *difference in subjective experience of swap: Swap usage is a terrible experience on a slow HDD. Swap usage might be tolerable (might not even be noticed?) on a fast SSD. I am biased as a slow HDD user. Maybe waiting really long to unload is the better user experience on a fast SSD? Sorry if this is a ramble. Short summary: *If browsers would more-aggressively unload tabs, I think it would be the best move forward.* -- Ubuntu-devel-discuss mailing list Ubuntu-devel-discuss@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel-discuss
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 08:24:41AM -0700, Steve Langasek wrote: > On Fri, Jun 10, 2022 at 11:40:52AM +0200, Julian Andres Klode wrote: > > On Thu, Jun 09, 2022 at 03:19:36PM -0400, Dan Streetman wrote: > > > > > I think that either option (1) or (3) would be the most reasonable -- > > > > maybe trying (1) first and falling back to (3) if necessary. If anyone > > > > has an opinion on this, or can think of other options, I would > > > > appreciate the input. > > > > Was systemd-oomd enabled by default for a specific reason? The kernel > > > is quite able to handle oom situations itself, and has been for years, > > > so while I'm not trying to suggest systemd-oomd is without any use > > > case, I'm skeptical that systemd-oomd should be enabled *by default*. > > > I think it's more likely to behave better when enabled to address a > > > specific system use case, and leave the default behavior of handling > > > oom to the kernel. > > > No what the kernel does is it starts stuttering, the system becomes > > unresponsive and eventually needs a hard reset maybe. > > > The bug reports we see show that systemd-oomd is working correctly: > > The browser gets killed, the system remains responsive without having > > become unresponsive as would be the usual case. > > If systemd-oomd is killing in-use processes before the user is bothered by > the sluggishness, then it's not working correctly. If the user is bothered by memory usage and swapping death, then we might already be in the point of no return territory and it's too late to be able to kill stuff. > > It's difficult to ensure the oom killer is working "correctly" given such a > soft definition, but I agree that the increase in user complaints on 22.04 > indicate we haven't found the right balance yet. It surely is annoying if stuff gets killed, but I think the main problem is that there is no desktop integration telling the user why something was killed so they assume malfunction. If we had a popup after an oom event saying The application firefox was closed to maintain system performance we'd have a lot less complaints. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 11:40:52AM +0200, Julian Andres Klode wrote: > On Thu, Jun 09, 2022 at 03:19:36PM -0400, Dan Streetman wrote: > > > I think that either option (1) or (3) would be the most reasonable -- > > > maybe trying (1) first and falling back to (3) if necessary. If anyone > > > has an opinion on this, or can think of other options, I would > > > appreciate the input. > > Was systemd-oomd enabled by default for a specific reason? The kernel > > is quite able to handle oom situations itself, and has been for years, > > so while I'm not trying to suggest systemd-oomd is without any use > > case, I'm skeptical that systemd-oomd should be enabled *by default*. > > I think it's more likely to behave better when enabled to address a > > specific system use case, and leave the default behavior of handling > > oom to the kernel. > No what the kernel does is it starts stuttering, the system becomes > unresponsive and eventually needs a hard reset maybe. > The bug reports we see show that systemd-oomd is working correctly: > The browser gets killed, the system remains responsive without having > become unresponsive as would be the usual case. If systemd-oomd is killing in-use processes before the user is bothered by the sluggishness, then it's not working correctly. It's difficult to ensure the oom killer is working "correctly" given such a soft definition, but I agree that the increase in user complaints on 22.04 indicate we haven't found the right balance yet. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Jun 9 2022, at 2:19 pm, Dan Streetman wrote: > Was systemd-oomd enabled by default for a specific reason? The kernel > is quite able to handle oom situations itself, and has been for years, > so while I'm not trying to suggest systemd-oomd is without any use > case, I'm skeptical that systemd-oomd should be enabled *by default*. > I think it's more likely to behave better when enabled to address a > specific system use case, and leave the default behavior of handling > oom to the kernel. > What the kernel does hasn't been working and we've been discussing fixes for it for years. The problem is that the kernel only considers processes, not applications. So then when it is freeing up memory it'll kill one process of an application, but not remove the entire application. Systemd, using cgroups, starts to think of applications together so that the memory can be managed in a more sensible manner. My guess, without any data to back it up, is that browsers have benefited from this oversight as they've also put tabs in separate processes and thus avoided OOM action. Ted-- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 5:21 AM Ernst Sjöstrand wrote:
>
> Den fre 10 juni 2022 kl 07:36 skrev Olivier Tilloy
> :
>>
>>
>>
>> On Fri, Jun 10, 2022 at 1:49 AM Steve Langasek
>> wrote:
>>>
>>> On Thu, Jun 09, 2022 at 01:00:45PM -0400, Nick Rosbrook wrote:
>>> > In the reports I refer to above, applications are being killed due to
>>> > (1). In practice, the SwapUsedLimit might be too easy to reach on
>>> > Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we
>>> > follow the suggestion of setting ManagedOOMSwap=kill on the root slice
>>> > [7], every cgroup is eligible for swap kill. When this condition is
>>> > met, user applications like browsers are going to be killed first.
>>>
>>> In terms of behavior that we want to see, this last sentence sets off red
>>> flags for me. There are times when, due to memory pressure, killing
>>> processes to reclaim memory is the right answer; and it is likely that the
>>> process using the most memory on a desktop system is the browser. But in
>>> terms of how a modern desktop is used, it's also quite likely that the
>>> browser is the most important process to the user experience on a desktop.
>>> (Cf. the Chromebook experience, where the browser effectively *is* the
>>> desktop.)
>>>
>>> I understand how we've arrived at the situation that browsers are the
>>> processes likely to be killed first when there's memory pressure; but
>>> separately from the question of what we should do for systemd-oomd overall,
>>> are there configuration changes we could make to lower the priority of the
>>> browser as a candidate for oom killing, and would those be reasonable
>>> configuration changes to make?
>>
>>
>> Also note that modern browsers (at least firefox and chrom{e,ium}) have
>> built-in mechanisms to discard/unload tabs they consider inactive to reclaim
>> memory, and these mechanisms are enabled by default. See about:unloads in
>> firefox, and chrome://discards in chromium. So it really shouldn't be
>> necessary to kill browsers the hard way.
>
>
> Yeah Firefox tracks memory pressure like this:
> https://hg.mozilla.org/mozilla-central/rev/267c8b31a3633ddfb4d7e29af56c82fc8745c0d0
>
> So systemd-oomd should trigger later than the Firefox memory pressure system.
>
> Regards
> //Ernst
This might be a silly suggestion, but it wouldn't be too hard to
implement. Why not:
1. Turn systemd-oomd off.
2. Enable the OOM killer Magic SysRq key sequence by default.
3. When the system is nearing the scary level of low memory, pop up a
message stating "You're almost out of memory. It's highly recommended
that you close some open programs immediately. If the system locks up,
press Alt+Print Screen+ to terminate a memory-hogging
application and restore functionality." (Obviously, replace
with the proper key, I don't remember which key is used to trigger the
Magic SysRq OOM killer, but I know that's a feature.)
That would give users a heads-up before things went south, and in the
event the system ran out of memory so fast that it locked up before
the user could kill an open application, they would have the
instructions for fixing the mess, possibly still on the screen in
front of them.
Another possible solution would be to make systemd-oomd start use some
sort of timing feature to determine if the system had locked up, and
then and only then would it kill a process. Not sure how easy that is
to implement, but it would be a little less in the user's face and
less likely to kill important data.
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Den fre 10 juni 2022 kl 07:36 skrev Olivier Tilloy <
olivier.til...@canonical.com>:
>
>
> On Fri, Jun 10, 2022 at 1:49 AM Steve Langasek
> wrote:
>
>> On Thu, Jun 09, 2022 at 01:00:45PM -0400, Nick Rosbrook wrote:
>> > In the reports I refer to above, applications are being killed due to
>> > (1). In practice, the SwapUsedLimit might be too easy to reach on
>> > Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we
>> > follow the suggestion of setting ManagedOOMSwap=kill on the root slice
>> > [7], every cgroup is eligible for swap kill. When this condition is
>> > met, user applications like browsers are going to be killed first.
>>
>> In terms of behavior that we want to see, this last sentence sets off red
>> flags for me. There are times when, due to memory pressure, killing
>> processes to reclaim memory is the right answer; and it is likely that the
>> process using the most memory on a desktop system is the browser. But in
>> terms of how a modern desktop is used, it's also quite likely that the
>> browser is the most important process to the user experience on a
>> desktop.
>> (Cf. the Chromebook experience, where the browser effectively *is* the
>> desktop.)
>>
>> I understand how we've arrived at the situation that browsers are the
>> processes likely to be killed first when there's memory pressure; but
>> separately from the question of what we should do for systemd-oomd
>> overall,
>> are there configuration changes we could make to lower the priority of the
>> browser as a candidate for oom killing, and would those be reasonable
>> configuration changes to make?
>>
>
> Also note that modern browsers (at least firefox and chrom{e,ium}) have
> built-in mechanisms to discard/unload tabs they consider inactive to
> reclaim memory, and these mechanisms are enabled by default. See
> about:unloads in firefox, and chrome://discards in chromium. So it really
> shouldn't be necessary to kill browsers the hard way.
>
Yeah Firefox tracks memory pressure like this:
https://hg.mozilla.org/mozilla-central/rev/267c8b31a3633ddfb4d7e29af56c82fc8745c0d0
So systemd-oomd should trigger later than the Firefox memory pressure
system.
Regards
//Ernst
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Le 10/06/2022 à 11:40, Julian Andres Klode a écrit : The bug reports we see show that systemd-oomd is working correctly: The browser gets killed, the system remains responsive without having become unresponsive as would be the usual case. It might be working 'correctly' but is not perceived as such by users. I've seen regular complains from users since the release stating that their browser or code editor got closed in front on them without warning, on a machine they had been using for years with the same configuration and software without issue. They might be getting short in resources but in practice they never experienced a sluggish system due to it and just see the feature as buggy. Cheers, Sebastien Bacher -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Thu, Jun 09, 2022 at 03:19:36PM -0400, Dan Streetman wrote: > > > > > I think that either option (1) or (3) would be the most reasonable -- > > maybe trying (1) first and falling back to (3) if necessary. If anyone > > has an opinion on this, or can think of other options, I would > > appreciate the input. > > Was systemd-oomd enabled by default for a specific reason? The kernel > is quite able to handle oom situations itself, and has been for years, > so while I'm not trying to suggest systemd-oomd is without any use > case, I'm skeptical that systemd-oomd should be enabled *by default*. > I think it's more likely to behave better when enabled to address a > specific system use case, and leave the default behavior of handling > oom to the kernel. No what the kernel does is it starts stuttering, the system becomes unresponsive and eventually needs a hard reset maybe. The bug reports we see show that systemd-oomd is working correctly: The browser gets killed, the system remains responsive without having become unresponsive as would be the usual case. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Thu, Jun 09, 2022 at 01:00:45PM -0400, Nick Rosbrook wrote: > Hi, > > During the 22.04 cycle, we enabled systemd-oomd [1] by default on > desktop. Since then, there have been reports of systemd-oomd killing > user applications too frequently (e.g. browsers, IDEs, and gnome-shell > in some cases). In addition to a couple of LPs [2][3], I have heard > these reports by word-of-mouth, and there have been discussions on > internal Mattermost. A common theme in these reports is that e.g. > Chrome is killed "suddenly" without any other observable symptoms of > the system nearing OOM. That's the whole point though, avoiding that slow spiral of death and keeping the system responsive. -- debian developer - deb.li/jak | jak-linux.org - free software dev ubuntu core developer i speak de, en -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
Hi, On Thu, 9 Jun 2022 at 20:20, Dan Streetman wrote: > On Thu, Jun 9, 2022 at 3:03 PM Nick Rosbrook > wrote: > > > > Hi, > > > > During the 22.04 cycle, we enabled systemd-oomd [1] by default on > > desktop. Since then, there have been reports of systemd-oomd killing > > user applications too frequently (e.g. browsers, IDEs, and gnome-shell > > in some cases). In addition to a couple of LPs [2][3], I have heard > > these reports by word-of-mouth, and there have been discussions on > > internal Mattermost. A common theme in these reports is that e.g. > > Chrome is killed "suddenly" without any other observable symptoms of > > the system nearing OOM. > When it comes to OOM situations, how hard would it be to: * provide a status indicator on the status of memory usage, possibly in red if memory usage is a little higher than is safe? * once an application has been killed by the OOMD, how hard is it to tell the user that this has happened to satisfy the OOMD? HTH, Ian -- -- ACCU - Professionalism in programming - http://www.accu.org -- My writing - https://sites.google.com/site/ianbruntlett/ -- Free Software page - https://sites.google.com/site/ianbruntlett/home/free-software -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Fri, Jun 10, 2022 at 1:49 AM Steve Langasek
wrote:
> On Thu, Jun 09, 2022 at 01:00:45PM -0400, Nick Rosbrook wrote:
> > In the reports I refer to above, applications are being killed due to
> > (1). In practice, the SwapUsedLimit might be too easy to reach on
> > Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we
> > follow the suggestion of setting ManagedOOMSwap=kill on the root slice
> > [7], every cgroup is eligible for swap kill. When this condition is
> > met, user applications like browsers are going to be killed first.
>
> In terms of behavior that we want to see, this last sentence sets off red
> flags for me. There are times when, due to memory pressure, killing
> processes to reclaim memory is the right answer; and it is likely that the
> process using the most memory on a desktop system is the browser. But in
> terms of how a modern desktop is used, it's also quite likely that the
> browser is the most important process to the user experience on a desktop.
> (Cf. the Chromebook experience, where the browser effectively *is* the
> desktop.)
>
> I understand how we've arrived at the situation that browsers are the
> processes likely to be killed first when there's memory pressure; but
> separately from the question of what we should do for systemd-oomd overall,
> are there configuration changes we could make to lower the priority of the
> browser as a candidate for oom killing, and would those be reasonable
> configuration changes to make?
>
Also note that modern browsers (at least firefox and chrom{e,ium}) have
built-in mechanisms to discard/unload tabs they consider inactive to
reclaim memory, and these mechanisms are enabled by default. See
about:unloads in firefox, and chrome://discards in chromium. So it really
shouldn't be necessary to kill browsers the hard way.
Olivier
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Thu, Jun 09, 2022 at 01:00:45PM -0400, Nick Rosbrook wrote: > In the reports I refer to above, applications are being killed due to > (1). In practice, the SwapUsedLimit might be too easy to reach on > Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we > follow the suggestion of setting ManagedOOMSwap=kill on the root slice > [7], every cgroup is eligible for swap kill. When this condition is > met, user applications like browsers are going to be killed first. In terms of behavior that we want to see, this last sentence sets off red flags for me. There are times when, due to memory pressure, killing processes to reclaim memory is the right answer; and it is likely that the process using the most memory on a desktop system is the browser. But in terms of how a modern desktop is used, it's also quite likely that the browser is the most important process to the user experience on a desktop. (Cf. the Chromebook experience, where the browser effectively *is* the desktop.) I understand how we've arrived at the situation that browsers are the processes likely to be killed first when there's memory pressure; but separately from the question of what we should do for systemd-oomd overall, are there configuration changes we could make to lower the priority of the browser as a candidate for oom killing, and would those be reasonable configuration changes to make? -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Thu, Jun 09, 2022 at 03:19:36PM -0400, Dan Streetman wrote: > > 4. Increase swap on Ubuntu. I am adding this for completeness, but I > > doubt this is a viable option. > Personally, I think this is the correct option. 1GB is not a good > default swap size. Could you elaborate why? This default was arrived at some time ago with input from the kernel team regarding the fact that the kernel behaves best when it has some swap space but shouldn't necessarily have a lot. If we think 1GB is a wrong value, we should change it from 22.04.1 forward... but don't have a good way to automatically change the allocation for existing installs. Fortunately, our use of swap files mean it's possible for the end user to non-destructively increase their swap space, but I wouldn't be comfortable with us doing this automatically as part of a release upgrade or in an SRU! -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Re: systemd-oomd issues on desktop
On Thu, Jun 9, 2022 at 3:03 PM Nick Rosbrook wrote: > > Hi, > > During the 22.04 cycle, we enabled systemd-oomd [1] by default on > desktop. Since then, there have been reports of systemd-oomd killing > user applications too frequently (e.g. browsers, IDEs, and gnome-shell > in some cases). In addition to a couple of LPs [2][3], I have heard > these reports by word-of-mouth, and there have been discussions on > internal Mattermost. A common theme in these reports is that e.g. > Chrome is killed "suddenly" without any other observable symptoms of > the system nearing OOM. > > For more context, systemd-oomd basically has two methods for deciding > a unit's cgroup is a candidate for OOM kill: > > 1. When total system memory usage _and_ swap usage both exceed > SwapUsedLimit (90% by default, and on Ubuntu) [4], monitored cgoups > with greater than 5% swap usage become OOM kill candidates, and > cgroups with the highest swap usage are acted on first. > > 2. When a unit's cgroup memory pressure exceeds MemoryPressureLimit > [5] for at least MemoryPressureDurationSec [6], monitored descendant > cgroups will be acted on starting from the ones with the most reclaim > activity to the least reclaim activity. > > In the reports I refer to above, applications are being killed due to > (1). In practice, the SwapUsedLimit might be too easy to reach on > Ubuntu, largely because Ubuntu provides just 1GB of swap. Since we > follow the suggestion of setting ManagedOOMSwap=kill on the root slice > [7], every cgroup is eligible for swap kill. When this condition is > met, user applications like browsers are going to be killed first. > > While investigating [2], we patched upstream systemd-oomd to fix how > "used memory" was calculated, and we brought the patch into Jammy. > This may have helped the situation a bit, but it does not appear this > was enough to fix the issue entirely. > > Given the current situation, I think we should re-consider how > systemd-oomd is configured on Ubuntu. These are the options that come > to mind: > > 1. Increase SwapUsedLimit (again, currently at 90%). I think this is > probably the safest change, but it is not clear to me how significant > of an impact this would have. > > 2. Set ManagedOOMSwap more selectively. Again, we currently follow the > recommendation of setting ManagedOOMSwap=kill on the root slice > (-.slice), so every descendant cgroup is a candidate for swap kill. It > _might_ be effective to say "do not swap kill cgroups descendant of > user's app.slice". The downsides of this approach would be that the > configuration does not scale well (i.e. a lot more configuration > needed to get the proper swap kill "coverage"), and this may just > place the problem onto a different class of processes. > > 3. Do not enable swap kill at all. This would mean systemd-oomd would > only act when memory pressure limits are reached. Given Ubuntu's swap > configuration, does it make sense for systemd-oomd to act on high swap > usage? > > 4. Increase swap on Ubuntu. I am adding this for completeness, but I > doubt this is a viable option. Personally, I think this is the correct option. 1GB is not a good default swap size. > > I think that either option (1) or (3) would be the most reasonable -- > maybe trying (1) first and falling back to (3) if necessary. If anyone > has an opinion on this, or can think of other options, I would > appreciate the input. Was systemd-oomd enabled by default for a specific reason? The kernel is quite able to handle oom situations itself, and has been for years, so while I'm not trying to suggest systemd-oomd is without any use case, I'm skeptical that systemd-oomd should be enabled *by default*. I think it's more likely to behave better when enabled to address a specific system use case, and leave the default behavior of handling oom to the kernel. > > Thanks, > > Nick 'enr0n' Rosbrook > > [1] https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html > [2] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1966381 > [3] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1972159 > [4] > https://www.freedesktop.org/software/systemd/man/oomd.conf.html#SwapUsedLimit= > [5] > https://www.freedesktop.org/software/systemd/man/oomd.conf.html#DefaultMemoryPressureLimit= > [6] > https://www.freedesktop.org/software/systemd/man/oomd.conf.html#DefaultMemoryPressureDurationSec= > [7] > https://www.freedesktop.org/software/systemd/man/systemd-oomd.service.html#Usage%20Recommendations > > -- > ubuntu-devel mailing list > ubuntu-devel@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-devel mailing list ubuntu-devel@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
