Re: swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

On Wed, Sep 18, 2019 at 6:57 PM Tom Seewald wrote: > > Hi Chris, > > Does zswap actually keep the data compressed when the DRAM-based swap is > full, and it writes to the spill-over non-volatile swap device? > > I'm not an expert on this at all, however my understanding was that zswap > must

Re: swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

Hi Chris, Does zswap actually keep the data compressed when the DRAM-based swap is full, and it writes to the spill-over non-volatile swap device? I'm not an expert on this at all, however my understanding was that zswap must decompress the data before it writes to the backing swap. But

Re: swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

Zbyszek, Do you have any advice on how to assess 'swap on ZRAM' versus 'zswap' by default for Fedora Workstation? They're really too similar from a user point of view, I think it really comes down to the technical arguments. 1a. 'swap on ZRAM' compresses only that which goes to the ZRAM device

Re: swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

Hi, thank you for all the testing and comparisons between different approaches. It looks really interesting. > The ideal scenario is to get everyone on the same page, and so far it > looks like systemd's zram-generator, built in Rust, meets all the > requirements. That needs to be confirmed, but

Re: swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

On Fri, Aug 30, 2019 at 1:55 PM Chris Murphy wrote: > > Hi, > This is yet another follow-up for this thread: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/XUZLHJ5O32OX24LG44R7UZ2TMN6NY47N/ (Benchmarks being fraught with peril, synthetic benchmarks even

Re: Better interactivity in low-memory situations

Our team at FB is working on a similar (but more generic) solution. All of our work is open source / upstreamed into the linux kernel and we're running it in production on quite a large scale already. Results are very promising. We'll be presenting about it at All Systems Go (multiple talks)

Re: Better interactivity in low-memory situations

> On Mo, 12.08.19 09:40, Chris Murphy (lists(a)colorremedies.com) wrote: > > > Ideally, GNOME would run all its apps as systemd --user services. We > could then set DefaultMemoryHigh= globally for the systemd --user > instance to some percentage value (which is taken relative to the > physical

Re: Better interactivity in low-memory situations

On 9/4/19 3:38 PM, Chris Murphy wrote: > On Wed, Sep 4, 2019 at 1:03 PM Chris Murphy wrote: >> I'm skeptical of swapfiles for this use case, because it likely >> increases the chance for file system disk contention, see section >> 11.17 >>

Re: Better interactivity in low-memory situations

On 9/4/19 3:16 PM, Florian Weimer wrote: > * Kyle Marek: > >> I'm finding that 32G is not necessarily sufficient for compiling clang >> itself. Similarly I've had a hard time compiling UnrealEngine from >> source. I usually see ld using up to 12G of memory to link each >> artifact. Using

Re: Better interactivity in low-memory situations

On Wed, Sep 4, 2019 at 1:03 PM Chris Murphy wrote: > > I'm skeptical of swapfiles for this use case, because it likely > increases the chance for file system disk contention, see section > 11.17 > https://www.kernel.org/doc/gorman/html/understand/understand014.html Actually that looks outdated

Re: Better interactivity in low-memory situations

per cgroup swap file support https://lwn.net/Articles/591495/ This might be interesting. What happens if swap is only assigned to new unprivileged tasks? Your compile might take three days but it doesn't make the GUI fall over? *shrug* --- Chris Murphy

Re: Better interactivity in low-memory situations

* Kyle Marek: > I'm finding that 32G is not necessarily sufficient for compiling clang > itself. Similarly I've had a hard time compiling UnrealEngine from > source. I usually see ld using up to 12G of memory to link each > artifact. Using -j$(nproc) on a 16 VCPU system amplifies the issue. I >

Re: Better interactivity in low-memory situations

On Wed, Sep 4, 2019 at 12:01 PM Kyle Marek wrote: > > I'm finding that 32G is not necessarily sufficient for compiling clang > itself. Similarly I've had a hard time compiling UnrealEngine from > source. I usually see ld using up to 12G of memory to link each > artifact. Using -j$(nproc) on a 16

Re: Better interactivity in low-memory situations

On 9/3/19 12:30 AM, John Harris wrote: > On Tuesday, August 20, 2019 10:48:06 PM MST John Harris wrote: >> On Sunday, August 18, 2019 4:33:47 AM MST Gordan Bobic wrote: >> >>> On Sun, Aug 11, 2019 at 10:36 AM wrote: >>> This seems like a distraction from the real goal here, which is to

Re: Better interactivity in low-memory situations

Chris Murphy writes: > On Mon, Aug 12, 2019 at 5:47 PM Emery Berger wrote: >> >> For what it's worth, my research group attacked basically exactly this >> problem some time ago. We built a modified Linux kernel that we called >> Redline that was utterly resilient to fork bombs, malloc bombs,

Re: Better interactivity in low-memory situations

On Tuesday, August 20, 2019 10:48:06 PM MST John Harris wrote: > On Sunday, August 18, 2019 4:33:47 AM MST Gordan Bobic wrote: > > > On Sun, Aug 11, 2019 at 10:36 AM wrote: > > > > > This seems like a distraction from the real goal here, which is to > > > ensure Fedora remains responsive under

Re: swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

- Original Message - > From: "Chris Murphy" > To: "Development discussions related to Fedora" > > Sent: Friday, August 30, 2019 9:55:52 PM > Subject: swap on ZRAM, zswap, and Rust was: Better interactivity in > low-memory situations &g

Re: testing oomd, cgroupsv2, was: Better interactivity in low-memory situations

Per this suggestion [1] by hakavlad (Alexey Avramov), I did a single test with earlyoom, a user space service that's already packaged for Fedora. I have not yet tested nohang, also mentioned. I chose a configuration that has fairly consistently (>80%) resulted in a total hang for more than 30m.

Re: Better interactivity in low-memory situations

On Mon, Aug 12, 2019 at 5:47 PM Emery Berger wrote: > > For what it's worth, my research group attacked basically exactly this > problem some time ago. We built a modified Linux kernel that we called > Redline that was utterly resilient to fork bombs, malloc bombs, and so on. No > process

swap on ZRAM, zswap, and Rust was: Better interactivity in low-memory situations

Hi, This is yet another follow-up for this thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/XUZLHJ5O32OX24LG44R7UZ2TMN6NY47N/ Basics: "zswap" compresses swap and uses a defined memory pool as a cache, with spill over (still compressed) going into a

testing oomd, cgroupsv2, was: Better interactivity in low-memory situations

Hi, This is a follow-up for this thread: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/XUZLHJ5O32OX24LG44R7UZ2TMN6NY47N/ Has anyone looked at oomd, or is anyone interested in testing and comparing it to alternatives?

Re: Better interactivity in low-memory situations

On Sunday, August 18, 2019 4:33:47 AM MST Gordan Bobic wrote: > On Sun, Aug 11, 2019 at 10:36 AM wrote: > > This seems like a distraction from the real goal here, which is to > > ensure Fedora remains responsive under heavy memory pressure, > > I think this is an overwhelmingly important point,

Re: Better interactivity in low-memory situations

On Tue, Aug 20, 2019 at 2:15 AM Lennart Poettering wrote: > > On Mo, 19.08.19 13:58, Chris Murphy (li...@colorremedies.com) wrote: > > > I'm skeptical as well. But to further explore this: > > > > 1. Does the kernel know better than to write a hibernation image (all > > or part) to a /dev/zram

Re: Better interactivity in low-memory situations

On Mo, 19.08.19 13:58, Chris Murphy (li...@colorremedies.com) wrote: > I'm skeptical as well. But to further explore this: > > 1. Does the kernel know better than to write a hibernation image (all > or part) to a /dev/zram device? e.g. a system with: 8GiB RAM, 8GiB > swap on ZRAM, 8GiB swap

Re: Better interactivity in low-memory situations

On Mon, Aug 12, 2019 at 10:20 AM Lennart Poettering wrote: > > On Mo, 12.08.19 09:40, Chris Murphy (li...@colorremedies.com) wrote: > > > Right now the only lever to avoid swap, is to not create a swap > > partition at installation time. Or create a smaller one instead of 1:1 > > ratio with RAM.

Re: Better interactivity in low-memory situations

This seems very reminiscent of what is often referred to as "swapping insanity", often in the context of MySQL. The root cause there is NUMA with memory allocation only happening on the node that requests the memory, resulting in the possibility of there being plenty of free memory on one node but

Re: Better interactivity in low-memory situations

* Gordan Bobic: > That may be so, but this thread started off with memory pressure also > being an issue for regular desktop x86 use. I think the problem there is that the system has sufficient reclaimable memory, but cannot make that memory available to applications in a timely fashion.

Re: Better interactivity in low-memory situations

For what it's worth, my research group attacked basically exactly this problem quite some time ago. We built a modified Linux kernel that we called Redline that was impervious to fork bombs, malloc bombs, and so on. No process could take down the system, much less unprivileged ones. I think some

Re: Better interactivity in low-memory situations

On Sun, Aug 18, 2019 at 2:55 PM Gordan Bobic wrote: > > On Sun, Aug 18, 2019 at 9:07 PM Kevin Kofler wrote: >> >> Gordan Bobic wrote: >> > Right, but is it better that _everything_ else suffers with more memory >> > pressure for the handful of relatively infrequent use cases for which >> >

Re: Better interactivity in low-memory situations

On Sun, Aug 18, 2019 at 9:07 PM Kevin Kofler wrote: > Gordan Bobic wrote: > > Right, but is it better that _everything_ else suffers with more memory > > pressure for the handful of relatively infrequent use cases for which > > ulimit can be used to explicitly raise the limit? > > Well, as I

Re: Better interactivity in low-memory situations

Gordan Bobic wrote: > Right, but is it better that _everything_ else suffers with more memory > pressure for the handful of relatively infrequent use cases for which > ulimit can be used to explicitly raise the limit? Well, as I wrote, a lower limit might actually make sense on ARM. But modern

Re: Better interactivity in low-memory situations

On Sun, Aug 18, 2019 at 8:51 PM Kevin Kofler wrote: > Gordan Bobic wrote: > > It may be simpler to approach the question from the other side, i.e. is > > there anything that actually ever needs more than 1MB of stack space? If > > there is, I haven't seen it in the decade since I've been using

Re: Better interactivity in low-memory situations

Gordan Bobic wrote: > Adding -ffunction-sections -fdata-sections to defaults can help > considerably in producing smaller binaries, and is not the default. > Linking with -Wl,--gc-sections helps a lot and is not the default Well, -ffunction-sections -fdata-sections -Wl,--gc-sections mostly helps

Re: Better interactivity in low-memory situations

Gordan Bobic wrote: > It may be simpler to approach the question from the other side, i.e. is > there anything that actually ever needs more than 1MB of stack space? If > there is, I haven't seen it in the decade since I've been using this tweak > with various Fedora derived distributions. I've

Re: Better interactivity in low-memory situations

> It may be simpler to approach the question from the other side, i.e. is there > anything that actually ever needs more than 1MB of stack space? If there is, > I haven't seen it in the decade since I've been using this tweak with various > Fedora derived distributions. Any application

Re: Better interactivity in low-memory situations

On Sun, Aug 18, 2019 at 2:33 PM Hans de Goede wrote: > > > > Adding -ffunction-sections -fdata-sections to defaults can help > considerably in producing smaller binaries, and is not the default. > > > Linking with -Wl,--gc-sections helps a lot and is not the default > > > > These

Re: Better interactivity in low-memory situations

Hi, On 18-08-19 15:25, Gordan Bobic wrote: On Sun, Aug 18, 2019 at 2:06 PM Hans de Goede mailto:hdego...@redhat.com>> wrote: Hi, On 18-08-19 13:33, Gordan Bobic wrote: > On Sun, Aug 11, 2019 at 10:36 AM http://gnome.org> > wrote: >  > This seems like a

Re: Better interactivity in low-memory situations

On Sun, Aug 18, 2019 at 2:06 PM Hans de Goede wrote: > Hi, > > On 18-08-19 13:33, Gordan Bobic wrote: > > On Sun, Aug 11, 2019 at 10:36 AM http://gnome.org/>> wrote: > > > This seems like a distraction from the real goal here, which is to > > > ensure Fedora remains responsive under heavy

Re: Better interactivity in low-memory situations

Hi, On 18-08-19 13:33, Gordan Bobic wrote: On Sun, Aug 11, 2019 at 10:36 AM http://gnome.org/>> wrote: > This seems like a distraction from the real goal here, which is to > ensure Fedora remains responsive under heavy memory pressure, I think this is an overwhelmingly important point, and

Re: Better interactivity in low-memory situations

On Sun, Aug 11, 2019 at 10:36 AM wrote: > This seems like a distraction from the real goal here, which is to > ensure Fedora remains responsive under heavy memory pressure, I think this is an overwhelmingly important point, and as somebody regularly working with ARM machines with tiny amounts of

Re: Better interactivity in low-memory situations

On Fri, Aug 16, 2019 at 7:48 AM Chris Murphy wrote: > > On Thu, Aug 15, 2019 at 2:19 PM Chris Murphy wrote: > > > > [ 718.068633] fmac.local kernel: SLUB: Unable to allocate memory on > > node -1, gfp=0x900(GFP_NOWAIT|__GFP_ZERO) > > [ 718.068636] fmac.local kernel: cache: page->ptl, object

Re: Better interactivity in low-memory situations

On Thu, Aug 15, 2019 at 2:19 PM Chris Murphy wrote: > > [ 718.068633] fmac.local kernel: SLUB: Unable to allocate memory on > node -1, gfp=0x900(GFP_NOWAIT|__GFP_ZERO) > [ 718.068636] fmac.local kernel: cache: page->ptl, object size: 72, > buffer size: 72, default order: 0, min order: 0 > [

Re: Better interactivity in low-memory situations

On Thu, Aug 15, 2019 at 1:51 AM Artem Tim wrote: > > BFQ scheduler help a lot with this issue. Using it on Fedora since 4.19 > kernel. Also there was previous discussion about make it default for > Workstation >

Re: Better interactivity in low-memory situations

BFQ scheduler help a lot with this issue. Using it on Fedora since 4.19 kernel. Also there was previous discussion about make it default for Workstation https://lists.fedoraproject.org/archives/list/ker...@lists.fedoraproject.org/message/I2OZWDD4QCDYUXJ5NHYTMGNAB4KLJN2K/

Re: Better interactivity in low-memory situations

(Oops, sorry, re-post because I messed up the threading.) I'm not a developer, nor do I pretend to understand the nuances of memory management. But I signed up for this list just to say "thanks" to all the devs and others that are finally discussing what I consider to be one of the biggest

re: Better interactivity in low-memory situations

I'm not a developer, nor do I pretend to understand the nuances of memory management. But I signed up for this list just to say "thanks" to all the devs and others that are finally discussing what I consider to be one of the biggest problems with Linux on the desktop. My experience with

Re: Better interactivity in low-memory situations

> On 10 Aug 2019, at 17:56, Georg Sauthoff wrote: > > On Fri, Aug 09, 2019 at 03:50:43PM -0600, Chris Murphy wrote: > [..] >> Problem and thesis statement: >> Certain workloads, such as building webkitGTK from source, results in >> heavy swap usage eventually leading to the system becoming

Re: Better interactivity in low-memory situations

On Mon, Aug 12, 2019 at 6:31 PM David Airlie wrote: > > On Sun, Aug 11, 2019 at 2:57 AM Georg Sauthoff wrote: > > > > On Fri, Aug 09, 2019 at 03:50:43PM -0600, Chris Murphy wrote: > > [..] > > > Problem and thesis statement: > > > Certain workloads, such as building webkitGTK from source,

Re: Better interactivity in low-memory situations

On Sun, Aug 11, 2019 at 2:57 AM Georg Sauthoff wrote: > > On Fri, Aug 09, 2019 at 03:50:43PM -0600, Chris Murphy wrote: > [..] > > Problem and thesis statement: > > Certain workloads, such as building webkitGTK from source, results in > > heavy swap usage eventually leading to the system becoming

Re: Better interactivity in low-memory situations

On Mon, Aug 12, 2019 at 11:07 AM Benjamin Kircher wrote: > > (… I definitely need to play around with Silverblue to learn what they are > doing.) I'm pretty sure Silverblue will be rebased on Fedora CoreOS which recently released a preview. I'm not sure what the time frame for that is, but

Re: Better interactivity in low-memory situations

On Mo, 12.08.19 19:06, Benjamin Kircher (benjamin.kirc...@gmail.com) wrote: > > > > On 12. Aug 2019, at 18:16, Lennart Poettering wrote: > > > > On Mo, 12.08.19 09:40, Chris Murphy (li...@colorremedies.com) wrote: > > > >> How to do this automatically? Could there be a mechanism for the > >>

Re: Better interactivity in low-memory situations

> On 12. Aug 2019, at 18:16, Lennart Poettering wrote: > > On Mo, 12.08.19 09:40, Chris Murphy (li...@colorremedies.com) wrote: > >> How to do this automatically? Could there be a mechanism for the >> system and the requesting application to negotiate resources? > > Ideally, GNOME would run

Re: Better interactivity in low-memory situations

> On 12. Aug 2019, at 17:40, Chris Murphy wrote: > > If I just run the example program, let's say systemd MemoryLimit is > set to /proc/meminfo MemAvailable, the program is still going to try > and bust out of that and fail. The failure reason is also non-obvious. > Yes this is definitely an

Re: Better interactivity in low-memory situations

On Mo, 12.08.19 09:40, Chris Murphy (li...@colorremedies.com) wrote: > How to do this automatically? Could there be a mechanism for the > system and the requesting application to negotiate resources? Ideally, GNOME would run all its apps as systemd --user services. We could then set

Re: Better interactivity in low-memory situations

On Mon, Aug 12, 2019 at 1:01 AM Florian Weimer wrote: > > * Chris Murphy: > > > Summary of findings (restated, but basically the same as found at [2]): > > Test system, Macbook Pro, Intel Core i7-2820QM (4/8 cores), 8GiB RAM, > > Samsung SSD 840 EVO, Fedora Rawhide Workstation. > > Do you use the

Re: Better interactivity in low-memory situations

On Mon, Aug 12, 2019 at 12:30 AM Benjamin Kircher wrote: > > > > > On 11. Aug 2019, at 23:05, Chris Murphy wrote: > > > > I think the point at which the mouse pointer has frozen, the user has > > no practical means of controlling or interacting with the system, it's > > a failure. > > > > In the

Re: Better interactivity in low-memory situations

* Petr Pisar: > On 2019-08-12, Florian Weimer wrote: >> Do you use the built-in Intel graphics? Can you test with something >> else? >> > Does it have any effect? It happens to me even with a discrete GPU. I expect that the GEM shrinker (or rather, the reason why it is needed) radically

Re: Better interactivity in low-memory situations

On 2019-08-12, Florian Weimer wrote: > Do you use the built-in Intel graphics? Can you test with something > else? > Does it have any effect? It happens to me even with a discrete GPU. As far as I know integrated graphics arrays do not share physical memory from point of view of the CPU address

Re: Better interactivity in low-memory situations

* Chris Murphy: > Summary of findings (restated, but basically the same as found at [2]): > Test system, Macbook Pro, Intel Core i7-2820QM (4/8 cores), 8GiB RAM, > Samsung SSD 840 EVO, Fedora Rawhide Workstation. Do you use the built-in Intel graphics? Can you test with something else? Thanks,

Re: Better interactivity in low-memory situations

> On 11. Aug 2019, at 23:05, Chris Murphy wrote: > > I think the point at which the mouse pointer has frozen, the user has > no practical means of controlling or interacting with the system, it's > a failure. > > In the short term, is it reasonable and possible, to get the oom > killer to

Re: Better interactivity in low-memory situations

On Sun, Aug 11, 2019 at 1:02 PM Jan Kratochvil wrote: > > On Sun, 11 Aug 2019 20:54:28 +0200, Chris Murphy wrote: > > and likely experiences data loss and possibly even file system > > corruption as a direct consequence of having to force power off on the > > machine because for all practical

Re: Better interactivity in low-memory situations

On Sun, 11 Aug 2019 20:54:28 +0200, Chris Murphy wrote: > and likely experiences data loss and possibly even file system > corruption as a direct consequence of having to force power off on the > machine because for all practical purposes normal control has been > lost. Not really, this is what

Re: Better interactivity in low-memory situations

On Sun, Aug 11, 2019 at 10:36 AM wrote: > > On Sun, Aug 11, 2019 at 10:50 AM, Chris Murphy > wrote: > > Let's take another argument. If the user manually specifies 'ninja -j > > 64' on this same system, is that sabotage? I'd say it is. And > > therefore why isn't it sabotage that the ninja

Re: Better interactivity in low-memory situations

On Sun, Aug 11, 2019 at 11:21 AM Jan Kratochvil wrote: > > On Sun, 11 Aug 2019 17:50:17 +0200, Chris Murphy wrote: > > I don't follow. You're saying RelWithDebInfo is never suitable for a > > local build? > > Most of the time. What is your use case for it? My use case is testing the

Re: Better interactivity in low-memory situations

On Sun, 11 Aug 2019 17:50:17 +0200, Chris Murphy wrote: > I don't follow. You're saying RelWithDebInfo is never suitable for a > local build? Most of the time. What is your use case for it? > isn't relevant to getting a successful build. With powerful enough machine everything is possible.

Re: Better interactivity in low-memory situations

On Sun, Aug 11, 2019 at 10:50 AM, Chris Murphy wrote: Let's take another argument. If the user manually specifies 'ninja -j 64' on this same system, is that sabotage? I'd say it is. And therefore why isn't it sabotage that the ninja default computes N jobs as nrcpus + 2? And also doesn't take

Re: Better interactivity in low-memory situations

On Sat, Aug 10, 2019 at 3:07 AM Jan Kratochvil wrote: > > On Fri, 09 Aug 2019 23:50:43 +0200, Chris Murphy wrote: > > $ cmake -DPORT=GTK -DCMAKE_BUILD_TYPE=RelWithDebInfo -GNinja > > RelWithDebInfo is -O2 -g build. That is not suitable for debugging, for > debugging you should use

Re: Better interactivity in low-memory situations

On Fri, Aug 09, 2019 at 03:50:43PM -0600, Chris Murphy wrote: [..] > Problem and thesis statement: > Certain workloads, such as building webkitGTK from source, results in > heavy swap usage eventually leading to the system becoming totally > unresponsive. Look into switching from disk based swap,

Re: Better interactivity in low-memory situations

On Fri, 09 Aug 2019 23:50:43 +0200, Chris Murphy wrote: > $ cmake -DPORT=GTK -DCMAKE_BUILD_TYPE=RelWithDebInfo -GNinja RelWithDebInfo is -O2 -g build. That is not suitable for debugging, for debugging you should use -DCMAKE_BUILD_TYPE=Debug (that is -g). RelWithDebInfo is useful for final rpm

Re: Better interactivity in low-memory situations

Just in case anyone wants to try to reproduce this particular example: 1. Grab latest stable from here and untar it https://webkitgtk.org/releases/ 2. Run this included script, which is dnf aware, to install dependencies ./Tools/gtk/install-dependencies 3. Additional packages I had to install to

Re: Better interactivity in low-memory situations

Hi, Chris Murphy writes: > Certain workloads, such as building webkitGTK from source, results in > heavy swap usage eventually leading to the system becoming totally > unresponsive. Look into switching from disk based swap, to swap on a > ZRAM device. It sounds like the same issue that has

Better interactivity in low-memory situations

This subject matches a Fedora Workstation Working Group issue of the same name [1], and this post is intended to be an independent summary of the findings so far, and call for additional testing and discussion, in particular subject matter experts. Problem and thesis statement: Certain workloads,