Re: [gentoo-user] Memory manager
On Sun, 20 Oct 2019 18:01:01 +0100, Mick wrote: > Now, in a gentoo scenario, say a mammoth compile like Chromium, with a > large count of jobs specified for it, you could end up swapping part or > all of one or more jobs into memory, only to swap it out again in order > to process it. The compile keeps swapping in and out a job at a time in > order to carry on compiling. The disk thrashing is now continuous and > indeed interacting with your desktop will be painful - potentially > waiting for minutes at a time before an application responds. The way > out of this bottleneck is to either increase your RAM, or minimise the > use of memory by reducing the job count in MAKEOPTS. Shutting down > desktop applications and login out of any desktop sessions to release > RAM will also help. > > On a laptop with 4G RAM compiling Chromium is quite challenging when > even a single gcc job could grow to 3G or more. Swapping and a disk > I/O bottleneck becomes unavoidable and moving the compile of binaries > to a bigger PC becomes a rather wise solution. That's why I have Chromium, as well and LO and qtwebengine, set to use my SSD for PORTAGE_TMPDIR on this laptop, which is limited to 8GB. MAKEOPTS is also constricted for Chromium. As a result, the packages build more quickly with minimal impact on using the system. -- Neil Bothwick Top Oxymorons Number 3: Working vacation pgpSZT2Se7d0N.pgp Description: OpenPGP digital signature
Re: [gentoo-user] Memory manager
Mick wrote: > On Sunday, 20 October 2019 16:03:42 BST Dale wrote: >> Here's the >> thing about using swap on my rig, once it does, the system gets >> extremely slow. Even switching desktops can take a minute or longer. >> Other than trying to get to what is eating up memory and killing it, the >> system is virtually useless. Even my video stops playing. > Swapping can bring the system to its knees, but only under certain operating > scenarios. This is how I understand it works: > > Say you're browsing and keep opening tabs. The browser application will > preemptively allocate memory for more tabs, in case you carry on opening even > more tabs. Then you open yet another big application in terms of memory > usage > and start running it. The kernel will reallocate some of the browser memory > not currently utilised to the other application and keep things working > smoothly. With more applications/tabs being opened you will eventually run > out of RAM and the kernel will swap some of the memory pages to disk. The > swapping is meant to be selective, i.e. things you haven't used in a while > will be taken out of RAM and saved onto your disk. > > Under the above scenario you may notice a momentary latency on your desktop > as > data is swapped onto the disk, but afterwards the desktop should be > responsive > once more - unless more swapping is again demanded by your actions. If you > try to access an application which has had parts of its memory allocation > swapped out to disk you will notice a delay in its reactions. > > Now, in a gentoo scenario, say a mammoth compile like Chromium, with a large > count of jobs specified for it, you could end up swapping part or all of one > or more jobs into memory, only to swap it out again in order to process it. > The compile keeps swapping in and out a job at a time in order to carry on > compiling. The disk thrashing is now continuous and indeed interacting with > your desktop will be painful - potentially waiting for minutes at a time > before an application responds. The way out of this bottleneck is to either > increase your RAM, or minimise the use of memory by reducing the job count in > MAKEOPTS. Shutting down desktop applications and login out of any desktop > sessions to release RAM will also help. > > On a laptop with 4G RAM compiling Chromium is quite challenging when even a > single gcc job could grow to 3G or more. Swapping and a disk I/O bottleneck > becomes unavoidable and moving the compile of binaries to a bigger PC becomes > a rather wise solution. > > Another occasion when swapping can cause havoc is when you have a memory leak > due to some buggy application and all your RAM followed by swap is chewed up > until an OOM ensues. > > For these reasons I always set up swap on my gentoo systems. What you describe in your first scenario, that is when it is so slow and virtually unresponsive. I don't generally run out of memory when compiling since I close other stuff, like Firefox, to free up memory, even in the 16GB days. What I run into is a tab, or tabs, in Firefox that are chewing through memory like a hungry junk yard dog on a crook. When Firefox does that, it is slow, extremely slow. When I get to where I can see what is going on, it is still slow to even close or kill that process. If I get to a Konsole, I usually run htop and kill it from there. Thing is, it may take a minute even for htop to start and show the problem. That's a fairly small program but it can take a minute or two to even load and show its screen. I have two sites in particular that does this. They run for days with no problem and I may not even be using that tab or that site. Then it's like it gets mad and starts using more and more memory. I have caught it in time to just refresh the tab and it go back to normal. Once it starts using swap tho, it's very slow. I wish Firefox had a way to fix that out of control memory usage. It may not be Firefox itself doing it but it would seem it can see something isn't right and put a stop to it. It's sad when a Linux desktop has 32GBs of memory eat up like that, usually by one program at that. Still, no swap at all would result in a crash or reset. It's better than nothing but I wish the root cause could be fixed. Dale :-) :-)
Re: [gentoo-user] Memory manager
On Sunday, 20 October 2019 16:03:42 BST Dale wrote: > Here's the > thing about using swap on my rig, once it does, the system gets > extremely slow. Even switching desktops can take a minute or longer. > Other than trying to get to what is eating up memory and killing it, the > system is virtually useless. Even my video stops playing. Swapping can bring the system to its knees, but only under certain operating scenarios. This is how I understand it works: Say you're browsing and keep opening tabs. The browser application will preemptively allocate memory for more tabs, in case you carry on opening even more tabs. Then you open yet another big application in terms of memory usage and start running it. The kernel will reallocate some of the browser memory not currently utilised to the other application and keep things working smoothly. With more applications/tabs being opened you will eventually run out of RAM and the kernel will swap some of the memory pages to disk. The swapping is meant to be selective, i.e. things you haven't used in a while will be taken out of RAM and saved onto your disk. Under the above scenario you may notice a momentary latency on your desktop as data is swapped onto the disk, but afterwards the desktop should be responsive once more - unless more swapping is again demanded by your actions. If you try to access an application which has had parts of its memory allocation swapped out to disk you will notice a delay in its reactions. Now, in a gentoo scenario, say a mammoth compile like Chromium, with a large count of jobs specified for it, you could end up swapping part or all of one or more jobs into memory, only to swap it out again in order to process it. The compile keeps swapping in and out a job at a time in order to carry on compiling. The disk thrashing is now continuous and indeed interacting with your desktop will be painful - potentially waiting for minutes at a time before an application responds. The way out of this bottleneck is to either increase your RAM, or minimise the use of memory by reducing the job count in MAKEOPTS. Shutting down desktop applications and login out of any desktop sessions to release RAM will also help. On a laptop with 4G RAM compiling Chromium is quite challenging when even a single gcc job could grow to 3G or more. Swapping and a disk I/O bottleneck becomes unavoidable and moving the compile of binaries to a bigger PC becomes a rather wise solution. Another occasion when swapping can cause havoc is when you have a memory leak due to some buggy application and all your RAM followed by swap is chewed up until an OOM ensues. For these reasons I always set up swap on my gentoo systems. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Memory manager
Mick wrote: > On Sunday, 20 October 2019 12:59:03 BST Wols Lists wrote: > >> Well, I do all my emerges on tmpfs, so if things like LO, firefox et al >> need maybe 10GB, I need at least that available ... (that said, 16GB ram >> could probably do it without needing swap :-) > Anecdotal evidence suggests 16G RAM may not be enough: > > A number of times I was emerging chromium with USE="jumbo-build" enabled on a > PC with 16G RAM, while a user was running Kmail, firefox and some other apps. > > FF in particular is a hog, as it pre-emptively allocates more RAM as you keep > opening more tabs. With 15-20 FF tabs open and MAKEOPTS="-j5", all RAM was > exhausted and swap started being used heavily by portage, thrashing the disk > as jobs were being swapped I/O. With no swap the emerge would have been > killed with an OOM. > > >> But seeing as I try to fill up my mobo ram, my disks are mirrored, and I >> still try and stick to the "twice ram" rule, this setup means any >> upgrades/changes to the computer means I don't break that rule. If it's >> overkill, well disk is cheap (and I can always nick a swap partition and >> repurpose it temporarily if needs be :-) >> >> Cheers, >> Wol > Quite, disks are cheap and having more swap won't break the bank. On PCs > with > limited RAM I often end up adding more swap space and reducing job counts to > get big emerges through. > This is some things I've noticed. I have swap but set it to not be used unless something bad happens. I think swappiness is set to like 10 or 20. It's rare that any swap is ever used. When it does, it is usually some tab in Firefox that has went wild eating up memory. Here's the thing about using swap on my rig, once it does, the system gets extremely slow. Even switching desktops can take a minute or longer. Other than trying to get to what is eating up memory and killing it, the system is virtually useless. Even my video stops playing. For me and my experience, I only want swap available to prevent the system from crashing and resetting itself. It gives me a little time to kill the memory hog and not have a crash. Other than that, I don't want swap used because it is just to slow. I have 32GBs so it's rare that I run out anymore. When I had 16GBs tho, Firefox was a regular abuser. Seamonkey never runs wild exactly but sometimes I close and restart it to force it to release memory. What I wish, Firefox and even Seamonkey would stop running wild with memory. Sometimes it gets really hungry and takes up several GBs when there is no reason to do so. Like with Mick tho, 16GBs got to where it wasn't enough. I had to close web browsers during large compiles. LOo, Firefox, Seamonkey etc etc. I have emerge's work directory on tmpfs here. That's my experience with swap, real world. Dale :-) :-)
Re: [gentoo-user] Memory manager
On Sunday, 20 October 2019 12:59:03 BST Wols Lists wrote: > Well, I do all my emerges on tmpfs, so if things like LO, firefox et al > need maybe 10GB, I need at least that available ... (that said, 16GB ram > could probably do it without needing swap :-) Anecdotal evidence suggests 16G RAM may not be enough: A number of times I was emerging chromium with USE="jumbo-build" enabled on a PC with 16G RAM, while a user was running Kmail, firefox and some other apps. FF in particular is a hog, as it pre-emptively allocates more RAM as you keep opening more tabs. With 15-20 FF tabs open and MAKEOPTS="-j5", all RAM was exhausted and swap started being used heavily by portage, thrashing the disk as jobs were being swapped I/O. With no swap the emerge would have been killed with an OOM. > But seeing as I try to fill up my mobo ram, my disks are mirrored, and I > still try and stick to the "twice ram" rule, this setup means any > upgrades/changes to the computer means I don't break that rule. If it's > overkill, well disk is cheap (and I can always nick a swap partition and > repurpose it temporarily if needs be :-) > > Cheers, > Wol Quite, disks are cheap and having more swap won't break the bank. On PCs with limited RAM I often end up adding more swap space and reducing job counts to get big emerges through. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Memory manager
On 20/10/19 10:59, Mick wrote: > On Sunday, 20 October 2019 00:35:56 BST Wol's lists wrote: > >> The original swap algorithm NEEDED twice ram as swap. And when Linus >> ripped out all the "optimisation", the vanilla kernels only needed to >> touch swap, and if they didn't have twice ram they would crash. > > Was this also the time when the default swappiness was set at 60? > > >> At that point, the recommendation changed to "no swap is fine, twice or >> more is fine, just don't have swap less than twice ram". > > Are you sure of this? At least on current kernels (I'm currently on 4.19.72- > gentoo) the overcommit_accounting kernel mechanism using a heuristic over- > commit memory handling is set at 0, which refuses wilder over commits, but > allows more measured over commits to use swap space. > > > From the above and without further experimentation I assume having a swap > slightly larger than my RAM is more than adequate for a desktop, including > hibernating on swap. > As per Richard Brown, the current recommendation seems to be even less than that ... "2GB is plenty". That's the SUSE default. > >> My personal rule is to take the motherboard's max ram, double it, and >> create a swap partition that size on every disk. So my current desktop >> system has 80GB of ram/swap - 4x4GB slots times 2 disk drives. And my >> new system has 4x8GB so that'll be 160GB!!! HOWEVER - Richard Brown of >> SUSE said that's dangerous - if somebody fork-bombs you it'll take a >> long time to fill that much swap and regaining control of your system >> could well be a big red switch job. >> > > Each to their own, but I tend to think this huge amount of swap is probably > excessive, unless you're running some scientific applications which require > big over commits for their calculations. > Well, I do all my emerges on tmpfs, so if things like LO, firefox et al need maybe 10GB, I need at least that available ... (that said, 16GB ram could probably do it without needing swap :-) But seeing as I try to fill up my mobo ram, my disks are mirrored, and I still try and stick to the "twice ram" rule, this setup means any upgrades/changes to the computer means I don't break that rule. If it's overkill, well disk is cheap (and I can always nick a swap partition and repurpose it temporarily if needs be :-) Cheers, Wol
Re: [gentoo-user] Memory manager
On Sunday, 20 October 2019 00:35:56 BST Wol's lists wrote: > The original swap algorithm NEEDED twice ram as swap. And when Linus > ripped out all the "optimisation", the vanilla kernels only needed to > touch swap, and if they didn't have twice ram they would crash. Was this also the time when the default swappiness was set at 60? > At that point, the recommendation changed to "no swap is fine, twice or > more is fine, just don't have swap less than twice ram". Are you sure of this? At least on current kernels (I'm currently on 4.19.72- gentoo) the overcommit_accounting kernel mechanism using a heuristic over- commit memory handling is set at 0, which refuses wilder over commits, but allows more measured over commits to use swap space. $ cat /proc/sys/vm/overcommit_memory 0 When set at 2, where no over-commit of memory is allowed, the mechanism will be informed by the overcommit_ratio by default set at 50%, to calculate how much RAM will be used by malloc. In this case, all swap will be used but only 50% of RAM. $ cat /proc/sys/vm/overcommit_ratio 50 Unless someone has purposefully set their overcommit_memory to 1, where potentially wild over commits are allowed and the risk of OOM increases, the kernel will not allow over-committing of RAM without checking how much RAM + swap is available. On my system with default overcommit_memory settings I have: $ free --kilo total usedfree shared buff/cache available Mem: 15975043 13598841384940722528 76575114126170 Swap: 17179865 017179865 but only 24577524 is reported as the commit limit: $ grep CommitLimit /proc/meminfo CommitLimit:24577524 kB This is all swap and a percentage of RAM: 24577524 - 17179865 = 7397659 of RAM, or 46.3%. >From the above and without further experimentation I assume having a swap slightly larger than my RAM is more than adequate for a desktop, including hibernating on swap. > My personal rule is to take the motherboard's max ram, double it, and > create a swap partition that size on every disk. So my current desktop > system has 80GB of ram/swap - 4x4GB slots times 2 disk drives. And my > new system has 4x8GB so that'll be 160GB!!! HOWEVER - Richard Brown of > SUSE said that's dangerous - if somebody fork-bombs you it'll take a > long time to fill that much swap and regaining control of your system > could well be a big red switch job. > > Cheers, > Wol Each to their own, but I tend to think this huge amount of swap is probably excessive, unless you're running some scientific applications which require big over commits for their calculations. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Memory manager
On 19/10/2019 16:24, Mick wrote: On Saturday, 19 October 2019 14:11:26 bstmad.scientist.at.la...@tutanota.com wrote: Do systems run different memory management when swap is on versus no swap? The answer to this question is an unqualified yes, although you do not define your meaning of "different memory management". The existence of swap space and the kernel's swappiness setting will change the way memory is dynamically allocated to processes at runtime and may affect the responsiveness of your system. Memory management was massively rewritten roundabout kernel 2.4. The original swap algorithm NEEDED twice ram as swap. And when Linus ripped out all the "optimisation", the vanilla kernels only needed to touch swap, and if they didn't have twice ram they would crash. At that point, the recommendation changed to "no swap is fine, twice or more is fine, just don't have swap less than twice ram". My personal rule is to take the motherboard's max ram, double it, and create a swap partition that size on every disk. So my current desktop system has 80GB of ram/swap - 4x4GB slots times 2 disk drives. And my new system has 4x8GB so that'll be 160GB!!! HOWEVER - Richard Brown of SUSE said that's dangerous - if somebody fork-bombs you it'll take a long time to fill that much swap and regaining control of your system could well be a big red switch job. Cheers, Wol
Re: [gentoo-user] Memory manager
On Saturday, 19 October 2019 14:11:26 BST mad.scientist.at.la...@tutanota.com wrote: > Do systems run different memory management when swap is on versus no swap? The answer to this question is an unqualified yes, although you do not define your meaning of "different memory management". The existence of swap space and the kernel's swappiness setting will change the way memory is dynamically allocated to processes at runtime and may affect the responsiveness of your system. However, you may not experience any difference by having swap on, if your swappinness setting is low and in addition you never run a high enough number of memory hungry processes to consume all available RAM. > I know that in the past it was recommended to have at least a small swap so > the system used a better memory manager. Wondering specifically if I > should set up a small ram drive for swap just to get better memory > management or if I should run swapless as this particular machine has a > slightly obscene amount of ram available so I shouldn't need a proper swap > partition unless it affects memory management. > > -- “The whole world is watching! The whole world is watching!” The handbook recommendation to set up a swap space (partition) is probably made as a low cost, safe configuration to have, which may even save your bacon one day. It is conceivable you could set portage to run a large enough number of jobs on a memory hungry compile, e.g. chromium, to end up with no memory left. In addition, if you tried to Suspend-to-RAM, at a point when all of your RAM is being used, you would soon discover you can't do it without any swap space made available. If you hibernate, then a swap space (file/ partition) will be used, which makes it reasonable to have a swap space set up for this purpose in advance. For a desktop, the relative low cost of disk space today suggests it is a good idea to set up some swap space, even if you hardly ever going to use it. With the arrival of SSDs the usage of swap was discouraged, because repeated read/writes tended to age prematurely the (early) SSDs. I still use spinning disks for swap, but I don't know what the prevailing wisdom/experience suggest these days. Have a look at this for generalities: https://wiki.gentoo.org/wiki/Knowledge_Base:Is_swap_space_really_necessary And this for an alternative, or complimentary solution: https://wiki.gentoo.org/wiki/Zram -- Regards, Mick signature.asc Description: This is a digitally signed message part.
[gentoo-user] Memory manager
Do systems run different memory management when swap is on versus no swap? I know that in the past it was recommended to have at least a small swap so the system used a better memory manager. Wondering specifically if I should set up a small ram drive for swap just to get better memory management or if I should run swapless as this particular machine has a slightly obscene amount of ram available so I shouldn't need a proper swap partition unless it affects memory management. -- “The whole world is watching! The whole world is watching!”
