Re: [gentoo-user] Re: webrsync {SOLVED}
Rich Freeman wrote: > On Sat, Feb 29, 2020 at 9:49 AM Dale wrote: >> I have noticed the OOM killing the wrong thing as well. In a way, how >> does it know what it should kill really??? After all, the process using >> the most memory may not be the problem but another one, or more, could. >> I guess in most cases the one using the most is the bad one but it may >> not always be the case. I'm not sure how OOM could determine that tho. >> Maybe having some setting like you mentions would help. It's a thought. > Oh, plenty of people have given thought to it. > > The algorithm is actually not as well-documented as I thought it was. > Lots of documents, but they're fragmented. Behavior is also > configurable. For example, you can just tell Linux to panic on OOM, > or just have it kill the process that triggered OOM even if it isn't > using much memory. > > Best docs I could find are at (among other places): > https://github.com/torvalds/linux/blob/master/Documentation/filesystems/proc.txt#L1520 > > Aside from setting limits on services so that they die/restart before > overall system memory is threatened, adjusting oom_score_adj lets you > tweak overall priorities for any process. > > By default it mostly comes down to what process is hogging the most > RAM, with slight preference for root-owned processes. > > Really though setting resource limits is your best bet. Then you can > set a threshold above normal use, and if something misbehaves it is > going to get restarted before most of RAM is taken. User session > cgroups can of course be limited as well so that interactive processes > can't just go nuts and use all your RAM. > Let's use my recent issue for example. Firefox going into memory eating like a starving sumo wrestler. Is there a way to limit Firefox itself? Set it to like 5GBs and when it hits that, it does a graceful exit, then a kill if that doesn't work? The reason I ask, I don't recall having a service ever eat memory but I'm sure some do. Firefox has been my biggest problem. It still on occasion gets up around 3 or 4GBs. I recompiled with different USE flags and that seems to help but still, rather than crash my system or just hope that OOM will get it right, I'd rather have it pick on what I know to be the biggest offender in my case. That same info could be used by someone else just with a different process being set to limit memory use. In other words, examples of the settings if possible??? Now to go read that link more thoroughly. ;-) Dale :-) :-)
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sat, Feb 29, 2020 at 9:49 AM Dale wrote: > > I have noticed the OOM killing the wrong thing as well. In a way, how > does it know what it should kill really??? After all, the process using > the most memory may not be the problem but another one, or more, could. > I guess in most cases the one using the most is the bad one but it may > not always be the case. I'm not sure how OOM could determine that tho. > Maybe having some setting like you mentions would help. It's a thought. Oh, plenty of people have given thought to it. The algorithm is actually not as well-documented as I thought it was. Lots of documents, but they're fragmented. Behavior is also configurable. For example, you can just tell Linux to panic on OOM, or just have it kill the process that triggered OOM even if it isn't using much memory. Best docs I could find are at (among other places): https://github.com/torvalds/linux/blob/master/Documentation/filesystems/proc.txt#L1520 Aside from setting limits on services so that they die/restart before overall system memory is threatened, adjusting oom_score_adj lets you tweak overall priorities for any process. By default it mostly comes down to what process is hogging the most RAM, with slight preference for root-owned processes. Really though setting resource limits is your best bet. Then you can set a threshold above normal use, and if something misbehaves it is going to get restarted before most of RAM is taken. User session cgroups can of course be limited as well so that interactive processes can't just go nuts and use all your RAM. -- Rich
Re: [gentoo-user] Re: webrsync {SOLVED}
Rich Freeman wrote: > On Sat, Feb 29, 2020 at 9:13 AM Dale wrote: >> Runaway processes is one reason I expanded my memory to 32GBs. It gives >> me more wiggle room for portage to be on tmpfs. >> > That is my other issue. 99% of the time the OOM killer is preferred > when this happens versus having the system just grind to a complete > halt. Either way some service is going to stop working, but at least > with the OOM killer it probably will only be one service. > > OOM doesn't always kill the right thing, but it happens so > infrequently I haven't bothered to address this. > > Setting limits on VM use on each service would of course be a good > idea. Also, you can tune OOM priority as well for any process. With > systemd these are unit config settings. I haven't looked at openrc > recently but certainly you can just edit the init.d script to set > these if there isn't just a config option. > > I've found OOM guessing wrong is more of an issue when you have a lot > of medium-sized processes vs one large one. If one process is using > 10GB of RAM and goes haywire it is very likely that this is the one > OOM-killer will go after. On the other hand if you're building with > make -j16 and you hit some really intensive part of a build and you > get 16 processes demanding half a GB each then it is more likely that > the OOM killer will first target some service that is RAM-hungry but > not usually a problem, because it is using more than any one of the > gcc processes. > > I wonder if you can make OOM-killer cgroup-aware. Services are > generally in separate cgroups while make would all be in another, so > if it looked at total use of the cgroup and not the individual process > then it would weigh something that forks heavily a lot higher. > I have noticed the OOM killing the wrong thing as well. In a way, how does it know what it should kill really??? After all, the process using the most memory may not be the problem but another one, or more, could. I guess in most cases the one using the most is the bad one but it may not always be the case. I'm not sure how OOM could determine that tho. Maybe having some setting like you mentions would help. It's a thought. A while back I had issues with Firefox getting hoggy on memory. One or two websites would just make it go nuts. I block all that crypto mining crap tho. Still, it would normally take a couple GBs when a lot of tabs are open but when those sites got a hold of it, it could swell to 5, 6, 8GBs or more in fairly short order. Thing is, at times OOM would kill the whole GUI not just Firefox. I'd be back at a login screen. Of course, when it does that, it not only kills firefox, it kills firefox running other profiles, Seamonkey and any other programs running within the GUI. As you say, it doesn't always do the "right thing" when it starts killing processes. I don't recall ever actually running out of memory when emerging. I know what the big packages are and set tmpfs to be on spinning rust for those. Of course, after the upgrade, I think only LOo is on that. I'm not sure why, but when LOo updates, so does either Firefox or Seamonkey and sometimes both. Given their size, they end up trying to compile together. That can take up some space on tmpfs if in memory instead of spinning rust. Then there is that HUGE qt package. O_O While I see swap as a necessary evil, I only want it used on very rare occasions. The OOM tool just doesn't get it right. Me killing the offender in htop does tho. BTW, Firefox stopped doing the memory hog thing so bad a few upgrades ago. I never did figure out exactly what it was that triggered that. :/ Dale :-) :-)
Re: [gentoo-user] Re: webrsync {SOLVED}
On Saturday, 29 February 2020 14:30:26 GMT Robert Bridge wrote: > > On 29 Feb 2020, at 13:57, Rich Freeman wrote: > > > > Maybe something has changed in the last few years and swap is actually > > useful, but I'm skeptical. I always tend to end up with GB of free > > RAM and a churning hard drive when I enable it. On SSD I'm sure it > > will perform better, but then I'm running through erase cycles > > instead. > > It may be a stupid question, but are you setting vm.swappiness to a > reasonable value? > > The default setting of 60 does tend to lead to swapping far earlier than is > reasonable for modern desktops with 8+GB RAM. > > Cheers, > Robert. For monster package compiles where each thread chews up more than 3G at a time, the race to/from swap is endless. Swappiness will only go so far. As Richard noted the only sensible solution is to reduce the number of jobs on the compiler. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: webrsync {SOLVED}
> On 29 Feb 2020, at 13:57, Rich Freeman wrote: > > Maybe something has changed in the last few years and swap is actually > useful, but I'm skeptical. I always tend to end up with GB of free > RAM and a churning hard drive when I enable it. On SSD I'm sure it > will perform better, but then I'm running through erase cycles > instead. It may be a stupid question, but are you setting vm.swappiness to a reasonable value? The default setting of 60 does tend to lead to swapping far earlier than is reasonable for modern desktops with 8+GB RAM. Cheers, Robert.
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sat, Feb 29, 2020 at 9:13 AM Dale wrote: > > Runaway processes is one reason I expanded my memory to 32GBs. It gives > me more wiggle room for portage to be on tmpfs. > That is my other issue. 99% of the time the OOM killer is preferred when this happens versus having the system just grind to a complete halt. Either way some service is going to stop working, but at least with the OOM killer it probably will only be one service. OOM doesn't always kill the right thing, but it happens so infrequently I haven't bothered to address this. Setting limits on VM use on each service would of course be a good idea. Also, you can tune OOM priority as well for any process. With systemd these are unit config settings. I haven't looked at openrc recently but certainly you can just edit the init.d script to set these if there isn't just a config option. I've found OOM guessing wrong is more of an issue when you have a lot of medium-sized processes vs one large one. If one process is using 10GB of RAM and goes haywire it is very likely that this is the one OOM-killer will go after. On the other hand if you're building with make -j16 and you hit some really intensive part of a build and you get 16 processes demanding half a GB each then it is more likely that the OOM killer will first target some service that is RAM-hungry but not usually a problem, because it is using more than any one of the gcc processes. I wonder if you can make OOM-killer cgroup-aware. Services are generally in separate cgroups while make would all be in another, so if it looked at total use of the cgroup and not the individual process then it would weigh something that forks heavily a lot higher. -- Rich
Re: [gentoo-user] Re: webrsync {SOLVED}
Rich Freeman wrote: > On Sat, Feb 29, 2020 at 4:33 AM Wols Lists wrote: >> I just have a massive swap space, and /var/tmp/portage is a tmpfs. So >> everything gets a fast tmpfs build, and it spills into swap as required >> (hopefully almost never). >> > I can articulate a bunch of reasons that on paper say that this is the > best approach. > > In practice I've found that swap on linux is sub-optimal at best. I > only enable swap when I absolutely have to as a result. I'll reduce > -j to lower memory demand before adding swap usually. On more > RAM-constrained hosts I'll enable swap when building specific > packages, or try to avoid those packages entirely. > > Maybe something has changed in the last few years and swap is actually > useful, but I'm skeptical. I always tend to end up with GB of free > RAM and a churning hard drive when I enable it. On SSD I'm sure it > will perform better, but then I'm running through erase cycles > instead. > > Like I said, on paper adding swap should only make things better. > But, that is only true if the kernel makes the correct choices about > prioritizing swap vs cache use. Sure, I could set swappiness to zero > or whatever, but then that just turns swap into a NOOP best case and > it isn't like I have OOM issues, so why bother?. > I have a large swap space on my rig. Like you tho, I hate when it is used because it renders my system almost unresponsive. Even switching from one desktop to another can take a minute or even longer. I think swappiness is set to 20 or some low number. I only use it because while slow, I can usually kill the offending process before things start crashing. I just start htop, highlight the offending process and kill it. Usually it is a web browser or something that doesn't get hurt to much from being killed. While swap can be a necessary evil, I hate when even 1MB is used. Runaway processes is one reason I expanded my memory to 32GBs. It gives me more wiggle room for portage to be on tmpfs. You are not alone in your experience with this. Dale :-) :-)
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sat, Feb 29, 2020 at 4:33 AM Wols Lists wrote: > > I just have a massive swap space, and /var/tmp/portage is a tmpfs. So > everything gets a fast tmpfs build, and it spills into swap as required > (hopefully almost never). > I can articulate a bunch of reasons that on paper say that this is the best approach. In practice I've found that swap on linux is sub-optimal at best. I only enable swap when I absolutely have to as a result. I'll reduce -j to lower memory demand before adding swap usually. On more RAM-constrained hosts I'll enable swap when building specific packages, or try to avoid those packages entirely. Maybe something has changed in the last few years and swap is actually useful, but I'm skeptical. I always tend to end up with GB of free RAM and a churning hard drive when I enable it. On SSD I'm sure it will perform better, but then I'm running through erase cycles instead. Like I said, on paper adding swap should only make things better. But, that is only true if the kernel makes the correct choices about prioritizing swap vs cache use. Sure, I could set swappiness to zero or whatever, but then that just turns swap into a NOOP best case and it isn't like I have OOM issues, so why bother?. -- Rich
Re: [gentoo-user] Re: webrsync {SOLVED}
On 24/02/20 08:30, Neil Bothwick wrote: > On Sun, 23 Feb 2020 18:59:27 -0800, Ian Zimmerman wrote: > >> In a desperate act to satisfy the ever increasing build space >> requirements for firefox and its kin, I'd symlinked /var/tmp/portage to >> a subdirectory of /usr/portage. And webrsync does "rsync ... --delete >> ...", so now you see where this is going. >> >> Fortunately, as you say, the recovery was easy once I knew what went >> wrong. >> >>> You might consider moving the sync location out of /usr while you're >>> at it. Also, if you haven't already done so get >>> /usr/portage/distfiles outside of the repo directory. For new >>> installs both of these are moved to /var in non-nested directories, >>> but existing installs will continue to use their present locations. >> >> But I already have a bunch of symlinks to juggle the space requirements >> and I would rather not revisit that, right now. On my system /var is in >> the / filesystem and doesn't have that much space (less than the 8G >> necessary to build firefox). At some future spring cleaning time, >> maybe. > > There's no need for any symlinks, just set $PORTAGE_TMPDIR globally then > separate for greedy packages. For example, in make.conf I have Ahhh. I didn't know about that trick. > > PORTAGE_TMPDIR="/tmp" > > Then /etc/portage/package.env/firefox contains > > www-client/firefox disk-tmpdir.conf > > and /etc/portage/env/disk-tmpdir.conf contains > > PORTAGE_TMPDIR="/mnt/scratch" > > So I get faster tmpfs based builds for most packages and use a disposable > disk partition for the greedy ones. > I just have a massive swap space, and /var/tmp/portage is a tmpfs. So everything gets a fast tmpfs build, and it spills into swap as required (hopefully almost never). Cheers, Wol
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sun, 23 Feb 2020 18:59:27 -0800, Ian Zimmerman wrote: > In a desperate act to satisfy the ever increasing build space > requirements for firefox and its kin, I'd symlinked /var/tmp/portage to > a subdirectory of /usr/portage. And webrsync does "rsync ... --delete > ...", so now you see where this is going. > > Fortunately, as you say, the recovery was easy once I knew what went > wrong. > > > You might consider moving the sync location out of /usr while you're > > at it. Also, if you haven't already done so get > > /usr/portage/distfiles outside of the repo directory. For new > > installs both of these are moved to /var in non-nested directories, > > but existing installs will continue to use their present locations. > > But I already have a bunch of symlinks to juggle the space requirements > and I would rather not revisit that, right now. On my system /var is in > the / filesystem and doesn't have that much space (less than the 8G > necessary to build firefox). At some future spring cleaning time, > maybe. There's no need for any symlinks, just set $PORTAGE_TMPDIR globally then separate for greedy packages. For example, in make.conf I have PORTAGE_TMPDIR="/tmp" Then /etc/portage/package.env/firefox contains www-client/firefox disk-tmpdir.conf and /etc/portage/env/disk-tmpdir.conf contains PORTAGE_TMPDIR="/mnt/scratch" So I get faster tmpfs based builds for most packages and use a disposable disk partition for the greedy ones. -- Neil Bothwick And if you say "No", I shall be forced to shoot you. pgpDD4kho2uhW.pgp Description: OpenPGP digital signature
