Re: EarlyOOM +ZRAM Only
On 8/17/20 12:00 PM, Chris Murphy wrote: I don't know what smem's Swap column is showing. It doesn't match /proc/pid/status -> VmSwap sudo smem -t --sort swap PID User Command Swap USS PSS RSS ... 3 chris/usr/bin/gnome-shell4912 119760 151274 238112 $ grep VmSwap /proc/3/status VmSwap: 0 kB From smem source, it's totaling up the "Swap:" lines from /proc/pid/smaps https://selenic.com/repo/smem/file/tip/smem#l35 -- Andrew Schultz ajschult...@gmail.com ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El lun., 17 ago. 2020 a las 13:02, Chris Murphy () escribió: > On Mon, Aug 17, 2020 at 8:09 AM Sergio Belkin wrote: > > > > > > > > El sáb., 15 ago. 2020 a las 21:46, Samuel Sieb () > escribió: > >> > >> On 8/15/20 1:32 PM, Sergio Belkin wrote: > >> > Nice, my only doubt is why smem and tools alike cannot show those > >> > processes using anon pages in swap... > >> > >> I posted a bash command line in an earlier email that will give you that > >> information. > >> ___ > >> > > > > Samue, Are you talking about this: > > > > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NNGPG7IXSNG4NU7O3X7PVABWROAS32CN/ > > -- > > Yes. > > I don't know what smem's Swap column is showing. It doesn't match > /proc/pid/status -> VmSwap > > sudo smem -t --sort swap > > PID User Command Swap USS PSS > RSS > ... > 3 chris/usr/bin/gnome-shell4912 119760 151274 > 238112 > > $ grep VmSwap /proc/3/status > VmSwap: 0 kB > > > More examples in different languages: https://github.com/lilydjwg/swapview-rosetta These scripts/programs are more updated that smem. I've tried python and bash examples and it seems to work fine. -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Mon, Aug 17, 2020 at 8:09 AM Sergio Belkin wrote: > > > > El sáb., 15 ago. 2020 a las 21:46, Samuel Sieb () escribió: >> >> On 8/15/20 1:32 PM, Sergio Belkin wrote: >> > Nice, my only doubt is why smem and tools alike cannot show those >> > processes using anon pages in swap... >> >> I posted a bash command line in an earlier email that will give you that >> information. >> ___ >> > > Samue, Are you talking about this: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NNGPG7IXSNG4NU7O3X7PVABWROAS32CN/ > -- Yes. I don't know what smem's Swap column is showing. It doesn't match /proc/pid/status -> VmSwap sudo smem -t --sort swap PID User Command Swap USS PSS RSS ... 3 chris/usr/bin/gnome-shell4912 119760 151274 238112 $ grep VmSwap /proc/3/status VmSwap: 0 kB -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El sáb., 15 ago. 2020 a las 21:46, Samuel Sieb () escribió: > On 8/15/20 1:32 PM, Sergio Belkin wrote: > > Nice, my only doubt is why smem and tools alike cannot show those > > processes using anon pages in swap... > > I posted a bash command line in an earlier email that will give you that > information. > ___ > > Samue, Are you talking about this: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NNGPG7IXSNG4NU7O3X7PVABWROAS32CN/ -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On 8/15/20 1:32 PM, Sergio Belkin wrote: Nice, my only doubt is why smem and tools alike cannot show those processes using anon pages in swap... I posted a bash command line in an earlier email that will give you that information. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El sáb., 15 ago. 2020 a las 17:09, Chris Murphy () escribió: > On Sat, Aug 15, 2020 at 1:59 PM Sergio Belkin wrote: > > > > Right now I have this with sudo smem -c "pid name swap pss" -s swap -k -t > > > > 4105 cleanupd 2.5M 132.0K > > 731103 firewalld2.7M23.7M > > 4528 powerline-daemo 3.3M11.2M > > 5078 colord 3.4M 2.0M > > 4062 (sd-pam 4.0M28.0K > > 4061 (sd-pam 4.1M12.0K > > - > > 206 70.0M 3.4G > > OK so 70M swap > > > > And with free -m: > > > > totalusedfree shared buff/cache available > > Mem: 1588732541662 804 10969 > 11486 > > Swap: 4095 624033 > > 62M swap > > > > > > Huge difference, I don't know, There is something bad with plasma, isn't > it? :) > > > > I don't know if is useful now: > > zramctl > > NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT > > /dev/zram0 lzo-rle 4G 62,7M 21M 53,8M 4 [SWAP] > > > 62M swap > > The log out and log back probably caused most of the anon pages in > swap to get dropped. So you'll need to use the system normally until > swap usage is back up to ~3+G and then run that smem command and see > what's using all of this swap... > > > > -- > Chris Murphy > Nice, my only doubt is why smem and tools alike cannot show those processes using anon pages in swap... -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Sat, Aug 15, 2020 at 1:59 PM Sergio Belkin wrote: > > Right now I have this with sudo smem -c "pid name swap pss" -s swap -k -t > > 4105 cleanupd 2.5M 132.0K > 731103 firewalld2.7M23.7M > 4528 powerline-daemo 3.3M11.2M > 5078 colord 3.4M 2.0M > 4062 (sd-pam 4.0M28.0K > 4061 (sd-pam 4.1M12.0K > - > 206 70.0M 3.4G OK so 70M swap > And with free -m: > > totalusedfree shared buff/cache available > Mem: 1588732541662 804 10969 > 11486 > Swap: 4095 624033 62M swap > > Huge difference, I don't know, There is something bad with plasma, isn't it? > :) > > I don't know if is useful now: > zramctl > NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT > /dev/zram0 lzo-rle 4G 62,7M 21M 53,8M 4 [SWAP] 62M swap The log out and log back probably caused most of the anon pages in swap to get dropped. So you'll need to use the system normally until swap usage is back up to ~3+G and then run that smem command and see what's using all of this swap... -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El sáb., 15 ago. 2020 a las 14:58, Chris Murphy ()
escribió:
> On Sat, Aug 15, 2020 at 11:54 AM Chris Murphy
> wrote:
> >
> > On Sat, Aug 15, 2020 at 11:09 AM Sergio Belkin wrote:
> > >
> > >
> > >
> > > > However swap usage is still high :
> > > > free -m
> > > > totalusedfree shared buff/cache
> available
> > > > Mem: 158878577118745876123
> 2382
> > > > Swap: 40953854 241
> > > >
> > > > It's weird, isn't it?
> > >
> > > >> It's consistent. Only ~240 MB of 900M for PK had been evicted to
> swap.
> > > >> When restarting PK, it dropped those 240MB in swap. Free memory also
> > > >> went up no doubt. I don't know why it's using so much memory, what
> all
> > > >> these anonymous pages are. If you catch it going above 500M, check
> > > >> /proc/pid/status and let's see what the breakdown is of memory
> usage.
> > >
> > > Ok but smem tell me that around 300M is swap used, and the ~3.5G
> remaining???
> > > Is there a way to find the culprit processes?
> > > Thanks in advance!
> >
> > I'm not familiar enough with smem to know what it does or why it's
> > missing things, but
> >
> > for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 "
> > "$3}END{ print ""}' $file; done | sort -k 2 -n -r | less
> >
> > Finds more things using swap than smem. As in, smem isn't showing
> > packagekitd for me at all, and yet /proc/pid/status for pk is showing
> > VmSwap is 16M, which at the moment is 20% of swap.
> >
> > What do you get for zramctl?
>
> Ok for some reason 'sudo' gets me different results with smem. Try this:
>
> sudo smem -t --sort swap
>
>
> --
> Chris Murphy
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
>
Oops, sorry I've read this a bit late... but I tell you what I did and HTH
someone:
I Closed app by app: firefox, two lxc containers, etc and then I logged out
(no reboot), and swap kept on the same level
Then I login again
After logout and re-login with plasma with wayland (I could not use kde
with Xorg I have issues re-logging, I guess that has to do with this
bug:
https://fedoraproject.org/wiki/Common_F32_bugs#User_switching.2Fre-logging_in_KDE_might_often_lead_to_a_black.2Ffrozen_screen
Right now I have this with sudo smem -c "pid name swap pss" -s swap -k -t
4105 cleanupd 2.5M 132.0K
731103 firewalld2.7M23.7M
4528 powerline-daemo 3.3M11.2M
5078 colord 3.4M 2.0M
4062 (sd-pam 4.0M28.0K
4061 (sd-pam 4.1M12.0K
-
206 70.0M 3.4G
And with free -m:
totalusedfree shared buff/cache available
Mem: 1588732541662 804 10969
11486
Swap: 4095 624033
Huge difference, I don't know, There is something bad with plasma, isn't
it? :)
I don't know if is useful now:
zramctl
NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle 4G 62,7M 21M 53,8M 4 [SWAP]
TIA
--
--
Sergio Belkin
LPIC-2 Certified - http://www.lpi.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Sat, Aug 15, 2020 at 11:54 AM Chris Murphy wrote:
>
> On Sat, Aug 15, 2020 at 11:09 AM Sergio Belkin wrote:
> >
> >
> >
> > > However swap usage is still high :
> > > free -m
> > > totalusedfree shared buff/cache
> > > available
> > > Mem: 158878577118745876123
> > > 2382
> > > Swap: 40953854 241
> > >
> > > It's weird, isn't it?
> >
> > >> It's consistent. Only ~240 MB of 900M for PK had been evicted to swap.
> > >> When restarting PK, it dropped those 240MB in swap. Free memory also
> > >> went up no doubt. I don't know why it's using so much memory, what all
> > >> these anonymous pages are. If you catch it going above 500M, check
> > >> /proc/pid/status and let's see what the breakdown is of memory usage.
> >
> > Ok but smem tell me that around 300M is swap used, and the ~3.5G
> > remaining???
> > Is there a way to find the culprit processes?
> > Thanks in advance!
>
> I'm not familiar enough with smem to know what it does or why it's
> missing things, but
>
> for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 "
> "$3}END{ print ""}' $file; done | sort -k 2 -n -r | less
>
> Finds more things using swap than smem. As in, smem isn't showing
> packagekitd for me at all, and yet /proc/pid/status for pk is showing
> VmSwap is 16M, which at the moment is 20% of swap.
>
> What do you get for zramctl?
Ok for some reason 'sudo' gets me different results with smem. Try this:
sudo smem -t --sort swap
--
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Sat, Aug 15, 2020 at 11:09 AM Sergio Belkin wrote:
>
>
>
> > However swap usage is still high :
> > free -m
> > totalusedfree shared buff/cache
> > available
> > Mem: 158878577118745876123
> > 2382
> > Swap: 40953854 241
> >
> > It's weird, isn't it?
>
> >> It's consistent. Only ~240 MB of 900M for PK had been evicted to swap.
> >> When restarting PK, it dropped those 240MB in swap. Free memory also
> >> went up no doubt. I don't know why it's using so much memory, what all
> >> these anonymous pages are. If you catch it going above 500M, check
> >> /proc/pid/status and let's see what the breakdown is of memory usage.
>
> Ok but smem tell me that around 300M is swap used, and the ~3.5G remaining???
> Is there a way to find the culprit processes?
> Thanks in advance!
I'm not familiar enough with smem to know what it does or why it's
missing things, but
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 "
"$3}END{ print ""}' $file; done | sort -k 2 -n -r | less
Finds more things using swap than smem. As in, smem isn't showing
packagekitd for me at all, and yet /proc/pid/status for pk is showing
VmSwap is 16M, which at the moment is 20% of swap.
What do you get for zramctl?
--
Chris Murphy
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
> However swap usage is still high : > free -m > totalusedfree shared buff/cache available > Mem: 158878577118745876123 2382 > Swap: 40953854 241 > > It's weird, isn't it? >> It's consistent. Only ~240 MB of 900M for PK had been evicted to swap. >> When restarting PK, it dropped those 240MB in swap. Free memory also >> went up no doubt. I don't know why it's using so much memory, what all >> these anonymous pages are. If you catch it going above 500M, check >> /proc/pid/status and let's see what the breakdown is of memory usage. Ok but smem tell me that around 300M is swap used, and the ~3.5G remaining??? Is there a way to find the culprit processes? Thanks in advance! -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Fri, Aug 14, 2020 at 3:04 PM Sergio Belkin wrote: > > Good question this the output of he last lines of smem -c "name swap" -s swap > -k -t > > kaccess 4.1M > kded54.5M > cadmus 5.0M > konsole 5.4M > xdg-desktop-por 10.6M > mount.ntfs 26.1M > Xorg30.0M > mysqld 36.6M > plasmashell 36.9M > packagekitd239.6M > -- >547.3M That's ~1/8th the size of the 100% full swap device though. Not enough is accounted for. Are there a ton of small processes created by something? Like many dozens of firefox tabs? Or? > systemctl status packagekit > ● packagekit.service - PackageKit Daemon > Loaded: loaded (/usr/lib/systemd/system/packagekit.service; static; > vendor preset: disabled) > Active: active (running) since Thu 2020-07-30 15:40:51 -03; 2 weeks 1 > days ago >Main PID: 5393 (packagekitd) > Tasks: 12 (limit: 19015) > Memory: 903.2M > Now I've restarted packagekit and it outputs: > > ● packagekit.service - PackageKit Daemon > Loaded: loaded (/usr/lib/systemd/system/packagekit.service; static; > vendor preset: disabled) > Active: active (running) since Fri 2020-08-14 17:58:31 -03; 44s ago >Main PID: 2073306 (packagekitd) > Tasks: 3 (limit: 19015) > Memory: 11.2M > However swap usage is still high : > free -m > totalusedfree shared buff/cache > available > Mem: 158878577118745876123 > 2382 > Swap: 40953854 241 > > It's weird, isn't it? It's consistent. Only ~240 MB of 900M for PK had been evicted to swap. When restarting PK, it dropped those 240MB in swap. Free memory also went up no doubt. I don't know why it's using so much memory, what all these anonymous pages are. If you catch it going above 500M, check /proc/pid/status and let's see what the breakdown is of memory usage. I want to see if we can stop these SIGTERMs from happening. I suggest bumping your zram size cap to 8G. Create this file /etc/systemd/zram-generator.conf Containing just two lines: [zram0] max-zram-size = 8192 Either reboot, or maybe logout or quit a bunch of things first because otherwise with a full swap, this command is risky: systemctl restart swap-create@zram0 For that to work, it must swapoff which means all anonymous pages still in swap must be restored to memory first, before the new sized zram swap device is created and started. This doesn't solve the problem of whatever is creating so many anonymous pages. But I wonder if it levels off at something like 6G or if it just keep spirally out of control, and what's doing it. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El vie., 14 ago. 2020 a las 16:32, Chris Murphy () escribió: > On Fri, Aug 14, 2020 at 12:42 PM Sergio Belkin wrote: > > > > 2 comments: > > - I don't use disk-based swap, only zram. > > - It happened again, and in this case there is no Virtual Machine nor > Zoom app running: > > ago 14 15:08:37 dublin.ireland.home earlyoom[888]: sending SIGTERM to > process 2052260 uid 1000 "Web Content": badness 322, VmRSS 447 MiB > > ago 14 15:08:40 dublin.ireland.home earlyoom[888]: process exited after > 2.7 seconds > > ago 14 15:12:15 dublin.ireland.home earlyoom[888]: mem avail: 395 of > 15887 MiB ( 2.49%), swap free:0 of 4095 MiB ( 0.00%) > > ago 14 15:12:15 dublin.ireland.home earlyoom[888]: low memory! at or > below SIGTERM limits: mem 2.52%, swap 10.00% > > ago 14 15:12:15 dublin.ireland.home earlyoom[888]: sending SIGTERM to > process 2055755 uid 1000 "Web Content": badness 319, VmRSS 392 MiB > > ago 14 15:12:17 dublin.ireland.home earlyoom[888]: process exited after > 2.8 seconds > > ago 14 15:28:35 dublin.ireland.home earlyoom[888]: mem avail: 371 of > 15887 MiB ( 2.34%), swap free:0 of 4095 MiB ( 0.00%) > > ago 14 15:28:35 dublin.ireland.home earlyoom[888]: low memory! at or > below SIGTERM limits: mem 2.52%, swap 10.00% > > ago 14 15:28:35 dublin.ireland.home earlyoom[888]: sending SIGTERM to > process 2062157 uid 1000 "Web Content": badness 327, VmRSS 553 MiB > > ago 14 15:28:37 dublin.ireland.home earlyoom[888]: process exited after > 2.3 seconds > > > What's the workload? What is filling up 4G of swap with inactive > pages such that none of them are being freed? The usual case is > available memory is well below the watermark before swap has filled. > But in this case, it's the opposite. > Good question this the output of he last lines of smem -c "name swap" -s swap -k -t kaccess 4.1M kded54.5M cadmus 5.0M konsole 5.4M xdg-desktop-por 10.6M mount.ntfs 26.1M Xorg30.0M mysqld 36.6M plasmashell 36.9M packagekitd239.6M -- 547.3M > > As sigterm is happening, memory available is still dropping. Some > process is still taking up more memory, but has low enough badness > that its not being terminated. It could be one of the exempt > processes. > > > > > Perhaps this ps_mem snippet is useful: > > > Private + Shared = RAM used Program > > > > > > > > 294.8 MiB + 4.7 MiB = 299.4 MiB telegram-desktop.bin > > 290.3 MiB + 22.1 MiB = 312.4 MiB rocketchat-desktop (5) > > 323.8 MiB + 9.5 MiB = 333.2 MiB kwin_x11 (9) > > 344.5 MiB + 1.4 MiB = 345.8 MiB nextcloud > > 373.2 MiB + 21.2 MiB = 394.4 MiB spotify (5) > > 416.6 MiB + 1.3 MiB = 417.9 MiB plasma-discover > > 448.5 MiB + 16.6 MiB = 465.2 MiB MainThread > > 892.8 MiB + 444.5 KiB = 893.2 MiB packagekitd > > 1.0 GiB + 8.6 MiB = 1.0 GiB plasmashell > > 1.9 GiB + 81.0 MiB = 1.9 GiB Web Content (9) > > 892M for packagekit? Seems excessive. I wonder what's up with that. It > is an exempt process. On my current system it's 1/2 that amount, which > is in turn twice that of GNOME shell. Hmm. > systemctl status packagekit ● packagekit.service - PackageKit Daemon Loaded: loaded (/usr/lib/systemd/system/packagekit.service; static; vendor preset: disabled) Active: active (running) since Thu 2020-07-30 15:40:51 -03; 2 weeks 1 days ago Main PID: 5393 (packagekitd) Tasks: 12 (limit: 19015) Memory: 903.2M CPU: 9min 6.976s CGroup: /system.slice/packagekit.service ├─ 5393 /usr/libexec/packagekitd ├─2019772 gpg-agent --homedir /var/cache/PackageKit/32/metadata/fedora-modular-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019790 gpg-agent --homedir /var/cache/PackageKit/32/metadata/updates-modular-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019808 gpg-agent --homedir /var/cache/PackageKit/32/metadata/google-chrome-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019819 gpg-agent --homedir /var/cache/PackageKit/32/metadata/updates-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019838 gpg-agent --homedir /var/cache/PackageKit/32/metadata/rpmfusion-free-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019857 gpg-agent --homedir /var/cache/PackageKit/32/metadata/ring-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019868 gpg-agent --homedir /var/cache/PackageKit/32/metadata/rpmfusion-free-updates-32-x86_64.tmp/gpgdir --use-standard-socket --daemon ├─2019885 gpg-agent --homedir /var/cache/PackageKit/32/metadata/fedora-32-x86_64.tmp/gpgdir --use-standard-socket --daemon └─2019926 gpg-agent --homedir /var/cache/P
Re: EarlyOOM +ZRAM Only
On Fri, Aug 14, 2020 at 12:42 PM Sergio Belkin wrote: > > 2 comments: > - I don't use disk-based swap, only zram. > - It happened again, and in this case there is no Virtual Machine nor Zoom > app running: > ago 14 15:08:37 dublin.ireland.home earlyoom[888]: sending SIGTERM to process > 2052260 uid 1000 "Web Content": badness 322, VmRSS 447 MiB > ago 14 15:08:40 dublin.ireland.home earlyoom[888]: process exited after 2.7 > seconds > ago 14 15:12:15 dublin.ireland.home earlyoom[888]: mem avail: 395 of 15887 > MiB ( 2.49%), swap free:0 of 4095 MiB ( 0.00%) > ago 14 15:12:15 dublin.ireland.home earlyoom[888]: low memory! at or below > SIGTERM limits: mem 2.52%, swap 10.00% > ago 14 15:12:15 dublin.ireland.home earlyoom[888]: sending SIGTERM to process > 2055755 uid 1000 "Web Content": badness 319, VmRSS 392 MiB > ago 14 15:12:17 dublin.ireland.home earlyoom[888]: process exited after 2.8 > seconds > ago 14 15:28:35 dublin.ireland.home earlyoom[888]: mem avail: 371 of 15887 > MiB ( 2.34%), swap free:0 of 4095 MiB ( 0.00%) > ago 14 15:28:35 dublin.ireland.home earlyoom[888]: low memory! at or below > SIGTERM limits: mem 2.52%, swap 10.00% > ago 14 15:28:35 dublin.ireland.home earlyoom[888]: sending SIGTERM to process > 2062157 uid 1000 "Web Content": badness 327, VmRSS 553 MiB > ago 14 15:28:37 dublin.ireland.home earlyoom[888]: process exited after 2.3 > seconds What's the workload? What is filling up 4G of swap with inactive pages such that none of them are being freed? The usual case is available memory is well below the watermark before swap has filled. But in this case, it's the opposite. As sigterm is happening, memory available is still dropping. Some process is still taking up more memory, but has low enough badness that its not being terminated. It could be one of the exempt processes. > > Perhaps this ps_mem snippet is useful: > Private + Shared = RAM used Program > > > > 294.8 MiB + 4.7 MiB = 299.4 MiB telegram-desktop.bin > 290.3 MiB + 22.1 MiB = 312.4 MiB rocketchat-desktop (5) > 323.8 MiB + 9.5 MiB = 333.2 MiB kwin_x11 (9) > 344.5 MiB + 1.4 MiB = 345.8 MiB nextcloud > 373.2 MiB + 21.2 MiB = 394.4 MiB spotify (5) > 416.6 MiB + 1.3 MiB = 417.9 MiB plasma-discover > 448.5 MiB + 16.6 MiB = 465.2 MiB MainThread > 892.8 MiB + 444.5 KiB = 893.2 MiB packagekitd > 1.0 GiB + 8.6 MiB = 1.0 GiB plasmashell > 1.9 GiB + 81.0 MiB = 1.9 GiB Web Content (9) 892M for packagekit? Seems excessive. I wonder what's up with that. It is an exempt process. On my current system it's 1/2 that amount, which is in turn twice that of GNOME shell. Hmm. > Well I'll report soon, any idea (remember I have 16 GB of RAM with 4G of > zram-based swap) will be welcome Very common. The defaults need to serve this use case. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El vie., 14 ago. 2020 a las 9:28, Przemek Klosowski via devel (< devel@lists.fedoraproject.org>) escribió: > On 8/14/20 7:33 AM, John M. Harris Jr wrote: > > On Wednesday, August 12, 2020 1:16:34 PM MST Przemek Klosowski via devel > > wrote: > >> > >> This is weird---your swap was 100% full, and ram almost full, and yet > >> killing 4GB VirtualBox didn't seem to free up memory. I suspect some > >> sort of measurement or reporting error---if these numbers were accurate, > >> EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 > >> MiB is crazy. > > > Are you kidding? The system still had over a quarter of a gigabyte of > free > > RAM. There's no reason to start killing off processes at that point. > That's > > tons of free memory. To put that into perspective, that's enough free > memory > > to store over 1000 average-length novels directly in memory. > > When the swap is 100% full, it is not like you have a bunch of processes > neatly stored in it, and some free memory available---you have a mess of > partly swapped out processes reading back their pages from swap and > pushing other processes' RAM pages onto the fragmented swap, when they > are trying to run. > > This is the worst case of disk usage, and as we discussed before, the > transfer speeds for such traffic will be hundreds/thousands times > slower---I would expect latencies going into tens of seconds. So, no, I > am not kidding. > ___ > 2 comments: - I don't use disk-based swap, only zram. - It happened again, and in this case there is no Virtual Machine nor Zoom app running: ago 14 15:08:37 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 2052260 uid 1000 "Web Content": badness 322, VmRSS 447 MiB ago 14 15:08:40 dublin.ireland.home earlyoom[888]: process exited after 2.7 seconds ago 14 15:12:15 dublin.ireland.home earlyoom[888]: mem avail: 395 of 15887 MiB ( 2.49%), swap free:0 of 4095 MiB ( 0.00%) ago 14 15:12:15 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 14 15:12:15 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 2055755 uid 1000 "Web Content": badness 319, VmRSS 392 MiB ago 14 15:12:17 dublin.ireland.home earlyoom[888]: process exited after 2.8 seconds ago 14 15:28:35 dublin.ireland.home earlyoom[888]: mem avail: 371 of 15887 MiB ( 2.34%), swap free:0 of 4095 MiB ( 0.00%) ago 14 15:28:35 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 14 15:28:35 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 2062157 uid 1000 "Web Content": badness 327, VmRSS 553 MiB ago 14 15:28:37 dublin.ireland.home earlyoom[888]: process exited after 2.3 seconds Perhaps this ps_mem snippet is useful: 294.8 MiB + 4.7 MiB = 299.4 MiB telegram-desktop.bin 290.3 MiB + 22.1 MiB = 312.4 MiB rocketchat-desktop (5) 323.8 MiB + 9.5 MiB = 333.2 MiB kwin_x11 (9) 344.5 MiB + 1.4 MiB = 345.8 MiB nextcloud 373.2 MiB + 21.2 MiB = 394.4 MiB spotify (5) 416.6 MiB + 1.3 MiB = 417.9 MiB plasma-discover 448.5 MiB + 16.6 MiB = 465.2 MiB MainThread 892.8 MiB + 444.5 KiB = 893.2 MiB packagekitd 1.0 GiB + 8.6 MiB = 1.0 GiB plasmashell 1.9 GiB + 81.0 MiB = 1.9 GiB Web Content (9) - 9.7 GiB = Well I'll report soon, any idea (remember I have 16 GB of RAM with 4G of zram-based swap) will be welcome -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On 8/14/20 7:33 AM, John M. Harris Jr wrote: On Wednesday, August 12, 2020 1:16:34 PM MST Przemek Klosowski via devel wrote: This is weird---your swap was 100% full, and ram almost full, and yet killing 4GB VirtualBox didn't seem to free up memory. I suspect some sort of measurement or reporting error---if these numbers were accurate, EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 MiB is crazy. Are you kidding? The system still had over a quarter of a gigabyte of free RAM. There's no reason to start killing off processes at that point. That's tons of free memory. To put that into perspective, that's enough free memory to store over 1000 average-length novels directly in memory. When the swap is 100% full, it is not like you have a bunch of processes neatly stored in it, and some free memory available---you have a mess of partly swapped out processes reading back their pages from swap and pushing other processes' RAM pages onto the fragmented swap, when they are trying to run. This is the worst case of disk usage, and as we discussed before, the transfer speeds for such traffic will be hundreds/thousands times slower---I would expect latencies going into tens of seconds. So, no, I am not kidding. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Fri, Aug 14, 2020 at 7:34 AM John M. Harris Jr wrote: > > On Wednesday, August 12, 2020 1:16:34 PM MST Przemek Klosowski via devel > wrote: > > On 8/12/20 2:27 PM, Sergio Belkin wrote: > > > > > Hi! > > > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based > > > swap partition. > > > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian > > > with 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( > > > > > > This is weird---your swap was 100% full, and ram almost full, and yet > > killing 4GB VirtualBox didn't seem to free up memory. I suspect some > > sort of measurement or reporting error---if these numbers were accurate, > > EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 > > MiB is crazy. > > > > One possibility that comes to mind is that there's a process not > > mentioned in the log (some system process???) that keeps allocating > > memory as soon as it is freed by EarlyOOM, so killing the processes does > > not result in increasing free memory (the first column). > > > > 399 of 15887 MiB ( 2.51%) SIGTERM "Web Content": badness 315, VmRSS 313 MiB > > 397 of 15887 MiB ( 2.50%) SIGTERM "Web Content": badness 304, VmRSS 80 MiB > > 379 of 15887 MiB ( 2.39%) SIGTERM "Web Content": badness 303, VmRSS 74 MiB > > 335 of 15887 MiB ( 2.11%) SIGTERM "Web Content": badness 303, VmRSS 70 MiB > > 315 of 15887 MiB ( 1.98%) SIGTERM "Web Content": badness 301, VmRSS 27 MiB > > 288 of 15887 MiB ( 1.81%) SIGTERM "VirtualBoxVM":badness 212, VmRSS 4244 > > MiB 337 of 15887 MiB ( 2.13%) SIGTERM "zoom":badness 36, VmRSS 721 > > MiB > > Note how the full log helpfully mentions the actual tresholds for > > SIGTERM: mem 2.52%, swap 10%, Where does 2.52 come from, pray? > > Are you kidding? The system still had over a quarter of a gigabyte of free > RAM. There's no reason to start killing off processes at that point. That's > tons of free memory. To put that into perspective, that's enough free memory > to store over 1000 average-length novels directly in memory. > It's also enough to store 1/8th of a significantly complex Electron process. It's *not enough*. Text is a terrible comparison when we're working with programs. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Wednesday, August 12, 2020 1:16:34 PM MST Przemek Klosowski via devel wrote: > On 8/12/20 2:27 PM, Sergio Belkin wrote: > > > Hi! > > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based > > swap partition. > > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian > > with 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( > > > This is weird---your swap was 100% full, and ram almost full, and yet > killing 4GB VirtualBox didn't seem to free up memory. I suspect some > sort of measurement or reporting error---if these numbers were accurate, > EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 > MiB is crazy. > > One possibility that comes to mind is that there's a process not > mentioned in the log (some system process???) that keeps allocating > memory as soon as it is freed by EarlyOOM, so killing the processes does > not result in increasing free memory (the first column). > > 399 of 15887 MiB ( 2.51%) SIGTERM "Web Content": badness 315, VmRSS 313 MiB > 397 of 15887 MiB ( 2.50%) SIGTERM "Web Content": badness 304, VmRSS 80 MiB > 379 of 15887 MiB ( 2.39%) SIGTERM "Web Content": badness 303, VmRSS 74 MiB > 335 of 15887 MiB ( 2.11%) SIGTERM "Web Content": badness 303, VmRSS 70 MiB > 315 of 15887 MiB ( 1.98%) SIGTERM "Web Content": badness 301, VmRSS 27 MiB > 288 of 15887 MiB ( 1.81%) SIGTERM "VirtualBoxVM":badness 212, VmRSS 4244 > MiB 337 of 15887 MiB ( 2.13%) SIGTERM "zoom":badness 36, VmRSS 721 > MiB > Note how the full log helpfully mentions the actual tresholds for > SIGTERM: mem 2.52%, swap 10%, Where does 2.52 come from, pray? Are you kidding? The system still had over a quarter of a gigabyte of free RAM. There's no reason to start killing off processes at that point. That's tons of free memory. To put that into perspective, that's enough free memory to store over 1000 average-length novels directly in memory. -- John M. Harris, Jr. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Wed, Aug 12, 2020 at 12:28 PM Sergio Belkin wrote: > > Hi! > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based swap > partition. > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian with 4GB > of RAM), and EarlyOOM killed Zoom in the middle of a call :( Hi, Could you open a bug report against earlyoom, and attach 'journalctl -b -o short-monotonic > bugID-journal.txt' It's not necessary to trim it. But if you do, keep about 5-10 minutes prior to the first SIGTERM. I think what's happened is earlyoom issues multiple SIGTERM, which are ignored or delayed in actually terminating by the processes with the most badness. And it just so happens that the zoom process badness reacted to the terminate request despite having an order magnitude lower badness. Off hand I'd say earlyoom probably should only SIGTERM the top 3 offenders, possibly with some delay in between each one. And then once we're below the kill watermark, issue SIGKILL to the top offender again. The difficulty is that's a VM, in this case. We might need an oom_score_adj for qemu and virtualbox, because I think it's a bad idea to kill off running VM's. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Wednesday, August 12, 2020 3:43:24 PM MST Samuel Sieb wrote: > On 8/12/20 12:06 PM, John M. Harris Jr wrote: > > > On Wednesday, August 12, 2020 11:27:37 AM MST Sergio Belkin wrote: > > > >> mem avail: 337 of 15887 MiB ( 2.13%), swap free:0 of 4095 MiB ( > >> 0.00%) > > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > >> sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS > >> 721 > >> MiB > >> process exited after 0.0 seconds > >> > >> > >> > >> > >> So I wonder if is advisable using EarlyOOM + ZRAM Only, what do you > >> think? > >> Thanks in advance! > > > > > > Please keep this in mind going forward, and take a moment to consider > > enabling EarlyOOM in Fedora. As it turns out, EarlyOOM does exactly what > > it says it does: It kills your programs when you've still got plenty of > > free memory. > > Where do you see any evidence of "plenty of free memory". There was > nothing left. Maybe there is a bug in the reporting, but it definitely > wasn't doing what you say it was. There was over 300 MiB of memory available. That's more than a quarter of a gigabyte. That's a ton of RAM. There was no *swap* left. -- John M. Harris, Jr. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Wed, Aug 12, 2020 at 5:43 pm, Michael Catanzaro wrote: but it might instead get confused I meant to write: it might *then* get confused. That is, it should always start by killing the right process, but it might then continue to kill more unnecessarily. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
earlyoom actually works best with only zram. If you have a disk-based swap partition, your system is likely to become frozen and unusable before earlyoom gets a chance to save you. On Wed, Aug 12, 2020 at 4:16 pm, Przemek Klosowski via devel wrote: This is weird---your swap was 100% full, and ram almost full, and yet killing 4GB VirtualBox didn't seem to free up memory. I suspect some sort of measurement or reporting error---if these numbers were accurate, EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 MiB is crazy. I've seen this before actually. I think it's just a bug. Maybe earlyoom doesn't allow enough time after killing to see how the kill affects system memory use. It should always kill the real memory hog first, but it might instead get confused and start killing a bunch of other unproblematic processes before it realizes that system memory use is back under control. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On 8/12/20 12:06 PM, John M. Harris Jr wrote: On Wednesday, August 12, 2020 11:27:37 AM MST Sergio Belkin wrote: mem avail: 337 of 15887 MiB ( 2.13%), swap free:0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS 721 MiB process exited after 0.0 seconds So I wonder if is advisable using EarlyOOM + ZRAM Only, what do you think? Thanks in advance! Please keep this in mind going forward, and take a moment to consider enabling EarlyOOM in Fedora. As it turns out, EarlyOOM does exactly what it says it does: It kills your programs when you've still got plenty of free memory. Where do you see any evidence of "plenty of free memory". There was nothing left. Maybe there is a bug in the reporting, but it definitely wasn't doing what you say it was. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El mié., 12 ago. 2020 a las 17:17, Przemek Klosowski via devel (< devel@lists.fedoraproject.org>) escribió: > On 8/12/20 2:27 PM, Sergio Belkin wrote: > > Hi! > > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based > > swap partition. > > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian > > with 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( > > This is weird---your swap was 100% full, and ram almost full, and yet > killing 4GB VirtualBox didn't seem to free up memory. I suspect some > sort of measurement or reporting error---if these numbers were accurate, > EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 > MiB is crazy. > > One possibility that comes to mind is that there's a process not > mentioned in the log (some system process???) that keeps allocating > memory as soon as it is freed by EarlyOOM, so killing the processes does > not result in increasing free memory (the first column). > > 399 of 15887 MiB ( 2.51%) SIGTERM "Web Content": badness 315, VmRSS 313 MiB > 397 of 15887 MiB ( 2.50%) SIGTERM "Web Content": badness 304, VmRSS 80 MiB > 379 of 15887 MiB ( 2.39%) SIGTERM "Web Content": badness 303, VmRSS 74 MiB > 335 of 15887 MiB ( 2.11%) SIGTERM "Web Content": badness 303, VmRSS 70 MiB > 315 of 15887 MiB ( 1.98%) SIGTERM "Web Content": badness 301, VmRSS 27 MiB > 288 of 15887 MiB ( 1.81%) SIGTERM "VirtualBoxVM":badness 212, VmRSS 4244 > MiB > 337 of 15887 MiB ( 2.13%) SIGTERM "zoom":badness 36, VmRSS 721 MiB > > Note how the full log helpfully mentions the actual tresholds for > SIGTERM: mem 2.52%, swap 10%, Where does 2.52 come from, pray? > ___ > The only that I've found in logs is the infamous akonadi :) and sysstat-collect... earlyoom More information: systemctl status --no-pager --full earlyoom.service ● earlyoom.service - Early OOM Daemon Loaded: loaded (/usr/lib/systemd/system/earlyoom.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2020-07-30 15:36:11 -03; 1 weeks 6 days ago Docs: man:earlyoom(1) https://github.com/rfjakob/earlyoom Main PID: 888 (earlyoom) Tasks: 1 (limit: 10) Memory: 1.8M (max: 50.0M) CPU: 1min 48.708s CGroup: /system.slice/earlyoom.service └─888 /usr/bin/earlyoom -r 0 -m 4 -M 409600 --prefer ^Web Content$ --avoid ^(dnf|packagekitd|gnome-shell|gnome-session-c|gnome-session-b|lightdm|sddm|sddm-helper|gdm|gdm-wayland-ses|gdm-session-wor|gdm-x-session|Xorg|Xwayland|systemd|systemd-logind|dbus-daemon|dbus-broker|cinnamon|cinnamon-sessio|kwin_x11|kwin_wayland|plasmashell|ksmserver|plasma_session|startplasma-way|xfce4-session|mate-session|marco|lxqt-session|openbox)$ ago 12 10:20:14 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1899907 uid 1000 "Web Content": badness 301, VmRSS 27 MiB ago 12 10:20:24 dublin.ireland.home earlyoom[888]: kill failed: Timer expired ago 12 10:20:24 dublin.ireland.home earlyoom[888]: mem avail: 288 of 15887 MiB ( 1.81%), swap free:0 of 4095 MiB ( 0.00%) ago 12 10:20:24 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 12 10:20:24 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1898928 uid 1000 "VirtualBoxVM": badness 212, VmRSS 4244 MiB ago 12 10:20:24 dublin.ireland.home earlyoom[888]: process exited after 0.0 seconds ago 12 10:20:24 dublin.ireland.home earlyoom[888]: mem avail: 337 of 15887 MiB ( 2.13%), swap free:0 of 4095 MiB ( 0.00%) ago 12 10:20:24 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 12 10:20:24 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS 721 MiB ago 12 10:20:24 dublin.ireland.home earlyoom[888]: process exited after 0.0 seconds -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El mié., 12 ago. 2020 a las 16:07, John M. Harris Jr () escribió: > On Wednesday, August 12, 2020 11:27:37 AM MST Sergio Belkin wrote: > > Hi! > > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based > swap > > partition. > > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian with > > 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( > > > > This is the log: > > mem avail: 399 of 15887 MiB ( 2.51%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899361 uid 1000 "Web Content": badness 315, > > VmRSS 313 MiB > > process exited after 2.2 seconds > > mem avail: 397 of 15887 MiB ( 2.50%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899726 uid 1000 "Web Content": badness 304, > > VmRSS 80 MiB > > process exited after 3.7 seconds > > mem avail: 379 of 15887 MiB ( 2.39%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1896099 uid 1000 "Web Content": badness 303, > > VmRSS 74 MiB > > process exited after 3.6 seconds > > mem avail: 335 of 15887 MiB ( 2.11%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899195 uid 1000 "Web Content": badness 303, > > VmRSS 70 MiB > > process exited after 3.6 seconds > > mem avail: 315 of 15887 MiB ( 1.98%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899907 uid 1000 "Web Content": badness 301, > > VmRSS 27 MiB > > kill failed: Timer expired > > mem avail: 288 of 15887 MiB ( 1.81%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1898928 uid 1000 "VirtualBoxVM": badness 212, > > VmRSS 4244 MiB > > process exited after 0.0 seconds > > mem avail: 337 of 15887 MiB ( 2.13%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS 721 > > MiB > > process exited after 0.0 seconds > > > > > > So I wonder if is advisable using EarlyOOM + ZRAM Only, what do you > > think? > > Thanks in advance! > > Please keep this in mind going forward, and take a moment to consider > enabling > EarlyOOM in Fedora. As it turns out, EarlyOOM does exactly what it says it > does: It kills your programs when you've still got plenty of free memory. > > -- > John M. Harris, Jr. > > ___ > The OS is Fedora 32... Debian is the guest OS running on VirtualBox -- -- Sergio Belkin LPIC-2 Certified - http://www.lpi.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
El mié., 12 ago. 2020 a las 15:49, Samuel Sieb () escribió: > On 8/12/20 11:27 AM, Sergio Belkin wrote: > > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based > > swap partition. > > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian > > with 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( > > > > This is the log: > > The log of what? Are there no timestamps? > > > mem avail: 399 of 15887 MiB ( 2.51%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899361 uid 1000 "Web Content": badness 315, > > VmRSS 313 MiB > > process exited after 2.2 seconds > > mem avail: 397 of 15887 MiB ( 2.50%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899726 uid 1000 "Web Content": badness 304, > > VmRSS 80 MiB > > process exited after 3.7 seconds > > mem avail: 379 of 15887 MiB ( 2.39%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1896099 uid 1000 "Web Content": badness 303, > > VmRSS 74 MiB > > process exited after 3.6 seconds > > mem avail: 335 of 15887 MiB ( 2.11%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899195 uid 1000 "Web Content": badness 303, > > VmRSS 70 MiB > > process exited after 3.6 seconds > > mem avail: 315 of 15887 MiB ( 1.98%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1899907 uid 1000 "Web Content": badness 301, > > VmRSS 27 MiB > > kill failed: Timer expired > > mem avail: 288 of 15887 MiB ( 1.81%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1898928 uid 1000 "VirtualBoxVM": badness 212, > > VmRSS 4244 MiB > > process exited after 0.0 seconds > > mem avail: 337 of 15887 MiB ( 2.13%), swap free:0 of 4095 MiB ( > 0.00%) > > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > > sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS > > 721 MiB > > process exited after 0.0 seconds > > According to this, you were completely out of memory. zoom was the last > resort. I wonder what was taking up the memory that even killing the VM > didn't free up enough. > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > It's the output of journalctl: ago 12 10:01:01 dublin.ireland.home earlyoom[888]: mem avail: 399 of 15887 MiB ( 2.51%), swap free:0 of 4095 MiB ( 0.00%) ago 12 10:01:01 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 12 10:01:01 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1899361 uid 1000 "Web Content": badness 315, VmRSS 313 MiB ago 12 10:01:03 dublin.ireland.home earlyoom[888]: process exited after 2.2 seconds ago 12 10:20:03 dublin.ireland.home earlyoom[888]: mem avail: 397 of 15887 MiB ( 2.50%), swap free:0 of 4095 MiB ( 0.00%) ago 12 10:20:03 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 12 10:20:03 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1899726 uid 1000 "Web Content": badness 304, VmRSS 80 MiB ago 12 10:20:07 dublin.ireland.home earlyoom[888]: process exited after 3.7 seconds ago 12 10:20:07 dublin.ireland.home earlyoom[888]: mem avail: 379 of 15887 MiB ( 2.39%), swap free:0 of 4095 MiB ( 0.00%) ago 12 10:20:07 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 12 10:20:07 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1896099 uid 1000 "Web Content": badness 303, VmRSS 74 MiB ago 12 10:20:10 dublin.ireland.home earlyoom[888]: process exited after 3.6 seconds ago 12 10:20:10 dublin.ireland.home earlyoom[888]: mem avail: 335 of 15887 MiB ( 2.11%), swap free:0 of 4095 MiB ( 0.00%) ago 12 10:20:10 dublin.ireland.home earlyoom[888]: low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% ago 12 10:20:10 dublin.ireland.home earlyoom[888]: sending SIGTERM to process 1899195 uid 1000 "Web Content": badness 303, VmRSS 70 MiB ago 12 10:20:14 dublin.ireland.home earlyoom[888]: process exited after 3.6 seconds ago 12 10:20:14 dublin.ireland.home earlyoom[888]: mem avail: 315 of 15887 MiB ( 1.98%), swap free:0 of 4095 MiB ( 0.
Re: EarlyOOM +ZRAM Only
On 8/12/20 2:27 PM, Sergio Belkin wrote: Hi! I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based swap partition. I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian with 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( This is weird---your swap was 100% full, and ram almost full, and yet killing 4GB VirtualBox didn't seem to free up memory. I suspect some sort of measurement or reporting error---if these numbers were accurate, EarlyOOM did have a reason to panic and kill zoom. BTW, zoom taking 721 MiB is crazy. One possibility that comes to mind is that there's a process not mentioned in the log (some system process???) that keeps allocating memory as soon as it is freed by EarlyOOM, so killing the processes does not result in increasing free memory (the first column). 399 of 15887 MiB ( 2.51%) SIGTERM "Web Content": badness 315, VmRSS 313 MiB 397 of 15887 MiB ( 2.50%) SIGTERM "Web Content": badness 304, VmRSS 80 MiB 379 of 15887 MiB ( 2.39%) SIGTERM "Web Content": badness 303, VmRSS 74 MiB 335 of 15887 MiB ( 2.11%) SIGTERM "Web Content": badness 303, VmRSS 70 MiB 315 of 15887 MiB ( 1.98%) SIGTERM "Web Content": badness 301, VmRSS 27 MiB 288 of 15887 MiB ( 1.81%) SIGTERM "VirtualBoxVM":badness 212, VmRSS 4244 MiB 337 of 15887 MiB ( 2.13%) SIGTERM "zoom": badness 36, VmRSS 721 MiB Note how the full log helpfully mentions the actual tresholds for SIGTERM: mem 2.52%, swap 10%, Where does 2.52 come from, pray? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On Wednesday, August 12, 2020 11:27:37 AM MST Sergio Belkin wrote: > Hi! > I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based swap > partition. > I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian with > 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( > > This is the log: > mem avail: 399 of 15887 MiB ( 2.51%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1899361 uid 1000 "Web Content": badness 315, > VmRSS 313 MiB > process exited after 2.2 seconds > mem avail: 397 of 15887 MiB ( 2.50%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1899726 uid 1000 "Web Content": badness 304, > VmRSS 80 MiB > process exited after 3.7 seconds > mem avail: 379 of 15887 MiB ( 2.39%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1896099 uid 1000 "Web Content": badness 303, > VmRSS 74 MiB > process exited after 3.6 seconds > mem avail: 335 of 15887 MiB ( 2.11%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1899195 uid 1000 "Web Content": badness 303, > VmRSS 70 MiB > process exited after 3.6 seconds > mem avail: 315 of 15887 MiB ( 1.98%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1899907 uid 1000 "Web Content": badness 301, > VmRSS 27 MiB > kill failed: Timer expired > mem avail: 288 of 15887 MiB ( 1.81%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1898928 uid 1000 "VirtualBoxVM": badness 212, > VmRSS 4244 MiB > process exited after 0.0 seconds > mem avail: 337 of 15887 MiB ( 2.13%), swap free:0 of 4095 MiB ( 0.00%) > low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% > sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS 721 > MiB > process exited after 0.0 seconds > > > So I wonder if is advisable using EarlyOOM + ZRAM Only, what do you > think? > Thanks in advance! Please keep this in mind going forward, and take a moment to consider enabling EarlyOOM in Fedora. As it turns out, EarlyOOM does exactly what it says it does: It kills your programs when you've still got plenty of free memory. -- John M. Harris, Jr. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: EarlyOOM +ZRAM Only
On 8/12/20 11:27 AM, Sergio Belkin wrote: I 've just had a problem using EarlyOOM + ZRAM. I haven't a disk-based swap partition. I was using mainly Zoom (desktop app) + Firefox + VirtualBox (Debian with 4GB of RAM), and EarlyOOM killed Zoom in the middle of a call :( This is the log: The log of what? Are there no timestamps? mem avail: 399 of 15887 MiB ( 2.51%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1899361 uid 1000 "Web Content": badness 315, VmRSS 313 MiB process exited after 2.2 seconds mem avail: 397 of 15887 MiB ( 2.50%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1899726 uid 1000 "Web Content": badness 304, VmRSS 80 MiB process exited after 3.7 seconds mem avail: 379 of 15887 MiB ( 2.39%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1896099 uid 1000 "Web Content": badness 303, VmRSS 74 MiB process exited after 3.6 seconds mem avail: 335 of 15887 MiB ( 2.11%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1899195 uid 1000 "Web Content": badness 303, VmRSS 70 MiB process exited after 3.6 seconds mem avail: 315 of 15887 MiB ( 1.98%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1899907 uid 1000 "Web Content": badness 301, VmRSS 27 MiB kill failed: Timer expired mem avail: 288 of 15887 MiB ( 1.81%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1898928 uid 1000 "VirtualBoxVM": badness 212, VmRSS 4244 MiB process exited after 0.0 seconds mem avail: 337 of 15887 MiB ( 2.13%), swap free: 0 of 4095 MiB ( 0.00%) low memory! at or below SIGTERM limits: mem 2.52%, swap 10.00% sending SIGTERM to process 1898342 uid 1000 "zoom": badness 36, VmRSS 721 MiB process exited after 0.0 seconds According to this, you were completely out of memory. zoom was the last resort. I wonder what was taking up the memory that even killing the VM didn't free up enough. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
