Re: [systemd-devel] Q: A way to log peak memory footprint of deactivating units?

[Michal Koutný]

Hello.

On Sat, Aug 13, 2022 at 04:39:50AM -0500, Russell Haley 
 wrote:
> Since systemd logs the total CPU time used when the unit deactivates, I
> wonder if there's a way to make it log the peak memory footprint too,
> kind of like the time command's "maxresident". The unit does have
> MemoryAccounting=yes.
> 
> It turned out that it was simple to write a shell script loop to wait
> until packagekit was activated and sample the memory usage before the
> timeout expired. (About 228 MiB.)  However, I am still interested to
> know if there might be a better/more general method.
> 
> P.S. Since I had to re-send this mail with the correct From: address, I
> looked into it and apparently kernel 5.19 added a memory.peak to cgroups
> v2, so I think it very recently become possible to have an elegant
> implementation of this.

You need the kernel with memory.peak (sampling is unrealiable) and then
quick n' dirty

> MemoryAccounting=yes
> ExecStopPost=/usr/bin/cat /sys/fs/cgroup/system.slice/%n/memory.peak

Output goes to journald by default:
> Aug 19 00:13:29 machine cat[7433]: 2924544

If you wanted to have this reported by default, PRs for memory.peak
support are welcomed :-)


Michal

[systemd-devel] Q: A way to log peak memory footprint of deactivating units?

[Russell Haley]

For background, this came up when I was trying to work around Fedora's
infamous PackageKit memory leak, reported in various places:

https://github.com/PackageKit/PackageKit/issues/460
https://bugzilla.redhat.com/show_bug.cgi?id=1354074
https://bugzilla.redhat.com/show_bug.cgi?id=1854875
https://bugzilla.redhat.com/show_bug.cgi?id=1896964

I set ShutdownTimeout=60 in PackageKit.conf, somewhat more aggressive
than the 300s suggested in the github issue.  This worked great and
solved the persistent memory hogging on my machine.

However, I got curious about how much filesystem cache might be getting
forced out by the daily update checks. The log for a typical run looks
like this:

> Aug 12 19:59:14 censored systemd[1]: Starting packagekit.service - PackageKit 
> Daemon...
> Aug 12 19:59:14 censored PackageKit[40462]: daemon start
> Aug 12 19:59:14 censored systemd[1]: Started packagekit.service - PackageKit 
> Daemon.
> Aug 12 19:59:14 censored PackageKit[40462]: uid 1000 is trying to obtain 
> org.freedesktop.packagekit.system-sources-refresh auth (only_trusted:0)
> Aug 12 19:59:14 censored PackageKit[40462]: uid 1000 obtained auth for 
> org.freedesktop.packagekit.system-sources-refresh
> Aug 12 19:59:45 censored PackageKit[40462]: refresh-cache transaction 
> /4084_eceebbea from uid 1000 finished with success after 30946ms
> Aug 12 19:59:48 censored PackageKit[40462]: get-updates transaction 
> /4086_aedbdccb from uid 1000 finished with success after 2373ms
> Aug 12 19:59:48 censored PackageKit[40462]: get-updates transaction 
> /4087_dcbdccac from uid 1000 finished with success after 200ms
> Aug 12 20:00:49 censored PackageKit[40462]: daemon quit
> Aug 12 20:00:49 censored systemd[1]: packagekit.service: Deactivated 
> successfully.
> Aug 12 20:00:49 censored systemd[1]: packagekit.service: Consumed 14.698s CPU 
> time.

Since systemd logs the total CPU time used when the unit deactivates, I
wonder if there's a way to make it log the peak memory footprint too,
kind of like the time command's "maxresident". The unit does have
MemoryAccounting=yes.

It turned out that it was simple to write a shell script loop to wait
until packagekit was activated and sample the memory usage before the
timeout expired. (About 228 MiB.)  However, I am still interested to
know if there might be a better/more general method.

P.S. Since I had to re-send this mail with the correct From: address, I
looked into it and apparently kernel 5.19 added a memory.peak to cgroups
v2, so I think it very recently become possible to have an elegant
implementation of this.

Thanks,
Russ