I was discussing cgroups with Lasse Collin recently, because there was a
feature request that xz-utils be cgroup aware [1] and Lasse was working
on a draft implementation alongside someone who kindly submitted a PR
[2]. The conversation was very surface level, primarily because my
knowledge of cgroups is very limited. I just remember that it was
commonly requested that 'nproc' account for cgroups until Pádraig added
it in coreutils-9.8 [3]. Looking back, I feel that change was an obvious
improvement given how common containers are. I don't remember if there
was a reason it wasn't added sooner, though.
The conversation did, however, remind me that coreutils, through the use
of the physmem module from Gnulib, does not account for cgroup memory
limits. This value is particularly important for 'sort' which will use
external sorting before reaching the systems memory limit. We can't just
allocate memory until it fails, since many systems will overcommit
memory until the process is killed or the system hangs.
Given that physmem doesn't account for cgroups though, 'sort' may
allocate more than allowed by a cgroup, leading to a process being
killed.
Here I create a container and allocate over the memory limit using
Python for the sake of simplicity:
$ podman run --memory 2G --rm -it fedora:latest
$ cat /sys/fs/cgroup/memory.max | numfmt --to=iec
2.0G
$ { dnf update -y; dnf install -y python3; } > /dev/null 2>&1
$ python3 -c 'x = bytearray(4 * 1024**3);'
Killed python3 -c 'x = bytearray(4 * 1024**3);'
We can see that it is killed because it exceeds the cgroup maxmemory:
$ sudo journalctl --since "1 minute ago" | grep -F 'Killed process'
Aug 19 20:09:32 fedora kernel: Memory cgroup out of memory: Killed process
239699 (python3) total-vm:4208896kB, anon-rss:2088464kB, file-rss:2472kB,
shmem-rss:0kB, UID:1000 pgtables:8260kB oom_score_adj:200
I think that it would be nice to account for cgroup memory limits here,
so that 'sort' doesn't allocate more memory than it is allowed by the
cgroup. Containers are useful, at least for me, as throw away
development environments. I suspect others would also appreciate the
improvements to 'sort' in this regard.
However, I figured that it was best to discuss on list before writing an
implementation. Lasse noted an increase in startup time from determining
memory.max and cpu.max that would add up when operating on many small
files [4]. He also made me aware of some SELinux policy changes that
were needed to accomodate various coreutils programs that use the nproc
module opening files to determine cpu.max after the coreutils-9.8
changes [5]. I guess it would be nice to give them a heads up WRT to any
changes in this area.
Any thoughts/concerns before myself (or anyone else) takes a look at
implementing this?
Collin
[1] https://github.com/tukaani-project/xz/issues/234
[2] https://github.com/tukaani-project/xz/pull/235
[3]
https://github.com/coreutils/coreutils/commit/c4df55be7f78d29415abd7765f863ff38184f86e
[4] https://github.com/tukaani-project/xz/issues/234#issuecomment-5026891074
[5]
https://github.com/SELinuxProject/refpolicy/commit/1e55618a24da234d8d3e629d9ec942e6026f7d94
