Milan Broz <[email protected]>:
> Anyway, my understanding is that all device-mapper targets use mempools,
> which should ensure that they can process even under memory pressure.
Let me give you more details.
Here is output of "free -h":
total used free shared buff/cache available
Mem: 62Gi 47Gi 924Mi 2.4Gi 17Gi 14Gi
Swap: 378Gi 95Gi 282Gi
Swap is located on dm-integrity on real partition.
As you can see, my data does not fit into physical memory, so swap is required
here.
But swap is big, so in theory allocations should always work.
I have a lot of Chromium windows opened (nearly 200).
My laptop is Dell Precision 7780. It is high speced expensive laptop.
I have 64 GiB ECC physical memory, btrfs raid on two 3.5 TiB partitions.
Everything is located on two 4 TiB NVMe SSD physical disks.
Sometimes whole system freezes for several minutes when I open new memory-hungry
Chromium tabs. In such cases I see in logs:
https://zerobin.net/?383b5c32b958aca8#yXmgYidkC8pUFixwQKB+v+O3bkbis4RHduz3gji4DxI=
Notice that all backtraces contain shmem_swapin_folio, so swap is involved here.
Hibernation works thanks to my patch
https://zerobin.net/?ad6142bd67df015a#68Az6yBUxHA3AXB7jY1+clSRnR745olFHAByxwPGM08=
.
My kernel is 6.12.48 from Debian with my local patches.
Sometimes I see messages "page allocation failure" in my logs. This is very
strange: I already explained above, that there is a plenty of space in swap.
Here is output of "journalctl | grep -B 10 -A 100 'page allocation failure'":
https://zerobin.net/?4170949dd9a8b25c#p5Z73TfGgpem4O4UsiWllrMCLCoHzDEw+KwJ7n8LWPA=
Maybe my swap is fragmented?
I that logs I notice that:
- Allocation failures often happen immidiately after wake up from hibernation
or suspend
- We try to alloc page of order 4 (what this means? 2^4 pages?)
- GFP mask is "GFP_KERNEL|__GFP_COMP" or "GFP_NOIO|__GFP_COMP". Failure to
allocate in "GFP_NOIO|__GFP_COMP" case is somewhat understandable. But
what about "GFP_KERNEL|__GFP_COMP"? As well as I understand, we are allowed
to do I/O, so we can drop everything to swap. And swap is big. So why we
fail?
- In all backtraces "dell_smbios_call" is involved
Hibernation always works, but takes a lot of time. Usually several minutes.
When hibernating, I see in logs this:
Dec 20 10:02:18 comp kernel: PM: hibernation: Allocated 26015132 kbytes in
193.21 seconds (134.64 MB/s)
I. e. 3 minutes to allocate space in memory for hibernation image.
And sometimes even this:
Dec 11 08:34:26 comp kernel: PM: hibernation: Allocated 25942484 kbytes in
348.90 seconds (74.35 MB/s)
Also sometimes I notice that in browser background for one site is replaced
with black rectangle. So, I assume that browser failed to allocate something,
too, but I unable to find this in logs.
> Anyway, my understanding is that all device-mapper targets use mempools,
> which should ensure that they can process even under memory pressure.
This seems to be not true. I see a lot of words "alloc" in dm-integrity
code:
$ grep alloc drivers/md/dm-integrity.c
And it seems that allocation happens not only in initialization,
but also in normal operations (but I didn't looked at code carefully).
Also, I see a lot of mentions of bufio in dm-integrity code.
As well as I understand, this is some cache layer. But, as well as I understand,
in my case there should no be any caches, everything should be written
directly to partition.
So, how to debug this next?
Maybe there are some ioctls, etc, to avoid this problems or to enable
more verbose logging?
I even okay with inserting some printfs to kernel code, just send me patch.
--
Askar Safin
