Modified script to run with a local deb package
** Description changed:
SRU Justification
[Impact]
Expected:
Page-cache reclaim runs without faulting, regardless of the upper MMIO region
splitting System RAM.
What's happened:
Three of our Azure VMs (`Standard_L16as_v4` with the upper MMIO region below
`0x1000000000` on `linux-azure-6.8 6.8.0-1062.69~22.04.1`) running as
Elasticsearch data-nodes oopsed - two in kswapd0, one in the elasticsearch
process itself. Crashed machines were rebuilt urgently, non-crashed ones a bit
later, so we got apport and /proc/iomem from them:
- oops-node1-kswapd.log # fragment from crashed node
- oops-node2-kswapd.log # fragment from crashed node
- oops-node3-elasticsearch.log # fragment from crashed node
- proc-iomem-node4.log # output of same set machine, not crashed
- apport-node4.txt # apport of same set machine, not crashed
Suspect:
An oops in `shrink_folio_list()`.
[Fix]
Probably it's in `mm/vmscan.c:1277`
if (folio_test_large(folio) && list_empty(&folio->_deferred_list))
flags |= TTU_SYNC;
The line appeared in `Ubuntu-6.8.0-48.48` with the backport for LP
#2076147.
73bc32875ee9 mm: hold PTL from the first PTE while reclaiming a
large folio
Mainline dropped it - it seems that wasn't backported.
e5a119c4a683 mm/vmscan: drop checking if _deferred_list is empty
before using TTU_SYNC
The `azure-1062-read-deleted` target applies exactly this one-line
removal.
* Clean cherry-pick of 5ed890ce5147: "mm: vmscan: avoid split during
shrink_folio_list()"
* Clean cherry-pick of e5a119c4a683: "mm/vmscan: drop checking if
_deferred_list is empty before using TTU_SYNC" upstream
[Test Plan]
Steps to reproduce:
./reproduce.sh azure-1062 # or: noble-31 noble-137 noble-139
azure-1064 azure-1062-read-deleted azure-1062-mglru-off
The script fetches the kernel from Launchpad, builds a rootfs, and boots
QEMU: 4 GiB guest, 4 x 128 MB holes via `memmap=`, XFS on /dev/vda with
a 16 GiB file, read via mmap with jumps to random offsets.
Ext4 is untested.
Versions tested:
target kernel result
log
noble-31 linux 6.8.0-31.31 no oops,
5,000,000 jumps reproduce-noble-31.log
noble-137 linux 6.8.0-137.137 oops, before
jump 200,000 reproduce-noble-137.log
noble-139 linux 6.8.0-139.139 (proposed) oops, before
jump 400,000 reproduce-noble-139.log
azure-1062 linux-azure-6.8 1062.69~22.04.1 oops, before
jump 200,000 reproduce-azure-1062.log
azure-1064 linux-azure-6.8 1064.72~22.04.1 oops, before
jump 800,000 reproduce-azure-1064.log
azure-1062-read-deleted rebuilt from 1062 source no oops,
5,000,000 jumps reproduce-azure-1062-read-deleted.log
azure-1062-mglru-off rebuilt from 1062 source oops, before
jump 800,000 reproduce-azure-1062-mglru-off.log
In most of these runs the fault came twice: first in kswapd0, then in
the reading process itself. This matches both patterns we saw in
production.
All these logs come from runs with 4 holes and a limit of 5,000,000
jumps. Fewer holes make the fault less likely - with a single hole we
still got it, but the run took much longer. With 4 holes a bad kernel
faults in minutes, so we did not have to run each version for days. We
can run more tests if needed.
137.137 (noble Updates) and 1064.72 (jammy Updates) are the newest
published builds; 31.31 is noble Release before the backport named under
[Cause].
* Compile tested
* Tested against reproducer-folio.sh (reproducer.sh with custom deb packages)
- [Regression Potential}
+ [Regression Potential]
- * Patches isolated to mm/vmscan.c, in a one-line fix. Possible
- regression could occur in workflows that partially unmap large folios
- that are not split, though it's expected that also taking 5ed890ce5147:
- "mm: vmscan: avoid split during shrink_folio_list()" will address that.
+ * Patches isolated to mm/vmscan.c. Possible regression could occur in
+ workflows that partially unmap large folios that are not split, though
+ it's expected that also taking 5ed890ce5147: "mm: vmscan: avoid split
+ during shrink_folio_list()" will address that.
[Other info]
* SF#00454392
** Attachment added: "reproduce-folio.sh"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2163714/+attachment/6015228/+files/reproduce-folio.sh
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2163714
Title:
oops in shrink_folio_list(): out-of-bounds read of
folio->_deferred_list
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2163714/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
