HI Sourabh,
On 19/07/26 9:59 AM, Sourabh Jain wrote:
On 16/07/26 10:21, Ritesh Harjani (IBM) wrote:
Sourabh Jain <[email protected]> writes:
This issue only occurs with the Radix MMU.
Booting the same kernel with disable_radix=1 does not reproduce the
issue.
The system also has pmem.
@Sourabh,
Few requests -
1. Could you also attach the boot logs from the same system with the
same kernel but with disable_radix=1?
2. I can't really tell on how is MMU related to this issue? But I think
if you could share the working dmesg logs from Hash that might give
some more info. I am suspecting what if the firmware negotiates
different page sizes even for TCEs for Radix v/s Hash.
Attached the dmesg of the kernel where issue is reproducible with and
without
disable_radix=1
3. Would it be possible for you to try git bisect and identify the
culprit commit?
git bisect start
# status: waiting for both good and bad commits
# good: [1471c517cf7dae1a6342fb821d8ed501af956dd0] powerpc/iommu: bypass
DMA APIs for coherent allocations for pre-mapped memory
git bisect good 1471c517cf7dae1a6342fb821d8ed501af956dd0
# status: waiting for bad commit, 1 good commit known
# bad: [328335a79487ec38d6b0e1aa807785b0f75e594d] powerpc/powernv/iommu:
iommu incorrectly bypass DMA APIs
git bisect bad 328335a79487ec38d6b0e1aa807785b0f75e594d
# bad: [939faf71cf7ca9ab3d1bd2912ac0e203d4d7156a] Merge tag 'drm-
next-2026-02-11' of https://gitlab.freedesktop.org/drm/kernel
git bisect bad 939faf71cf7ca9ab3d1bd2912ac0e203d4d7156a
# good: [66bbe4a8ed73f1187a4271c58f0ea30f42debe0d] Merge tag 'irq-
core-2026-02-09' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect good 66bbe4a8ed73f1187a4271c58f0ea30f42debe0d
# bad: [d061251387903e8502843ac983553f0b2e098ef8] Merge tag 'sound-7.0-
rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad d061251387903e8502843ac983553f0b2e098ef8
# good: [f51d34065de4f29fd237276a49cad06ac7356a55] Merge tag
'socfpga_dts_updates_for_v6.20_v3' of git://git.kernel.org/pub/scm/
linux/kernel/git/dinguyen/linux into soc/dt
git bisect good f51d34065de4f29fd237276a49cad06ac7356a55
# bad: [e86dda7bde8801d32ffe7d1570fe173cab14d1ba] Merge tag 'spi-v6.20'
of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/spi
git bisect bad e86dda7bde8801d32ffe7d1570fe173cab14d1ba
# good: [bdbddf72a2ab1cfea699959795d70df3931eefe7] Merge tag 'soc-
drivers-7.0' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect good bdbddf72a2ab1cfea699959795d70df3931eefe7
# good: [8ea39d960c9f890e9213cdcfcbe4b3f281acd12f] spi: add multi-lane
support
git bisect good 8ea39d960c9f890e9213cdcfcbe4b3f281acd12f
# bad: [d5cbd9f332c70be9589201474b9477baf9b5a24d] Merge tag 'regmap-
v6.20' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/regmap
git bisect bad d5cbd9f332c70be9589201474b9477baf9b5a24d
# good: [8ad8d24d962804d3c54b5b7542c4cb7cd6bacb3b] Merge tag 'parisc-
for-7.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/deller/
parisc-linux
git bisect good 8ad8d24d962804d3c54b5b7542c4cb7cd6bacb3b
# bad: [1d72a02d65a733ad124bacc2db3fb90fa81a612b] docs: trusted-
encryped: add PKWM as a new trust source
git bisect bad 1d72a02d65a733ad124bacc2db3fb90fa81a612b
# bad: [11d45eee9f42f7d3f9c9f190a6f79e570d345fe2] powerpc64/bpf:
Additional NVR handling for bpf_throw
git bisect bad 11d45eee9f42f7d3f9c9f190a6f79e570d345fe2
# bad: [2ed2d8f6fb3853daa536d79e0799cbac91fac9f5] powerpc64/bpf: Support
tailcalls with subprogs
git bisect bad 2ed2d8f6fb3853daa536d79e0799cbac91fac9f5
# bad: [815a8d2feb5615ae7f0b5befd206af0b0160614c] powerpc/eeh: fix
recursive pci_lock_rescan_remove locking in EEH event handling
git bisect bad 815a8d2feb5615ae7f0b5befd206af0b0160614c
Reverting commit 815a8d2feb56 ("powerpc/eeh: fix recursive
pci_lock_rescan_remove locking in EEH event handling")
on top of commit fce2dfa773ced15f27d resolves the issue.
- Sourabh Jain
Btw, to clarify I think there are two issues here:
1. EEH triggered for the USB adapter - I think you meant that with Hash
you don't even see EEH error being triggered at all.
2. Second problem is - EEH recovery causing this WARNING.
[ 17.375534] [ T118] WARNING: arch/powerpc/kernel/eeh_pe.c:439 at
eeh_pe_tree_remove+0xf8/0x2f0, CPU#10: eehd/118
<...>
[ 17.375599] [ T118] CPU: 10 UID: 0 PID: 118 Comm: eehd Tainted:
G E 7.2.0-rc3upstream+ #3 PREEMPT(lazy)
f29942c1738362234fefd88de0d1ed711f1a557e
[ 17.375603] [ T118] Tainted: [E]=UNSIGNED_MODULE
[ 17.375604] [ T118] Hardware name: IBM,9080-HEX Power11
(architected) 0x820200 0xf000007 of:IBM,FW1120.00 (NH1120_135) hv:phyp
pSeries
<...>
[ 17.375650] [ T118] NIP [c00000000004a048]
eeh_pe_tree_remove+0xf8/0x2f0
[ 17.375654] [ T118] LR [c00000000004ba6c]
eeh_pe_detach_dev+0x5c/0xa0
[ 17.375658] [ T118] Call Trace:
[ 17.375659] [ T118] [c00000000a357c90] [c00000000dd32800]
0xc00000000dd32800 (unreliable)
[ 17.375668] [ T118] [c00000000a357d30] [c00000000004ba6c]
eeh_pe_detach_dev+0x5c/0xa0
[ 17.375672] [ T118] [c00000000a357d60] [c0000000000496ec]
eeh_pe_traverse+0x6c/0xf0
[ 17.375676] [ T118] [c00000000a357da0] [c00000000004e100]
eeh_reset_device+0x1f8/0x298
[ 17.375680] [ T118] [c00000000a357e50] [c00000000004ce5c]
eeh_handle_normal_event+0x61c/0xa60
[ 17.375685] [ T118] [c00000000a357f30] [c00000000004e298]
eeh_event_handler+0xf8/0x190
[ 17.375689] [ T118] [c00000000a357f90] [c00000000025dab4]
kthread+0x194/0x1b0
-ritesh
Thanks for the bisect output — that's very helpful.
Commit 1010b4c012b0 ("powerpc/eeh: Make EEH driver device hotplug safe")
restructured EEH driver synchronization with the PCI hotplug layer, but
inadvertently introduced a deadlock that was silently preventing eehd
from detecting and recovering from EEH events.
Commit 815a8d2feb56 ("powerpc/eeh: fix recursive pci_lock_rescan_remove
locking in EEH event handling") resolved that deadlock — but as a
consequence, EEH processing now proceeds further and exposes this
pre-existing WARNING in eeh_pe_tree_remove().
This means reverting 815a8d2feb56 makes the issue disappear only because
it re-introduces the deadlock that was masking it — it is not a real fix.
To further validate this, it would be useful to inject an error into the
PE on a kernel with 815a8d2feb56 applied, and confirm whether EEH
detection and recovery can proceed correctly end-to-end once the
WARNING in eeh_pe_tree_remove() is also addressed.
Thanks,
Narayana.
