Hi Paul,
Paul Klapperich <[email protected]> writes: > Hi everyone, > > I am encountering a reproducible PCIe/EEH regression on a PowerNV > platform (Talos II, POWER9) when moving from kernel 6.18.x to the > 7.1.x series. > > Hardware Context: > > Platform: Raptor Computing Systems Talos II (PowerNV, POWER9) > > Storage Controller: 0001:01:00.0 SATA controller: JMicron Technology > Corp. JMB58x AHCI SATA controller (direct root port, no PLX bridge) > > The Issue: On mainline/Arch POWER kernel 7.1.4, the kernel's > enforcement of a 32-bit DMA mask (AHCI_HFLAG_32BIT_ONLY) on the JMB58x > controller appears to be causing a low-memory translation failure > during early initialization. This immediately trips the PowerNV Host > Bridge (PHB) into an Enhanced Error Handling (EEH) frozen state and > locks up the bus. > > Conversely, booting the exact same hardware setup with a 6.18.7 kernel > succeeds completely, as it negotiates the 64-bit DMA window without > triggering the platform translation fault. > > Boot Log Contrast: > > Under working 6.18.7 kernel: > > # dmesg | grep -iE "ahci|eeh" 1. Could you please attach and share the full dmesg (with no grep filtering) for both 6.18.7 (working kernel) and 7.1.4 (non-working kernel) along with their respective .config files. That should clarify few other things. 2. cat /sys/firmware/opal/msglog From both the kernels (after bootup and possibly after an EEH error has occurred at-least once on the non-working kernel) -ritesh > [ 0.211910] EEH: PowerNV platform initialized > [ 0.391733] EEH: Capable adapter found: recovery enabled. > [ 2.192909] ahci 0001:01:00.0: enabling device (0540 -> 0542) > [ 2.193223] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled > [ 2.193264] ahci 0001:01:00.0: AHCI vers 0001.0301, 32 command > slots, 6 Gbps, SATA mode > [ 2.193270] ahci 0001:01:00.0: 5/5 ports implemented (port mask 0x1f) > [ 2.193275] ahci 0001:01:00.0: flags: 64bit ncq sntf stag pm led > clo pmp fbs pio slum part ccc apst boh > [ 2.196895] scsi host0: ahci > [ 2.197276] scsi host1: ahci > [ 2.198768] scsi host2: ahci > [ 2.199068] scsi host3: ahci > [ 2.199995] scsi host4: ahci > > Under broken 7.1.4 kernel: > > # dmesg | grep -iE "ahci|eeh" > [ 3.298353] EEH: PowerNV platform initialized > [ 3.460692] EEH: Capable adapter found: recovery enabled. > [ 4.510396] ahci 0001:01:00.0: enabling device (0540 -> 0542) > [ 4.510428] ahci 0001:01:00.0: controller can't do 64bit DMA, forcing > 32bit > [ 4.510691] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled > [ 4.510735] ahci 0001:01:00.0: AHCI vers 0001.0301, 32 command > slots, 6 Gbps, SATA mode > [ 4.510742] ahci 0001:01:00.0: 5/5 ports implemented (port mask 0x1f) > [ 4.510747] ahci 0001:01:00.0: flags: ncq sntf stag pm led clo pmp > fbs pio slum part ccc apst boh > [ 4.512167] scsi host0: ahci > [ 4.512556] scsi host1: ahci > [ 4.512831] scsi host2: ahci > [ 4.513076] scsi host3: ahci > [ 4.513466] scsi host4: ahci > [ 4.824011] EEH: Recovering PHB#1-PE#fd > [ 4.824019] EEH: PE location: UOPWR.A100019-Node0-CPU1 Slot1 (8x), > PHB location: N/A > [ 4.824023] EEH: Frozen PHB#1-PE#fd detected > [ 4.824025] EEH: Call Trace: > [ 4.824027] EEH: [000000007ad80bda] __eeh_send_failure_event+0x68/0x140 > [ 4.824038] EEH: [0000000042fa5ce7] > eeh_dev_check_failure.part.0+0x280/0x6d0 > [ 4.824044] EEH: [00000000ca5dc769] ioread32+0xd0/0x190 > [ 4.824051] EEH: [000000000ad730be] ata_wait_register+0xa4/0x108 [libata] > [ 4.824064] EEH: [000000007fb91492] > ahci_exec_polled_cmd.constprop.0+0x104/0x210 [libahci] > [ 4.824070] EEH: [0000000063044e17] ahci_do_softreset+0x140/0x380 > [libahci] > [ 4.824074] EEH: [00000000d97bc211] > ata_do_reset.constprop.0+0xbc/0xf0 [libata] > [ 4.824084] EEH: [0000000093546eba] ata_eh_reset+0xb14/0x12d0 [libata] > [ 4.824094] EEH: [00000000b2341f96] ata_eh_recover+0x3f0/0x1580 [libata] > [ 4.824102] EEH: [00000000bbcd8f21] > sata_pmp_error_handler+0x388/0xb90 [libata] > [ 4.824109] EEH: [00000000b87225ad] ahci_error_handler+0x6c/0x100 > [libahci] > [ 4.824114] EEH: [00000000eda810ea] > ata_scsi_port_error_handler+0x4a8/0x9a0 [libata] > [ 4.824123] EEH: [0000000006f6b52d] ata_scsi_error+0xc0/0x130 [libata] > [ 4.824131] EEH: [000000002528ea40] scsi_error_handler+0x11c/0x760 > [scsi_mod] > [ 4.824142] EEH: [000000005ba0d15d] kthread+0x17c/0x190 > [ 4.824148] EEH: [0000000067c7c775] start_kernel_thread+0x14/0x18 > [ 4.824153] EEH: This PCI device has failed 1 times in the last > hour and will be permanently disabled after 5 failures. > [ 4.824156] EEH: Notify device drivers to shutdown > [ 4.824159] EEH: Beginning: 'error_detected(IO frozen)' > [ 4.824162] PCI 0001:01:00.0#00fd: EEH: driver not EEH aware > [ 4.824165] EEH: Finished:'error_detected(IO frozen)' with > aggregate recovery state:'none' > [ 4.824170] EEH: Collect temporary log > [ 4.824198] EEH: of node=0001:01:00.0 > [ 4.824203] EEH: PCI device/vendor: 0585197b > [ 4.824207] EEH: PCI cmd/status register: 00100546 > [ 4.824209] EEH: PCI-E capabilities and status follow: > [ 4.824220] EEH: PCI-E 00: 00120010 10008102 0000285e 0041a023 > [ 4.824230] EEH: PCI-E 10: 00230008 00000000 00000000 00000000 > [ 4.824232] EEH: PCI-E 20: 00000000 > [ 4.824234] EEH: PCI-E AER capability register set follows: > [ 4.824245] EEH: PCI-E AER 00: 15020001 00000000 00400000 00462030 > [ 4.824254] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000 > [ 4.824275] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000 > [ 4.824279] EEH: PCI-E AER 30: 00000000 00000000 > [ 4.824313] EEH: Reset with hotplug activity > [ 14.513778] ahci 0001:01:00.0: AHCI controller unavailable! > [ 14.513794] ahci 0001:01:00.0: AHCI controller unavailable! > [ 15.561995] ahci 0001:01:00.0: AHCI controller unavailable! > [ 15.591811] ahci 0001:01:00.0: AHCI controller unavailable! > [ 15.591821] ahci 0001:01:00.0: AHCI controller unavailable! > [ 15.591828] ahci 0001:01:00.0: AHCI controller unavailable! > [ 15.591834] ahci 0001:01:00.0: AHCI controller unavailable! > [ 15.591840] ahci 0001:01:00.0: AHCI controller unavailable! > [ 20.592398] EEH: Sleep 5s ahead of complete hotplug > [ 26.110843] ahci 0001:01:00.0: enabling device (0140 -> 0142) > [ 26.110876] ahci 0001:01:00.0: controller can't do 64bit DMA, forcing > 32bit > [ 26.111136] ahci 0001:01:00.0: SSS flag set, parallel bus scan disabled > > this repeats until the device is disabled after 5 resets. > > Confirmation: > I built 7.1.4 after reverting commit 105c4256 "ata: ahci: force 32-bit > DMA for JMicron JMB582/JMB585" and boot completed successfully with an > apparently working JMB585 sata controller. > > Conclusion: > It appears that the recent JMicron 32-bit DMA workaround interacts > poorly with PowerNV. Is there anything a user might do to work around > this issue without reverting 105c4256? > > Let me know what additional information would be helpful. > > Thanks, > Paul
