Re: ASMedia ASM2142 USB host controller tries to DMA to address zero when doing bulk reads from multiple devices
On Wed, Jul 29, 2020 at 8:22 AM Oliver O'Halloran wrote: > > On Tue, Jul 21, 2020 at 3:51 PM Forest Crossman wrote: > > > > Hello, again! > > > > After fixing the issue in my previous thread using this patch[1], I > > decided to do some stress-testing of the controller to make sure it > > could handle my intended workloads and that there were no further DMA > > address issues that would need to be fixed. Unfortunately, it looks > > like there's still more work to be done: when I try to do long bulk > > reads from multiple devices simultaneously, eventually the host > > controller sends a DMA write to address zero, which then triggers EEH > > in my POWER9 system, causing the controller card to get hotplug-reset, > > which of course kills the disk-reading processes. For more details on > > the EEH errors, you can see my kernel's EEH message log[2]. > > Take the logged address with a grain of salt. If an error occurs while > translating the DMA address the PHB logs all zeros as the "DMA > Address" because it only keeps around the bits that it needs to fetch > the next level of the TCE table. The EEH dump says the error is due to > a TCE permission mis-match so odds the ASmedia controller is writing > to an address that's already been DMA unmapped, hence the logged > address being zeros. Interesting, that's good to know. I saw that the RXE_TCE_FESR had the "TCE Response Page Access Error" bit set, and had originally assumed that just meant the DMA to address zero was triggering that error since it wasn't in a mapped page, but after reading that bit's description again I think I understand it now. > Sorry, I probably should have mentioned that quirk in the last mail. > > > The results of the various tests I performed are listed below. > > > > Test results (all failures are due to DMA writes to address zero, all > > hubs are USB 3.0/3.1 Gen1 only, and all disks are accessed via the > > usb-storage driver): > > - Reading simultaneously from two or more disks behind a hub connected > > to one port on the host controller: > > - FAIL after 20-50 GB of data transferred for each device. > > - Reading simultaneously from two disks, each connected directly to > > one port on the host controller: > > - FAIL after about 800 GB of data transferred for each device. > > - Reading from one disk behind a hub connected to one port on the host > > controller: > > - OK for at least 2.7 TB of data transferred (I didn't test the > > whole 8 TB disk). > > - Writing simultaneously to two FL2000 dongles (using osmo-fl2k's > > "fl2k_test"), each connected directly to one port on the host > > controller: > > - OK, was able to write several dozen terabytes to each device over > > the course of a little over 21 hours. > > > > Seeing how simultaneous writes to multiple devices and reads from > > single devices both seem to work fine, I assume that means this is > > being caused by some race condition in the host controller firmware > > when it responds to multiple read requests. > > Most likely. It's possible it's a platform specific race with DMA > map/unmap too, but I think we would be seeing similar issues with > other devices if it was. Yeah, I have several other xHCI controllers connected to this system, and I've never experienced this issue with any of them. If the problem was a POWER-specific DMA map/unmap race I would expect to be having problems with those controllers as well. > > I also assume we're not > > going to be able to convince ASMedia to both fix the bug in their > > firmware and release the details on how to flash it from Linux, so I > > guess we'll just have to figure out how to make the driver talk to the > > controller in a way that avoids triggering the bad DMA write. As > > before, I decided to try a little kernel hacking of my own before > > sending this email, and tried separately enabling the > > XHCI_BROKEN_STREAMS and XHCI_ASMEDIA_MODIFY_FLOWCONTROL quirks in an > > attempt to fix this. As you might expect since you're reading this > > message, neither of those quirks fixed the issue, nor did they even > > make the transfers last any longer before failing. > > > > So now I've reached the limits of my understanding, and I need some > > help devising a fix. If anyone has any comments to that effect, or any > > questions about my hardware configuration, testing methodology, etc., > > please don't hesitate to air them. Also, if anyone needs me to perform > > additional tests, or collect more log information, I'd be happy to do > > that as well. > > I started writing a tool a whil
ASMedia ASM2142 USB host controller tries to DMA to address zero when doing bulk reads from multiple devices
Hello, again! After fixing the issue in my previous thread using this patch[1], I decided to do some stress-testing of the controller to make sure it could handle my intended workloads and that there were no further DMA address issues that would need to be fixed. Unfortunately, it looks like there's still more work to be done: when I try to do long bulk reads from multiple devices simultaneously, eventually the host controller sends a DMA write to address zero, which then triggers EEH in my POWER9 system, causing the controller card to get hotplug-reset, which of course kills the disk-reading processes. For more details on the EEH errors, you can see my kernel's EEH message log[2]. The results of the various tests I performed are listed below. Test results (all failures are due to DMA writes to address zero, all hubs are USB 3.0/3.1 Gen1 only, and all disks are accessed via the usb-storage driver): - Reading simultaneously from two or more disks behind a hub connected to one port on the host controller: - FAIL after 20-50 GB of data transferred for each device. - Reading simultaneously from two disks, each connected directly to one port on the host controller: - FAIL after about 800 GB of data transferred for each device. - Reading from one disk behind a hub connected to one port on the host controller: - OK for at least 2.7 TB of data transferred (I didn't test the whole 8 TB disk). - Writing simultaneously to two FL2000 dongles (using osmo-fl2k's "fl2k_test"), each connected directly to one port on the host controller: - OK, was able to write several dozen terabytes to each device over the course of a little over 21 hours. Seeing how simultaneous writes to multiple devices and reads from single devices both seem to work fine, I assume that means this is being caused by some race condition in the host controller firmware when it responds to multiple read requests. I also assume we're not going to be able to convince ASMedia to both fix the bug in their firmware and release the details on how to flash it from Linux, so I guess we'll just have to figure out how to make the driver talk to the controller in a way that avoids triggering the bad DMA write. As before, I decided to try a little kernel hacking of my own before sending this email, and tried separately enabling the XHCI_BROKEN_STREAMS and XHCI_ASMEDIA_MODIFY_FLOWCONTROL quirks in an attempt to fix this. As you might expect since you're reading this message, neither of those quirks fixed the issue, nor did they even make the transfers last any longer before failing. So now I've reached the limits of my understanding, and I need some help devising a fix. If anyone has any comments to that effect, or any questions about my hardware configuration, testing methodology, etc., please don't hesitate to air them. Also, if anyone needs me to perform additional tests, or collect more log information, I'd be happy to do that as well. Thanks in advance for your help, Forest [1]: https://patchwork.kernel.org/patch/11669989/ [2]: https://paste.debian.net/hidden/2a442aa6
Re: ASMedia USB 3.x host controllers triggering EEH on POWER9
> In the future you can use this script to automate some of the tedium > of parsing the eeh dumps: > https://patchwork.ozlabs.org/project/skiboot/patch/20200717044243.1195833-1-ooh...@gmail.com/ Ah, nice, thanks for showing me this! I had written my own parser that just dumped a few register names and what bits were set in each, but your script seems much more feature-complete. > Anyway, for background the way PHB3 and PHB4 handle incoming DMAs goes > as follows: > > 1. Map the 16bit number to an IOMMU context, we call > those PEs. PE means "partitionable endpoint", but for the purpose of > processing DMAs you can ignore that and just treat it as an IOMMU > context ID. > 2. Use the PE number and some of the upper bits of the DMA address to > form the index into the Translation Validation Table. > 3. Use the table entry to validate the DMA address is within bounds > and whether it should be translated by the IOMMU or used as-is. > > If the table entry says the DMA needs to be translated by the IOMMU we'll > also: > 4. Walk the IOMMU table to get the relevant IOMMU table entry. > 5. Validate the device has permission to read/write to that address. > > The "TVT Address Range Error" you're seeing means that the bounds > checks done in 3) is failing. OPAL configures the PHB so there's two > TVT entries (TVEs for short) assigned to each PE. Bit 59 of the DMA > address is used to select which TVE to use. We typically configure > TVE#0 to map 0x0...0x8000_ so there's a 2GB 32bit DMA window. > TVE#1 is configured for no-translate (bypass) mode so you can convert > from a system physical address to a DMA address by ORing in bit 59. Thanks for the in-depth explanation, I find these low-level details really fascinating. > From word 2 of the PEST entry the faulting DMA address is: > 0x203974c0. That address is interesting since it looks a lot > like a memory address on the 2nd chip, but it doesn't have bit 59 set > so TVE#0 is used to validate it. Obviously that address is above 2GB > so we get the error. Ah, I see. Do you know if the information on the PEST registers is documented publicly somewhere? I tried searching for what those registers meant in the PHB4 spec but it basically just said, "the PEST registers contain PEST data," which isn't particularly helpful. > What's probably happening is that the ASmedia controller doesn't > actually implement all 64 address bits and truncates the upper bits of > the DMA address. Doing that is a blatant violation of the PCIe (and > probably the xHCI) spec, but it's also pretty common since "it works > on x86." Something to try would be booting with the iommu=nobypass in > the kernel command line. That'll disable TVE#1 and force all DMAs to > go through TVE#0. Thanks, iommu=nobypass fixed it! Plugging in one or more USB devices no longer triggers any EEH errors. > Assuming the nobypass trick above works, what you really need to do is > have the driver report that it can't address all 64bits by setting its > DMA mask accordingly. For the xhci driver it looks like this is done > in xhci_gen_setup(), there might be a quirks-style interface for > working around bugs in specific controllers that you can use. Have a > poke around and see what you can find :) Yup, the xhci driver has a quirks system, and conveniently one of those is XHCI_NO_64BIT_SUPPORT. After making a 3-line patch to xhci-pci.c to add that quirk for this chip, the host controller is now able to work without setting iommu=nobypass in the kernel arguments. Thank you so much for your help! You've likely saved me several hours of reading documentation, as well as several more hours of fiddling around with the xhci driver. I'm almost disappointed the fix was so simple, but the time savings alone more than makes up for it. I'll submit the patch to the USB ML shortly. Thanks again! Forest
ASMedia USB 3.x host controllers triggering EEH on POWER9
Hi, all, I have several ASMedia USB 3.x host controllers (ASM2142 and ASM3142, both share the same Vendor ID/Device ID pair) that I'd like to use with a POWER9 system (a Raptor Computing Systems Talos II). Unfortunately, while the kernel recognizes the controllers just fine, as soon as I plug in a device, an EEH error occurs and the host controller gets repeatedly reset until it eventually gets disabled. An example of one of these errors can be seen here: https://paste.debian.net/hidden/e39698eb Based on the "PHB4 Diag-data" reported by the kernel, it seems that LEM_WOF_R bit 35, PHB_FESR bit 20, and RXE_ARB_FESR bit 28 have been set. According to the PHB4 specification (https://ibm.ent.box.com/s/jftnfhceul07qjh9jtn91xwjmclabc71), they respectively mean the following: - ARB: IODA TVT Errors - "TCE Validation Table error occurred. The entry is invalid, or the PCI Address was out of range as defined by the TTA bounds in the TVE entry." - RXE_ARB OR Error Status - "RXE_ARB error bits, ... OR of all error status bits." - IODA TVT Address Range Error - "IODA Error: The PCI Address was out of range as defined by the TTA bounds in the TVE entry." In other words, the ASMedia USB controllers seem to be trying to write to addresses they're not supposed to, and thankfully the PHB4 is catching these bad writes before they can cause any corruption of my system's memory. Of course, this has the unfortunate side-effect that these devices are completely unable to operate with my computer, and since it seems to be possible to use these controllers on x86 systems (presumably because of the less-strict/disabled-by-default IOMMU), I wonder if maybe it would be possible to work around these errors in either the kernel or the OPAL firmware? My thinking is that instead of disconnecting the misbehaving devices, maybe the errors could be "forgiven" (but still blocked) and the device permitted to continue operating, possibly with some USB data loss from "writes to nowhere" or retries that may reduce performance. Or maybe if the issue is caused by some high address bits being set to random values, those bits could be masked-off so as to not trigger the errors and even avoid data loss. So, my question is, is any of this possible? I know the simple solution for me is to just RMA the cards and avoid purchasing ASMedia-based USB host controllers in the future, but the fact that they still seem to work "mostly ok" on x86 systems (with the occasional kernel panics and BSODs reported by users) piques my curiosity and makes me wonder if maybe there's a way for me to have my cheap, buggy hardware cake and eat it, too. Now, I'm a novice at kernel hacking, so I don't really know what I'm doing, but just for fun I did try to paper over the issue by adding an EEH handler to the xhci driver (https://paste.debian.net/hidden/16081515), but as you might expect, that didn't do anything but prevent further communication with the device. I also read a bunch of the PHB4 and IODA2 specs to see if maybe there'd be a way to implement that bit-masking thing I mentioned, but both of those documents are, uh, rather dry reading, so I haven't read them in their entirety, and I don't know enough about how this all works to try to search the text for what I need. All that said, if anyone has any suggestions or comments, I'd be really interested to hear them, even if it's just to question why I'd go to such ridiculous lengths to try to get software to account for buggy hardware. All the best, Forest
