Re: [PATCH v4 0/4] Add system mmu support for Armada-806
Hi, > Am 23.10.2020 um 14:19 schrieb Tomasz Nowicki : > > Hi Denis, > > Sorry for late response, we had to check few things. Please see comments > inline. > > On 10/6/20 3:16 PM, Denis Odintsov wrote: >> Hi, >>> Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki : >>> >>> The series is meant to support SMMU for AP806 and a workaround >>> for accessing ARM SMMU 64bit registers is the gist of it. >>> >>> For the record, AP-806 can't access SMMU registers with 64bit width. >>> This patches split the readq/writeq into two 32bit accesses instead >>> and update DT bindings. >>> >>> The series was successfully tested on a vanilla v5.8-rc3 kernel and >>> Intel e1000e PCIe NIC. The same for platform devices like SATA and USB. >>> >>> For reference, previous versions are listed below: >>> V1: https://lkml.org/lkml/2018/10/15/373 >>> V2: https://lkml.org/lkml/2019/7/11/426 >>> V3: https://lkml.org/lkml/2020/7/2/1114 >>> >> 1) After enabling SMMU on Armada 8040, and >> ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since >> 954a03be033c7cef80ddc232e7cbdb17df735663, >> internal eMMC is prevented from being initialised (as there is no iommus >> property for ap_sdhci0) >> Disabling "Disable bypass by default" make it work, but the patch highly >> suggest doing it properly. >> I wasn't able to find correct path for ap_sdhci for iommus in any publicly >> available documentation, >> would be highly appreciated addressed properly, thank you! >> 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is >> mpci ath10k card. >> It is found, it is enumerated, it is visible in lspci, but it fails to be >> initialised. Here is the log: > > Firmware has to configure and assign device StreamIDs. Most of the devices > are configured properly and supported in public FW. However, for both these > cases (ap_sdhci0 and PCIe) some extra (u-boot/UEFI/ATF) patches are required > which are not available yet. Sorry we let that happen. > > Since we have dependency on custom FW and we cannot enforce people to patch > their FW we will send the follow up fix patch (v5.9+) and revert respective > DTS changes. > > The most important Armada-806 SMMU driver enhancements were merged so people > who still willing to use SMMU need to provide proper DTB and use > ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=n (or via kernel command line) with extra > cautious. > Thank you very much for the reply, I'm a mere computer enthusiast who like to progress with the technology and keep up with the software. There was no harm at all with all those changes, and I see how they all are planned for a good reason. I'm looking forward for that patches and further progress, and thank you for your work. Denis. > Thanks, > Tomasz > >> [1.743754] armada8k-pcie f260.pcie: host bridge /cp0/pcie@f260 >> ranges: >> [1.751116] armada8k-pcie f260.pcie: MEM >> 0x00f600..0x00f6ef -> 0x00f600 >> [1.964690] armada8k-pcie f260.pcie: Link up >> [1.969379] armada8k-pcie f260.pcie: PCI host bridge to bus :00 >> [1.976026] pci_bus :00: root bus resource [bus 00-ff] >> [1.981537] pci_bus :00: root bus resource [mem 0xf600-0xf6ef] >> [1.988462] pci :00:00.0: [11ab:0110] type 01 class 0x060400 >> [1.994504] pci :00:00.0: reg 0x10: [mem 0x-0x000f] >> [2.000843] pci :00:00.0: supports D1 D2 >> [2.005132] pci :00:00.0: PME# supported from D0 D1 D3hot >> [2.011853] pci :01:00.0: [168c:003c] type 00 class 0x028000 >> [2.018001] pci :01:00.0: reg 0x10: [mem 0x-0x001f 64bit] >> [2.025002] pci :01:00.0: reg 0x30: [mem 0x-0x pref] >> [2.032111] pci :01:00.0: supports D1 D2 >> [2.049409] pci :00:00.0: BAR 14: assigned [mem 0xf600-0xf61f] >> [2.056322] pci :00:00.0: BAR 0: assigned [mem 0xf620-0xf62f] >> [2.063142] pci :00:00.0: BAR 15: assigned [mem 0xf630-0xf63f >> pref] >> [2.070484] pci :01:00.0: BAR 0: assigned [mem 0xf600-0xf61f >> 64bit] >> [2.077880] pci :01:00.0: BAR 6: assigned [mem 0xf630-0xf630 >> pref] >> [2.085135] pci :00:00.0: PCI bridge to [bus 01-ff] >> [2.090384] pci :00:00.0: bridge window [mem 0xf600-0xf61f] >> [2.097202] pci :00:00.0: bridge window [mem 0xf630-0xf63f >> pref] >> [2.104539] pcieport :00:00.0: Adding
Re: [PATCH v4 0/4] Add system mmu support for Armada-806
Hi, 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card. It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log: [1.743754] armada8k-pcie f260.pcie: host bridge /cp0/pcie@f260 ranges: [1.751116] armada8k-pcie f260.pcie: MEM 0x00f600..0x00f6ef -> 0x00f600 [1.964690] armada8k-pcie f260.pcie: Link up [1.969379] armada8k-pcie f260.pcie: PCI host bridge to bus :00 [1.976026] pci_bus :00: root bus resource [bus 00-ff] [1.981537] pci_bus :00: root bus resource [mem 0xf600-0xf6ef] [1.988462] pci :00:00.0: [11ab:0110] type 01 class 0x060400 [1.994504] pci :00:00.0: reg 0x10: [mem 0x-0x000f] [2.000843] pci :00:00.0: supports D1 D2 [2.005132] pci :00:00.0: PME# supported from D0 D1 D3hot [2.011853] pci :01:00.0: [168c:003c] type 00 class 0x028000 [2.018001] pci :01:00.0: reg 0x10: [mem 0x-0x001f 64bit] [2.025002] pci :01:00.0: reg 0x30: [mem 0x-0x pref] [2.032111] pci :01:00.0: supports D1 D2 [2.049409] pci :00:00.0: BAR 14: assigned [mem 0xf600-0xf61f] [2.056322] pci :00:00.0: BAR 0: assigned [mem 0xf620-0xf62f] [2.063142] pci :00:00.0: BAR 15: assigned [mem 0xf630-0xf63f pref] [2.070484] pci :01:00.0: BAR 0: assigned [mem 0xf600-0xf61f 64bit] [2.077880] pci :01:00.0: BAR 6: assigned [mem 0xf630-0xf630 pref] [2.085135] pci :00:00.0: PCI bridge to [bus 01-ff] [2.090384] pci :00:00.0: bridge window [mem 0xf600-0xf61f] [2.097202] pci :00:00.0: bridge window [mem 0xf630-0xf63f pref] [2.104539] pcieport :00:00.0: Adding to iommu group 4 [2.110232] pcieport :00:00.0: PME: Signaling with IRQ 38 [2.116141] pcieport :00:00.0: AER: enabled with IRQ 38 [8.131135] ath10k_pci :01:00.0: Adding to iommu group 4 [8.131874] ath10k_pci :01:00.0: enabling device ( -> 0002) [8.132203] ath10k_pci :01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0 up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37 Does forcing ath10k to use legacy interrupts rather than MSIs make a difference? Judging by the DT it looks like MSIs ought to be targeting the GICv2M widget, but if things somehow end up trying to use the PCIe controller's internal MSI doorbell (upstream of SMMU translation) instead, then that might account for general interrupt-related weirdness. Robin. Frankly speaking you quickly overcome here my knowledge depth, this is already way far from what I understand about PCI devices. But I tried my best to try it out what you suggested. putting 0 to /sys/bus/pci/devices/\:01\:00.0/msi_bus (bus of ath10k) and reloading the driver didn't make a difference with those almost same messages: [ 103.245287] ath10k_pci :01:00.0: MSI/MSI-X disallowed for future drivers [ 145.938562] ath10k_pci :01:00.0: pci irq legacy oper_irq_mode 1 irq_mode 0 reset_mode 0 [ 146.053590] ath10k_pci :01:00.0: failed to poke copy engine: -16 [ 146.161637] ath10k_pci :01:00.0: failed to poke copy engine: -16 [ 146.269515] ath10k_pci :01:00.0: failed to poke copy engine: -16 [ 146.453633] ath10k_pci :01:00.0: failed to poke copy engine: -16 [ 146.561589] ath10k_pci :01:00.0: failed to poke copy engine: -16 [ 146.669550] ath10k_pci :01:00.0: failed to poke copy engine: -16 [ 146.753456] ath10k_pci :01:00.0: Failed to get pcie state addr: -16 [ 146.760129] ath10k_pci :01:00.0: failed to setup init config: -16 [ 146.766695] ath10k_pci :01:00.0: could not power on hif bus (-16) [ 146.773196] ath10k_pci :01:00.0: could not probe fw (-16) lspci -v says Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit- So it seems it does what you suggested, disabled the MSI. With not much differences unfortunately. =( I also compared lscpi output also with non-SMMU-dts (with which ath10k works) and SMMU-dts, and there is a difference I guess, I'm not sure how affecting it is. On non-SMMU dts host controller (served by pcieport driver) IRQ is 37 and ath10k IRQ is 82: 00:00.0 PCI bridge: Marvell Technology Group Ltd. Device 0110 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 37 ... 01:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter Flags: bus master, fast devsel, latency 0, IRQ 82 On SMMU dt's though both IRQ's are same and are 38: 00:00.0 PCI bridge: Marvell Technology Group Ltd. Device 0110 (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0, IRQ 38 ... 01:00.0 Network controller: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter Flags: bus master, fast devsel,
Re: [PATCH v4 0/4] Add system mmu support for Armada-806
Hi, > Am 15.07.2020 um 09:06 schrieb Tomasz Nowicki : > > The series is meant to support SMMU for AP806 and a workaround > for accessing ARM SMMU 64bit registers is the gist of it. > > For the record, AP-806 can't access SMMU registers with 64bit width. > This patches split the readq/writeq into two 32bit accesses instead > and update DT bindings. > > The series was successfully tested on a vanilla v5.8-rc3 kernel and > Intel e1000e PCIe NIC. The same for platform devices like SATA and USB. > > For reference, previous versions are listed below: > V1: https://lkml.org/lkml/2018/10/15/373 > V2: https://lkml.org/lkml/2019/7/11/426 > V3: https://lkml.org/lkml/2020/7/2/1114 > 1) After enabling SMMU on Armada 8040, and ARM_SMMU_DISABLE_BYPASS_BY_DEFAUL=y by default in kernel since 954a03be033c7cef80ddc232e7cbdb17df735663, internal eMMC is prevented from being initialised (as there is no iommus property for ap_sdhci0) Disabling "Disable bypass by default" make it work, but the patch highly suggest doing it properly. I wasn't able to find correct path for ap_sdhci for iommus in any publicly available documentation, would be highly appreciated addressed properly, thank you! 2) Second issue I got (btw I have ClearFog GT 8k armada-8040 based board) is mpci ath10k card. It is found, it is enumerated, it is visible in lspci, but it fails to be initialised. Here is the log: [1.743754] armada8k-pcie f260.pcie: host bridge /cp0/pcie@f260 ranges: [1.751116] armada8k-pcie f260.pcie: MEM 0x00f600..0x00f6ef -> 0x00f600 [1.964690] armada8k-pcie f260.pcie: Link up [1.969379] armada8k-pcie f260.pcie: PCI host bridge to bus :00 [1.976026] pci_bus :00: root bus resource [bus 00-ff] [1.981537] pci_bus :00: root bus resource [mem 0xf600-0xf6ef] [1.988462] pci :00:00.0: [11ab:0110] type 01 class 0x060400 [1.994504] pci :00:00.0: reg 0x10: [mem 0x-0x000f] [2.000843] pci :00:00.0: supports D1 D2 [2.005132] pci :00:00.0: PME# supported from D0 D1 D3hot [2.011853] pci :01:00.0: [168c:003c] type 00 class 0x028000 [2.018001] pci :01:00.0: reg 0x10: [mem 0x-0x001f 64bit] [2.025002] pci :01:00.0: reg 0x30: [mem 0x-0x pref] [2.032111] pci :01:00.0: supports D1 D2 [2.049409] pci :00:00.0: BAR 14: assigned [mem 0xf600-0xf61f] [2.056322] pci :00:00.0: BAR 0: assigned [mem 0xf620-0xf62f] [2.063142] pci :00:00.0: BAR 15: assigned [mem 0xf630-0xf63f pref] [2.070484] pci :01:00.0: BAR 0: assigned [mem 0xf600-0xf61f 64bit] [2.077880] pci :01:00.0: BAR 6: assigned [mem 0xf630-0xf630 pref] [2.085135] pci :00:00.0: PCI bridge to [bus 01-ff] [2.090384] pci :00:00.0: bridge window [mem 0xf600-0xf61f] [2.097202] pci :00:00.0: bridge window [mem 0xf630-0xf63f pref] [2.104539] pcieport :00:00.0: Adding to iommu group 4 [2.110232] pcieport :00:00.0: PME: Signaling with IRQ 38 [2.116141] pcieport :00:00.0: AER: enabled with IRQ 38 [8.131135] ath10k_pci :01:00.0: Adding to iommu group 4 [8.131874] ath10k_pci :01:00.0: enabling device ( -> 0002) [8.132203] ath10k_pci :01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0 up to that point the log is the same as without SMMU enabled, except "Adding to iommu group N" lines, and IRQ being 37 [8.221328] ath10k_pci :01:00.0: failed to poke copy engine: -16 [8.313362] ath10k_pci :01:00.0: failed to poke copy engine: -16 [8.409373] ath10k_pci :01:00.0: failed to poke copy engine: -16 [8.553433] ath10k_pci :01:00.0: failed to poke copy engine: -16 [8.641370] ath10k_pci :01:00.0: failed to poke copy engine: -16 [8.737979] ath10k_pci :01:00.0: failed to poke copy engine: -16 [8.807356] ath10k_pci :01:00.0: Failed to get pcie state addr: -16 [8.814032] ath10k_pci :01:00.0: failed to setup init config: -16 [8.820605] ath10k_pci :01:00.0: could not power on hif bus (-16) [8.827111] ath10k_pci :01:00.0: could not probe fw (-16) Thank you! > v3 -> v4 > - call cfg_probe() impl hook a bit earlier which simplifies errata handling > - use hi_lo_readq_relaxed() and hi_lo_writeq_relaxed() for register accessors > - keep SMMU status disabled by default and enable where possible (DTS changes) > - commit logs improvements and other minor fixes > > Hanna Hawa (1): > iommu/arm-smmu: Workaround for Marvell Armada-AP806 SoC erratum >#582743 > > Marcin Wojtas (1): > arm64: dts: marvell: add SMMU support > > Tomasz Nowicki (2): > iommu/arm-smmu: Call configuration impl hook before consuming features > dt-bindings: arm-smmu: add compatible string for Marvell Armada-AP806 >SMMU-500 > > Documentation/arm64/silicon-errata.rst| 3 ++ >
