Re: d63e2e1f3df breaks sparc/T5-8
From: Bjorn Helgaas Date: Fri, 3 Apr 2015 10:45:26 -0500 > On Sun, Mar 29, 2015 at 11:32:50AM -0700, David Miller wrote: >> From: Bjorn Helgaas >> Date: Sun, 29 Mar 2015 08:30:40 -0500 >> >> > Help me understand the sparc64 situation: are you saying that BAR >> > addresses, i.e., MMIO transactions from a CPU or a peer-to-peer DMA can be >> > 64 bits, but a DMA to main memory can only be 32 bits? >> > >> > I assume this would work if we made dma_addr_t 64 bits on sparc64. What >> > would be the cost of doing that? >> >> The cost is 4 extra bytes in every datastructure, kernel wide, that >> stores DMA addresses. > > That much is fairly obvious. What I don't know is how much difference this > makes in the end. Networking drivers, and perhaps block drivers too, have a data structure for each entry in the send/receive rings for the device and these can be huge. And each ring entry stores one or more DMA addresses. Larger types mean more memory, but also more capacity cache misses in critical code paths. I'm really sorry if this isn't painfully obvious to you. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Sun, Mar 29, 2015 at 11:32:50AM -0700, David Miller wrote: > From: Bjorn Helgaas > Date: Sun, 29 Mar 2015 08:30:40 -0500 > > > Help me understand the sparc64 situation: are you saying that BAR > > addresses, i.e., MMIO transactions from a CPU or a peer-to-peer DMA can be > > 64 bits, but a DMA to main memory can only be 32 bits? > > > > I assume this would work if we made dma_addr_t 64 bits on sparc64. What > > would be the cost of doing that? > > The cost is 4 extra bytes in every datastructure, kernel wide, that > stores DMA addresses. That much is fairly obvious. What I don't know is how much difference this makes in the end. > Don't use DMA addresses for PCI addresses. They are absolutely not > the same, especially when an IOMMU is always present because in that > case all DMA addresses are virtual and exist in a different realm > and set of constraints/restrictions. I'm still trying to figure out a clear description of how a DMA address is different from a PCI address. If you capture a transaction with a PCI analyzer, I don't think you can tell a DMA address from a PCI address unless you know how bridge windows are programmed. Even then, I'm not sure you can tell a CPU-generated PCI address from a DMA address in a device-generated peer-to-peer transaction. Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/31/15 4:38 PM, Yinghai Lu wrote: On Tue, Mar 31, 2015 at 3:29 PM, David Ahern wrote: Please check attached three patches on top of current linus tree. new log. Same as before -- PCI_DEBUG, ignore_loglevel ofpci_debug=1 Good, it is clean now. Yes. You can add a Tested-by to all 3. These are all 4.0 material I hope with a mark for 3.19 stable? Thanks, Yinghai. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Tue, Mar 31, 2015 at 3:29 PM, David Ahern wrote: >> >> Please check attached three patches on top of current linus tree. >> > > new log. Same as before -- PCI_DEBUG, ignore_loglevel ofpci_debug=1 > Good, it is clean now. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Tue, Mar 31, 2015 at 10:04 AM, David Ahern wrote: >> Clear out the old and apply these new ones. > > > I take DaveM's response to mean the patches (3rd one?) needs another > version. > > I will be on PTO Wed-Fri with limited access through Sunday. If you have > something to try out later today I can do that; else it needs to wait until > next week. Given the likelihood that Linus will release 4.0 this weekend > that means both 3.19 and 4.0 will be broken for these systems. Please check attached three patches on top of current linus tree. Thanks Yinghai Subject: [RFC PATCH v2] PCI: Introduce pci_bus_addr_t David Ahern found commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows") broke sparc/T5-8. In the boot log, there is pci :06:00.0: reg 0x184: can't handle BAR above 4GB (bus address 0x110204000) but that only could happen when dma_addr_t is 32-bit. According to David Miller, all DMA occurs behind an IOMMU and these IOMMUs only support 32-bit addressing, therefore dma_addr_t is 32-bit on sparc64. Let's introduce pci_bus_addr_t instead of using dma_addr_t, and pci_bus_addr_t will be 64-bit on 64-bit platform or X86 PAE kernel. Fixes: commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows") Fixes: commit 23b13bc76f35 ("PCI: Fail safely if we can't handle BARs larger than 4GB") Link: http://lkml.kernel.org/r/cae9fiqu1gjy1lyrxs+ma5lcteee4xmtjrg0axj9k_tsu+m9...@mail.gmail.com Reported-by: David Ahern Signed-off-by: Yinghai Lu --- -v2: use PCI_BUS_ADDR in drivers/pci/bus.c put config option in pci/Kconfig: as David Miller said: PCI addresses being 64-bit or not is an attribute of the PCI controller and the geography of the bridges behind it, not the cpu architecture. --- drivers/pci/Kconfig |4 drivers/pci/bus.c | 10 +- drivers/pci/probe.c | 12 ++-- include/linux/pci.h | 12 +--- 4 files changed, 24 insertions(+), 14 deletions(-) Index: linux-2.6/drivers/pci/bus.c === --- linux-2.6.orig/drivers/pci/bus.c +++ linux-2.6/drivers/pci/bus.c @@ -92,11 +92,11 @@ void pci_bus_remove_resources(struct pci } static struct pci_bus_region pci_32_bit = {0, 0xULL}; -#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT +#ifdef CONFIG_PCI_BUS_ADDR_T_64BIT static struct pci_bus_region pci_64_bit = {0, -(dma_addr_t) 0xULL}; -static struct pci_bus_region pci_high = {(dma_addr_t) 0x1ULL, -(dma_addr_t) 0xULL}; +(pci_bus_addr_t) 0xULL}; +static struct pci_bus_region pci_high = {(pci_bus_addr_t) 0x1ULL, +(pci_bus_addr_t) 0xULL}; #endif /* @@ -200,7 +200,7 @@ int pci_bus_alloc_resource(struct pci_bu resource_size_t), void *alignf_data) { -#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT +#ifdef CONFIG_PCI_BUS_ADDR_T_64BIT int rc; if (res->flags & IORESOURCE_MEM_64) { Index: linux-2.6/drivers/pci/probe.c === --- linux-2.6.orig/drivers/pci/probe.c +++ linux-2.6/drivers/pci/probe.c @@ -254,8 +254,8 @@ int __pci_read_base(struct pci_dev *dev, } if (res->flags & IORESOURCE_MEM_64) { - if ((sizeof(dma_addr_t) < 8 || sizeof(resource_size_t) < 8) && - sz64 > 0x1ULL) { + if ((sizeof(pci_bus_addr_t) < 8 || sizeof(resource_size_t) < 8) + && sz64 > 0x1ULL) { res->flags |= IORESOURCE_UNSET | IORESOURCE_DISABLED; res->start = 0; res->end = 0; @@ -264,7 +264,7 @@ int __pci_read_base(struct pci_dev *dev, goto out; } - if ((sizeof(dma_addr_t) < 8) && l) { + if ((sizeof(pci_bus_addr_t) < 8) && l) { /* Above 32-bit boundary; try to reallocate */ res->flags |= IORESOURCE_UNSET; res->start = 0; @@ -399,7 +399,7 @@ static void pci_read_bridge_mmio_pref(st struct pci_dev *dev = child->self; u16 mem_base_lo, mem_limit_lo; u64 base64, limit64; - dma_addr_t base, limit; + pci_bus_addr_t base, limit; struct pci_bus_region region; struct resource *res; @@ -426,8 +426,8 @@ static void pci_read_bridge_mmio_pref(st } } - base = (dma_addr_t) base64; - limit = (dma_addr_t) limit64; + base = (pci_bus_addr_t) base64; + limit = (pci_bus_addr_t) limit64; if (base != base64) { dev_err(&dev->dev, "can't handle bridge window above 4GB (bus address %#010llx)\n", Index: linux-2.6/include/linux/pci.h === --- linux-2.6.orig/include/linux/pci.h +++ linux-2.6/include/linux/pci.h @@ -576,9 +576,15 @@ int raw_pci_read(unsigned int domain, un int raw_pci_write(unsigned int domain, unsigned int bus, unsigned int devfn, int reg, int len, u32 val); +#ifdef CONFIG_PCI_BUS_ADDR_T_64BIT +typedef u64 pci_bus_addr_t; +#else +typedef u32 pci_bus_addr_t; +#endif + struct pci_bus_region { - dma_addr_t start; - dma_addr_t end; + pci_bus_addr_t start; + pci_bus_add
Re: d63e2e1f3df breaks sparc/T5-8
On Tue, Mar 31, 2015 at 11:19 AM, David Miller wrote: > I totally disagree, I think your test is too stringent and should be > significantly relaxed if not removed entirely. ok, will produce one version that only verify first criteria. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
From: Yinghai Lu Date: Tue, 31 Mar 2015 11:16:11 -0700 > On Tue, Mar 31, 2015 at 8:06 AM, David Miller wrote: >> Your patch only allows the condition behind resources that have 64-bits >> of significance, but that is not what the document above says about >> when this situation is allowed. >> >> Please implement the check either exactly as stated in the errata >> document, or more loosely if that is not possible, rather than more >> strictly than allowed. >> >> Your overly strict and restrictive checks are what got us into this >> predicament in the first place. :-( > > From that errata: > --- > Here are criteria that are sufficient to guarantee correctness for a > given candidate BAR: > > The entire path from the host to the adapter is over PCI Express. > > No conventional PCI or PCI-X devices do peer-t o-peer reads to the > range mapped by the BAR. > > The PCI Express Host Bridge does no byte merging. (This is believed to > be true on most > platforms.) > > Any locations with read side-effects are never the target of Memory > Reads with the TH bit Set. > See Section 2.2.5 > --- > > We can verify first one that we have all pcie device all the way to > the hostbridge. > > But we can not verify or guarantee other three. Lack of peer-to-peer reads we can assume, the byte merging we can add as a per-controller attribute in the future if it turns out there are some that do and it matters (I doubt this will ever be necessary) and the side-effect issue is outside the scope of the PCI layer. > System should get better about the constraints with system design. > So if it would assign 64bit (and above 4G) mmio to those non-pref 64bit BAR, > that means it already make sure system design will follow those criteria. > and then we can safely set pref bit in resource flags. I totally disagree, I think your test is too stringent and should be significantly relaxed if not removed entirely.
Re: d63e2e1f3df breaks sparc/T5-8
On Tue, Mar 31, 2015 at 8:06 AM, David Miller wrote: > Your patch only allows the condition behind resources that have 64-bits > of significance, but that is not what the document above says about > when this situation is allowed. > > Please implement the check either exactly as stated in the errata > document, or more loosely if that is not possible, rather than more > strictly than allowed. > > Your overly strict and restrictive checks are what got us into this > predicament in the first place. :-( >From that errata: --- Here are criteria that are sufficient to guarantee correctness for a given candidate BAR: The entire path from the host to the adapter is over PCI Express. No conventional PCI or PCI-X devices do peer-t o-peer reads to the range mapped by the BAR. The PCI Express Host Bridge does no byte merging. (This is believed to be true on most platforms.) Any locations with read side-effects are never the target of Memory Reads with the TH bit Set. See Section 2.2.5 --- We can verify first one that we have all pcie device all the way to the hostbridge. But we can not verify or guarantee other three. System should get better about the constraints with system design. So if it would assign 64bit (and above 4G) mmio to those non-pref 64bit BAR, that means it already make sure system design will follow those criteria. and then we can safely set pref bit in resource flags. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/31/15 10:53 AM, Yinghai Lu wrote: On Mon, Mar 30, 2015 at 9:10 PM, David Ahern wrote: On 3/30/15 7:06 PM, Yinghai Lu wrote: To make sure I am on the same page: these are a new round of patches? clear out the old, apply these? Clear out the old and apply these new ones. I take DaveM's response to mean the patches (3rd one?) needs another version. I will be on PTO Wed-Fri with limited access through Sunday. If you have something to try out later today I can do that; else it needs to wait until next week. Given the likelihood that Linus will release 4.0 this weekend that means both 3.19 and 4.0 will be broken for these systems. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Mon, Mar 30, 2015 at 9:10 PM, David Ahern wrote: > On 3/30/15 7:06 PM, Yinghai Lu wrote: > To make sure I am on the same page: these are a new round of patches? clear > out the old, apply these? Clear out the old and apply these new ones. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
From: Yinghai Lu Date: Mon, 30 Mar 2015 18:06:02 -0700 > ok, that is really non-pref mmio 64bit. > We can workaround the problem by honoring firmware setting, according > to > https://www.pcisig.com/specifications/pciexpress/base2/PCIe_Base_r2.1_Errata_08Jun10.pdf > page 13 > > Please check attached updated patches that should fix the regression > and kill those "no compatible window" warnings. Your patch only allows the condition behind resources that have 64-bits of significance, but that is not what the document above says about when this situation is allowed. Please implement the check either exactly as stated in the errata document, or more loosely if that is not possible, rather than more strictly than allowed. Your overly strict and restrictive checks are what got us into this predicament in the first place. :-( -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/30/15 7:06 PM, Yinghai Lu wrote: ok, that is really non-pref mmio 64bit. We can workaround the problem by honoring firmware setting, according to https://www.pcisig.com/specifications/pciexpress/base2/PCIe_Base_r2.1_Errata_08Jun10.pdf page 13 Please check attached updated patches that should fix the regression and kill those "no compatible window" warnings. To make sure I am on the same page: these are a new round of patches? clear out the old, apply these? David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Mon, Mar 30, 2015 at 3:54 PM, David Ahern wrote: > On 3/29/15 2:07 PM, Yinghai Lu wrote: >> >> [ 286.647560] PCI: scan_bus[/pci@300/pci@1/pci@0/pci@6] bus no 8 >> [ 286.921232] PCI: Claiming :00:01.0: Resource 15: >> 8001..8004afff [220c] >> [ 287.229190] PCI: Claiming :01:00.0: Resource 15: >> 8001..8004afff [220c] >> [ 287.533428] PCI: Claiming :02:04.0: Resource 15: >> 8001..80012fff [220c] >> [ 288.149831] PCI: Claiming :03:00.0: Resource 15: >> 8001..80012fff [220c] >> [ 288.252466] PCI: Claiming :04:06.0: Resource 14: >> 8001..80010fff [220c] >> [ 288.867196] PCI: Claiming :05:00.0: Resource 0: >> 8001..80011fff [204] >> [ 288.968221] pci :05:00.0: can't claim BAR 0 [mem >> 0x8001-0x80011fff]: no compatible bridge window >> >> the bridge resource has IORESOURCE_PREFETCH, but the device doesn't have >> that. > > # lspci -vvxxx -s :05:00.0 > :05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host > Controller (rev 03) (prog-if 30 [XHCI]) > Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- > Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > SERR- Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin A routed to IRQ 0004 > Region 0: Memory at 1 (64-bit, non-prefetchable) [size=8K] ok, that is really non-pref mmio 64bit. We can workaround the problem by honoring firmware setting, according to https://www.pcisig.com/specifications/pciexpress/base2/PCIe_Base_r2.1_Errata_08Jun10.pdf page 13 Please check attached updated patches that should fix the regression and kill those "no compatible window" warnings. Thanks Yinghai Subject: [RFC PATCH v2] PCI: Introduce pci_bus_addr_t David Ahern found commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows") broke sparc/T5-8. In the boot log, there is pci :06:00.0: reg 0x184: can't handle BAR above 4GB (bus address 0x110204000) but that only could happen when dma_addr_t is 32-bit. According to David Miller, all DMA occurs behind an IOMMU and these IOMMUs only support 32-bit addressing, therefore dma_addr_t is 32-bit on sparc64. Let's introduce pci_bus_addr_t instead of using dma_addr_t, and pci_bus_addr_t will be 64-bit on 64-bit platform or X86 PAE kernel. Fixes: commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows") Fixes: commit 23b13bc76f35 ("PCI: Fail safely if we can't handle BARs larger than 4GB") Link: http://lkml.kernel.org/r/cae9fiqu1gjy1lyrxs+ma5lcteee4xmtjrg0axj9k_tsu+m9...@mail.gmail.com Reported-by: David Ahern Signed-off-by: Yinghai Lu --- -v2: use PCI_BUS_ADDR in drivers/pci/bus.c put config option in pci/Kconfig: as David Miller said: PCI addresses being 64-bit or not is an attribute of the PCI controller and the geography of the bridges behind it, not the cpu architecture. --- drivers/pci/Kconfig |4 drivers/pci/bus.c | 10 +- drivers/pci/probe.c | 12 ++-- include/linux/pci.h | 12 +--- 4 files changed, 24 insertions(+), 14 deletions(-) Index: linux-2.6/drivers/pci/bus.c === --- linux-2.6.orig/drivers/pci/bus.c +++ linux-2.6/drivers/pci/bus.c @@ -92,11 +92,11 @@ void pci_bus_remove_resources(struct pci } static struct pci_bus_region pci_32_bit = {0, 0xULL}; -#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT +#ifdef CONFIG_PCI_BUS_ADDR_T_64BIT static struct pci_bus_region pci_64_bit = {0, -(dma_addr_t) 0xULL}; -static struct pci_bus_region pci_high = {(dma_addr_t) 0x1ULL, -(dma_addr_t) 0xULL}; +(pci_bus_addr_t) 0xULL}; +static struct pci_bus_region pci_high = {(pci_bus_addr_t) 0x1ULL, +(pci_bus_addr_t) 0xULL}; #endif /* @@ -200,7 +200,7 @@ int pci_bus_alloc_resource(struct pci_bu resource_size_t), void *alignf_data) { -#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT +#ifdef CONFIG_PCI_BUS_ADDR_T_64BIT int rc; if (res->flags & IORESOURCE_MEM_64) { Index: linux-2.6/drivers/pci/probe.c === --- linux-2.6.orig/drivers/pci/probe.c +++ linux-2.6/drivers/pci/probe.c @@ -254,8 +254,8 @@ int __pci_read_base(struct pci_dev *dev, } if (res->flags & IORESOURCE_MEM_64) { - if ((sizeof(dma_addr_t) < 8 || sizeof(resource_size_t) < 8) && - sz64 > 0x1ULL) { + if ((sizeof(pci_bus_addr_t) < 8 || sizeof(resource_size_t) < 8) + && sz64 > 0x1ULL) { res->flags |= IORESOURCE_UNSET | IORESOURCE_DISABLED; res->start = 0; res->end = 0; @@ -264,7 +264,7 @@ int __pci_read_base(struct pci_dev *dev, goto out; } - if ((sizeof(dma_addr_t) < 8) && l)
Re: d63e2e1f3df breaks sparc/T5-8
On 3/29/15 2:07 PM, Yinghai Lu wrote: [ 286.647560] PCI: scan_bus[/pci@300/pci@1/pci@0/pci@6] bus no 8 [ 286.921232] PCI: Claiming :00:01.0: Resource 15: 8001..8004afff [220c] [ 287.229190] PCI: Claiming :01:00.0: Resource 15: 8001..8004afff [220c] [ 287.533428] PCI: Claiming :02:04.0: Resource 15: 8001..80012fff [220c] [ 288.149831] PCI: Claiming :03:00.0: Resource 15: 8001..80012fff [220c] [ 288.252466] PCI: Claiming :04:06.0: Resource 14: 8001..80010fff [220c] [ 288.867196] PCI: Claiming :05:00.0: Resource 0: 8001..80011fff [204] [ 288.968221] pci :05:00.0: can't claim BAR 0 [mem 0x8001-0x80011fff]: no compatible bridge window the bridge resource has IORESOURCE_PREFETCH, but the device doesn't have that. So pci_claim_resource can not find parent resource for device resource: we can not put non pref mem under pref mem. Can you send out result from ? lspci -vvxxx -s :05:00.0 # lspci -vvxxx -s :05:00.0 :05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 0004 Region 0: Memory at 1 (64-bit, non-prefetchable) [size=8K] Region 2: [virtual] Memory at 8000 (32-bit, non-prefetchable) Region 3: [virtual] Memory at 8000 (32-bit, non-prefetchable) Region 4: [virtual] Memory at 8000 (32-bit, non-prefetchable) Region 5: [virtual] Memory at 8000 (32-bit, non-prefetchable) [virtual] Expansion ROM at 8000 [disabled] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: Data: Capabilities: [90] MSI-X: Enable+ Count=8 Masked- Vector table: BAR=0 offset=1000 PBA: BAR=0 offset=1080 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [150 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: xhci_hcd 00: 12 19 14 00 06 04 10 00 03 30 03 0c 10 00 00 00 10: 04 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 50 00 00 00 00 00 00 00 00 01 00 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 01 70 c3 c9 08 00 00 00 00 00 00 00 00 00 00 00 60:
Re: d63e2e1f3df breaks sparc/T5-8
On Sun, Mar 29, 2015 at 7:47 AM, David Ahern wrote: > On 3/28/15 2:24 PM, Yinghai Lu wrote: >> >> Can you append "ofpci_debug=1" in boot command line? > > > here you go. [ 286.647560] PCI: scan_bus[/pci@300/pci@1/pci@0/pci@6] bus no 8 [ 286.921232] PCI: Claiming :00:01.0: Resource 15: 8001..8004afff [220c] [ 287.229190] PCI: Claiming :01:00.0: Resource 15: 8001..8004afff [220c] [ 287.533428] PCI: Claiming :02:04.0: Resource 15: 8001..80012fff [220c] [ 288.149831] PCI: Claiming :03:00.0: Resource 15: 8001..80012fff [220c] [ 288.252466] PCI: Claiming :04:06.0: Resource 14: 8001..80010fff [220c] [ 288.867196] PCI: Claiming :05:00.0: Resource 0: 8001..80011fff [204] [ 288.968221] pci :05:00.0: can't claim BAR 0 [mem 0x8001-0x80011fff]: no compatible bridge window the bridge resource has IORESOURCE_PREFETCH, but the device doesn't have that. So pci_claim_resource can not find parent resource for device resource: we can not put non pref mem under pref mem. Can you send out result from ? lspci -vvxxx -s :05:00.0 If it does have pref, could be of device layer does not pass the pref flag properly via pci_parse_of_addrs/of_get_property(node, "assigned-addresses". or the addr0 from "ranges" and "assigned-address" has different definition. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
From: Bjorn Helgaas Date: Sun, 29 Mar 2015 08:30:40 -0500 > Help me understand the sparc64 situation: are you saying that BAR > addresses, i.e., MMIO transactions from a CPU or a peer-to-peer DMA can be > 64 bits, but a DMA to main memory can only be 32 bits? > > I assume this would work if we made dma_addr_t 64 bits on sparc64. What > would be the cost of doing that? The cost is 4 extra bytes in every datastructure, kernel wide, that stores DMA addresses. This choice was very intentional, and well thought out. Don't use DMA addresses for PCI addresses. They are absolutely not the same, especially when an IOMMU is always present because in that case all DMA addresses are virtual and exist in a different realm and set of constraints/restrictions. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Sat, Mar 28, 2015 at 7:48 AM, David Ahern wrote: > On 3/27/15 11:26 PM, Yinghai Lu wrote: >> >> >> looks like the offset for mem64 is not right. >> >> Please try attached v2. still have problem. [139295.760918] pci_sun4v f02dbcfc: PCI host bridge to bus :00 [139295.831448] pci_bus :00: root bus resource [io 0x8040-0x80400fff] (bus address [0x-0xfff]) [139295.958623] pci_bus :00: root bus resource [mem 0x8000-0x80007eff] (bus address [0x-0x7eff]) [139296.091392] pci_bus :00: root bus resource [mem 0x8001-0x8007] (bus address [0x1-0x7]) [139296.225779] pci_bus :00: root bus resource [bus 00-77] ... [139298.092814] pci :05:00.0: can't claim BAR 0 [mem 0x8001-0x80011fff]: no compatible bridge window [139298.215949] pci :06:00.0: can't claim BAR 7 [mem 0x800110204000-0x800110303fff 64bit]: no compatible bridge window [139298.346479] pci :06:00.0: can't claim BAR 10 [mem 0x800110304000-0x800110403fff 64bit]: no compatible bridge window [139298.476953] pci :06:00.1: can't claim BAR 7 [mem 0x800110408000-0x800110507fff 64bit]: no compatible bridge window [139298.607105] pci :06:00.1: can't claim BAR 10 [mem 0x80011080-0x8001108f 64bit]: no compatible bridge window [139298.739341] pci :07:00.0: can't claim BAR 1 [mem 0x80012000-0x80012000]: no compatible bridge window [139298.862467] pci :07:00.0: can't claim BAR 3 [mem 0x80012004-0x80012007]: no compatible bridge window Can you append "ofpci_debug=1" in boot command line? So we can check if the bridge get 64 bus from the of layer. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Sat, Mar 28, 2015 at 11:16 AM, David Miller wrote: > PCI addresses being 64-bit or not is an attribute of the PCI > controller and the geography of the bridges behind it, not the > cpu architecture. Good point. We should add one choice in pci subsystem Kconfig. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
From: Yinghai Lu Date: Fri, 27 Mar 2015 16:57:23 -0700 > On Fri, Mar 27, 2015 at 2:01 PM, Yinghai Lu wrote: >> On Thu, Mar 26, 2015 at 4:27 PM, David Ahern wrote: > >> Also please make sure your config have >> >> CONFIG_PCI_DEBUG=y >> >> and capture serial console with "debug ignore_loglevel", so we check if >> pci :00:01.0 really have resource assigned. > > Please check attached patch and send out boot log with above config. PCI addresses being 64-bit or not is an attribute of the PCI controller and the geography of the bridges behind it, not the cpu architecture. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/27/15 11:26 PM, Yinghai Lu wrote: On Fri, Mar 27, 2015 at 8:45 PM, David Ahern wrote: On 3/27/15 9:19 PM, Yinghai Lu wrote: Good. But we still have annoying warning about "no compatible window". Please try attached patch that support 64bit pci mem space for sparc. BTW, looks like you still do not have CONFIG_PCI_DEBUG=y in your .config. otherwise we should more verbose print out. Or did you have DYNAMIC_DEBUG enabled ? If it is that case, can you remove that? done. attached. ok, we are almost there. [99510.956337] pci_sun4v f02dbcfc: PCI host bridge to bus :00 [99511.025812] pci_bus :00: root bus resource [io 0x8040-0x80400fff] (bus address [0x-0xfff]) [99511.151039] pci_bus :00: root bus resource [mem 0x8000-0x80007eff] (bus address [0x-0x7eff]) [99511.282693] pci_bus :00: root bus resource [mem 0x8001-0x8007] (bus address [0x-0x6]) [99511.414878] pci_bus :00: root bus resource [bus 00-77] looks like the offset for mem64 is not right. Please try attached v2. sshlab root@ca-qasparc24 dmesg [0.00] PROMLIB: Sun IEEE Boot Prom 'OBP 4.36.2 2014/10/24 08:15' [0.00] PROMLIB: Root node compatible: sun4v [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.0.0-rc5+ (root@ca-qasparc24) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4.0.9) (GCC) ) #1 SMP Sat Mar 28 10:35:16 EDT 2015 [0.00] debug: skip boot console de-registration. [0.00] debug: ignoring loglevel setting. [0.00] bootconsole [earlyprom0] enabled [0.00] ARCH: SUN4V [0.00] Ethernet address: 00:10:e0:35:15:f6 [0.00] MM: PAGE_OFFSET is 0xfff8 (max_phys_bits == 47) [0.00] MM: VMALLOC [0x0001 --> 0x0006] [0.00] MM: VMEMMAP [0x0006 --> 0x000c] [0.00] Kernel: Using 3 locked TLB entries for main kernel image. [0.00] Remapping the kernel... done. [0.00] OF stdout device is: /virtual-devices@100/console@1 [0.00] PROM: Built device tree with 1478490 bytes of memory. [0.00] MDESC: Size is 774368 bytes. [0.00] PLATFORM: banner-name [SPARC T5-8] [0.00] PLATFORM: name [ORCL,SPARC-T5-8] [0.00] PLATFORM: hostid [863515f6] [0.00] PLATFORM: serial# [003515f6] [0.00] PLATFORM: stick-frequency [3b9aca00] [0.00] PLATFORM: mac-address [10e03515f6] [0.00] PLATFORM: watchdog-resolution [1000 ms] [0.00] PLATFORM: watchdog-max-timeout [3153600 ms] [0.00] PLATFORM: max-cpus [1024] [0.00] Allocated 81920 bytes for kernel page tables. [0.00] Zone ranges: [0.00] DMA [mem 0x3040-0x] [0.00] Normal [mem 0x-0x383fffd11fff] [0.00] Movable zone start for each node [0.00] Early memory node ranges [0.00] node 0: [mem 0x3040-0x003fddee] [0.00] node 0: [mem 0x003fddef6000-0x003fddef7fff] [0.00] node 1: [mem 0x0800-0x083f] [0.00] node 2: [mem 0x1000-0x103f] [0.00] node 3: [mem 0x1800-0x183f] [0.00] node 4: [mem 0x2000-0x203f] [0.00] node 5: [mem 0x2800-0x283f] [0.00] node 6: [mem 0x3000-0x303f] [0.00] node 7: [mem 0x3800-0x383fffcddfff] [0.00] node 7: [mem 0x383fffcee000-0x383fffd11fff] [0.00] Initmem setup node 0 [mem 0x3040-0x003fddef7fff] [0.00] On node 0 totalpages: 33385849 [0.00] Normal zone: 293431 pages used for memmap [0.00] Normal zone: 33385849 pages, LIFO batch:15 [0.00] Initmem setup node 1 [mem 0x0800-0x083f] [0.00] On node 1 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 2 [mem 0x1000-0x103f] [0.00] On node 2 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 3 [mem 0x1800-0x183f] [0.00] On node 3 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 4 [mem 0x2000-0x203f] [0.00] On node 4 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15
Re: d63e2e1f3df breaks sparc/T5-8
> >> config ARM_THUMB > >> bool "Support Thumb user binaries" if !CPU_THUMBONLY > >> depends on CPU_ARM720T || CPU_ARM740T || CPU_ARM920T || CPU_ARM922T > >> || \ > >> Index: linux-2.6/arch/arm64/Kconfig > >> === > >> --- linux-2.6.orig/arch/arm64/Kconfig > >> +++ linux-2.6/arch/arm64/Kconfig > >> @@ -125,6 +125,9 @@ config HAVE_GENERIC_RCU_GUP > >> config ARCH_DMA_ADDR_T_64BIT > >> def_bool y > >> > >> +config ARCH_PCI_BUS_ADDR_T_64BIT > >> + def_bool y > >> + > > Use select > ... > > I was trying to keep the style to be consistent with old one in each Kconfig. > > So do we just need to make sure new added lines to use new style? Use new style select everywhere. That could maybe in the future trigger someone to update the different arch configs to use select for the remaining bits. > >> Index: linux-2.6/include/linux/types.h > >> === > >> --- linux-2.6.orig/include/linux/types.h > >> +++ linux-2.6/include/linux/types.h > >> @@ -146,6 +146,13 @@ typedef u64 dma_addr_t; > >> typedef u32 dma_addr_t; > >> #endif /* dma_addr_t */ > >> > >> +/* A pci_bus_addr_t can hold pci bus address for the platform */ > >> +#ifdef CONFIG_ARCH_PCI_BUS_ADDR_T_64BIT > >> +typedef u64 pci_bus_addr_t; > >> +#else > >> +typedef u32 pci_bus_addr_t; > >> +#endif /* pci_bus_addr_t */ > >> + > > Looking att all the Kconfig logic I get the impression that > > the type of pci_bus_addr_t equals unsigned long. > > If this is correct then you can skip that and things > > gets much simpler. > > For x86 32bit with PAE support, we could use 64bit for bus address. > but unsigned long is 32bit. Missed that part - OK. Sam -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Fri, Mar 27, 2015 at 8:45 PM, David Ahern wrote: > On 3/27/15 9:19 PM, Yinghai Lu wrote: >> >> Good. But we still have annoying warning about "no compatible window". >> >> Please try attached patch that support 64bit pci mem space for sparc. >> >> BTW, looks like you still do not have CONFIG_PCI_DEBUG=y in your >> .config. >> otherwise we should more verbose print out. >> Or did you have DYNAMIC_DEBUG enabled ? >> If it is that case, can you remove that? >> > > done. attached. > ok, we are almost there. [99510.956337] pci_sun4v f02dbcfc: PCI host bridge to bus :00 [99511.025812] pci_bus :00: root bus resource [io 0x8040-0x80400fff] (bus address [0x-0xfff]) [99511.151039] pci_bus :00: root bus resource [mem 0x8000-0x80007eff] (bus address [0x-0x7eff]) [99511.282693] pci_bus :00: root bus resource [mem 0x8001-0x8007] (bus address [0x-0x6]) [99511.414878] pci_bus :00: root bus resource [bus 00-77] looks like the offset for mem64 is not right. Please try attached v2. Thanks Yinghai Subject: [RFC PATCH v2] sparc/PCI: Add mem64 resource parsing for root bus Found no compatible bridge window warning in boot log from T5-8. pci :00:01.0: can't claim BAR 15 [mem 0x1-0x4afff pref]: no compatible bridge window and root bus only report io and mem32. pci_sun4v f02dbcfc: PCI host bridge to bus :00 pci_bus :00: root bus resource [io 0x8040-0x80400fff] (bus address [0x-0xfff]) pci_bus :00: root bus resource [mem 0x8000-0x80007eff] (bus address [0x-0x7eff]) pci_bus :00: root bus resource [bus 00-77] Add mem64 handling in pci_common for sparc, so we can have 64bit resource registered for bus at first. Signed-off-by: Yinghai Lu --- -v2: mem64_space should use mem_space.start as offset. --- arch/sparc/kernel/pci.c|3 +++ arch/sparc/kernel/pci_common.c | 15 +-- arch/sparc/kernel/pci_impl.h |1 + 3 files changed, 17 insertions(+), 2 deletions(-) Index: linux-2.6/arch/sparc/kernel/pci.c === --- linux-2.6.orig/arch/sparc/kernel/pci.c +++ linux-2.6/arch/sparc/kernel/pci.c @@ -663,6 +663,9 @@ struct pci_bus *pci_scan_one_pbm(struct pbm->io_space.start); pci_add_resource_offset(&resources, &pbm->mem_space, pbm->mem_space.start); + if (pbm->mem64_space.flags) + pci_add_resource_offset(&resources, &pbm->mem64_space, + pbm->mem_space.start); pbm->busn.start = pbm->pci_first_busno; pbm->busn.end = pbm->pci_last_busno; pbm->busn.flags = IORESOURCE_BUS; Index: linux-2.6/arch/sparc/kernel/pci_common.c === --- linux-2.6.orig/arch/sparc/kernel/pci_common.c +++ linux-2.6/arch/sparc/kernel/pci_common.c @@ -406,6 +406,7 @@ void pci_determine_mem_io_space(struct p } num_pbm_ranges = i / sizeof(*pbm_ranges); + memset(&pbm->mem64_space, 0, sizeof(struct resource)); for (i = 0; i < num_pbm_ranges; i++) { const struct linux_prom_pci_ranges *pr = &pbm_ranges[i]; @@ -451,7 +452,11 @@ void pci_determine_mem_io_space(struct p break; case 3: - /* XXX 64-bit MEM handling XXX */ + /* 64-bit MEM handling */ + pbm->mem64_space.start = a; + pbm->mem64_space.end = a + size - 1UL; + pbm->mem64_space.flags = IORESOURCE_MEM; + break; default: break; @@ -465,15 +470,21 @@ void pci_determine_mem_io_space(struct p prom_halt(); } - printk("%s: PCI IO[%llx] MEM[%llx]\n", + printk("%s: PCI IO[%llx] MEM[%llx]", pbm->name, pbm->io_space.start, pbm->mem_space.start); + if (pbm->mem64_space.flags) + printk(" MEM64[%llx]", + pbm->mem64_space.start); + printk("\n"); pbm->io_space.name = pbm->mem_space.name = pbm->name; request_resource(&ioport_resource, &pbm->io_space); request_resource(&iomem_resource, &pbm->mem_space); + if (pbm->mem64_space.flags) + request_resource(&iomem_resource, &pbm->mem64_space); pci_register_legacy_regions(&pbm->io_space, &pbm->mem_space); Index: linux-2.6/arch/sparc/kernel/pci_impl.h === --- linux-2.6.orig/arch/sparc/kernel/pci_impl.h +++ linux-2.6/arch/sparc/kernel/pci_impl.h @@ -97,6 +97,7 @@ struct pci_pbm_info { /* PBM I/O and Memory space resources. */ struct resource io_space; struct resource mem_space; + struct resource mem64_space; struct resource busn; /* Base of PCI Config space, can be per-PBM or shared. */
Re: d63e2e1f3df breaks sparc/T5-8
On 3/27/15 9:19 PM, Yinghai Lu wrote: Good. But we still have annoying warning about "no compatible window". Please try attached patch that support 64bit pci mem space for sparc. BTW, looks like you still do not have CONFIG_PCI_DEBUG=y in your .config. otherwise we should more verbose print out. Or did you have DYNAMIC_DEBUG enabled ? If it is that case, can you remove that? done. attached. sshlab root@ca-qasparc24 dmesg [0.00] PROMLIB: Sun IEEE Boot Prom 'OBP 4.36.2 2014/10/24 08:15' [0.00] PROMLIB: Root node compatible: sun4v [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.0.0-rc5+ (root@ca-qasparc24) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4.0.9) (GCC) ) #1 SMP Fri Mar 27 23:32:21 EDT 2015 [0.00] debug: skip boot console de-registration. [0.00] debug: ignoring loglevel setting. [0.00] bootconsole [earlyprom0] enabled [0.00] ARCH: SUN4V [0.00] Ethernet address: 00:10:e0:35:15:f6 [0.00] MM: PAGE_OFFSET is 0xfff8 (max_phys_bits == 47) [0.00] MM: VMALLOC [0x0001 --> 0x0006] [0.00] MM: VMEMMAP [0x0006 --> 0x000c] [0.00] Kernel: Using 3 locked TLB entries for main kernel image. [0.00] Remapping the kernel... done. [0.00] OF stdout device is: /virtual-devices@100/console@1 [0.00] PROM: Built device tree with 1478492 bytes of memory. [0.00] MDESC: Size is 774368 bytes. [0.00] PLATFORM: banner-name [SPARC T5-8] [0.00] PLATFORM: name [ORCL,SPARC-T5-8] [0.00] PLATFORM: hostid [863515f6] [0.00] PLATFORM: serial# [003515f6] [0.00] PLATFORM: stick-frequency [3b9aca00] [0.00] PLATFORM: mac-address [10e03515f6] [0.00] PLATFORM: watchdog-resolution [1000 ms] [0.00] PLATFORM: watchdog-max-timeout [3153600 ms] [0.00] PLATFORM: max-cpus [1024] [0.00] Allocated 81920 bytes for kernel page tables. [0.00] Zone ranges: [0.00] DMA [mem 0x3040-0x] [0.00] Normal [mem 0x-0x383fffd11fff] [0.00] Movable zone start for each node [0.00] Early memory node ranges [0.00] node 0: [mem 0x3040-0x003fddee] [0.00] node 0: [mem 0x003fddef6000-0x003fddef7fff] [0.00] node 1: [mem 0x0800-0x083f] [0.00] node 2: [mem 0x1000-0x103f] [0.00] node 3: [mem 0x1800-0x183f] [0.00] node 4: [mem 0x2000-0x203f] [0.00] node 5: [mem 0x2800-0x283f] [0.00] node 6: [mem 0x3000-0x303f] [0.00] node 7: [mem 0x3800-0x383fffcddfff] [0.00] node 7: [mem 0x383fffcee000-0x383fffd11fff] [0.00] Initmem setup node 0 [mem 0x3040-0x003fddef7fff] [0.00] On node 0 totalpages: 33385849 [0.00] Normal zone: 293431 pages used for memmap [0.00] Normal zone: 33385849 pages, LIFO batch:15 [0.00] Initmem setup node 1 [mem 0x0800-0x083f] [0.00] On node 1 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 2 [mem 0x1000-0x103f] [0.00] On node 2 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 3 [mem 0x1800-0x183f] [0.00] On node 3 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 4 [mem 0x2000-0x203f] [0.00] On node 4 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 5 [mem 0x2800-0x283f] [0.00] On node 5 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 6 [mem 0x3000-0x303f] [0.00] On node 6 totalpages: 33554432 [0.00] Normal zone: 294912 pages used for memmap [0.00] Normal zone: 33554432 pages, LIFO batch:15 [0.00] Initmem setup node 7 [mem 0x3800-0x383fffd11fff] [0.00] On node 7 totalpages: 33554049 [0.00] Normal zone: 294909 pages used for memmap [0.00] No
Re: d63e2e1f3df breaks sparc/T5-8
On Fri, Mar 27, 2015 at 8:22 PM, David Ahern wrote: > On 3/27/15 9:19 PM, Yinghai Lu wrote: >> >> Good. But we still have annoying warning about "no compatible window". >> >> Please try attached patch that support 64bit pci mem space for sparc. > > > in place of or on top of the previous patch? on top of the previous patch. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/27/15 9:19 PM, Yinghai Lu wrote: Good. But we still have annoying warning about "no compatible window". Please try attached patch that support 64bit pci mem space for sparc. in place of or on top of the previous patch? BTW, looks like you still do not have CONFIG_PCI_DEBUG=y in your .config. otherwise we should more verbose print out. # zcat /proc/config.gz | grep PCI_DEBUG CONFIG_PCI_DEBUG=y Or did you have DYNAMIC_DEBUG enabled ? # zcat /proc/config.gz | grep DYNAMIC CONFIG_NETCONSOLE_DYNAMIC=y CONFIG_SND_DYNAMIC_MINORS=y # CONFIG_USB_DYNAMIC_MINORS is not set CONFIG_DYNAMIC_DEBUG=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_DYNAMIC_FTRACE=y If it is that case, can you remove that? will do; wasn't aware of it before now. Let me know about the patch ordering -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Fri, Mar 27, 2015 at 5:36 PM, David Ahern wrote: >>> Also please make sure your config have CONFIG_PCI_DEBUG=y and capture serial console with "debug ignore_loglevel", so we check if pci :00:01.0 really have resource assigned. >>> >>> >>> Please check attached patch and send out boot log with above config. >> >> >> patched applied. PCI_DEBUG enabled. options added. log attached. This is >> with d63e2e1f3df reverted still. >> > > oops lied about that last one; I forgot to revert the patch this round. That > the system booted with it applied is progress. Good. But we still have annoying warning about "no compatible window". Please try attached patch that support 64bit pci mem space for sparc. BTW, looks like you still do not have CONFIG_PCI_DEBUG=y in your .config. otherwise we should more verbose print out. Or did you have DYNAMIC_DEBUG enabled ? If it is that case, can you remove that? Thanks Yinghai Subject: [RFC PATCH] sparc/PCI: Add mem64 resource parsing for root bus Found no compatible bridge window warning in boot log from T5-8. pci :00:01.0: can't claim BAR 15 [mem 0x1-0x4afff pref]: no compatible bridge window and root bus only report io and mem32. pci_sun4v f02dbcfc: PCI host bridge to bus :00 pci_bus :00: root bus resource [io 0x8040-0x80400fff] (bus address [0x-0xfff]) pci_bus :00: root bus resource [mem 0x8000-0x80007eff] (bus address [0x-0x7eff]) pci_bus :00: root bus resource [bus 00-77] Add mem64 handling in pci_common for sparc, so we can have 64bit resource registered for bus at first. Signed-off-by: Yinghai Lu --- arch/sparc/kernel/pci.c|3 +++ arch/sparc/kernel/pci_common.c | 15 +-- arch/sparc/kernel/pci_impl.h |1 + 3 files changed, 17 insertions(+), 2 deletions(-) Index: linux-2.6/arch/sparc/kernel/pci.c === --- linux-2.6.orig/arch/sparc/kernel/pci.c +++ linux-2.6/arch/sparc/kernel/pci.c @@ -663,6 +663,9 @@ struct pci_bus *pci_scan_one_pbm(struct pbm->io_space.start); pci_add_resource_offset(&resources, &pbm->mem_space, pbm->mem_space.start); + if (pbm->mem64_space.flags) + pci_add_resource_offset(&resources, &pbm->mem64_space, + pbm->mem64_space.start); pbm->busn.start = pbm->pci_first_busno; pbm->busn.end = pbm->pci_last_busno; pbm->busn.flags = IORESOURCE_BUS; Index: linux-2.6/arch/sparc/kernel/pci_common.c === --- linux-2.6.orig/arch/sparc/kernel/pci_common.c +++ linux-2.6/arch/sparc/kernel/pci_common.c @@ -406,6 +406,7 @@ void pci_determine_mem_io_space(struct p } num_pbm_ranges = i / sizeof(*pbm_ranges); + memset(&pbm->mem64_space, 0, sizeof(struct resource)); for (i = 0; i < num_pbm_ranges; i++) { const struct linux_prom_pci_ranges *pr = &pbm_ranges[i]; @@ -451,7 +452,11 @@ void pci_determine_mem_io_space(struct p break; case 3: - /* XXX 64-bit MEM handling XXX */ + /* 64-bit MEM handling */ + pbm->mem64_space.start = a; + pbm->mem64_space.end = a + size - 1UL; + pbm->mem64_space.flags = IORESOURCE_MEM; + break; default: break; @@ -465,15 +470,21 @@ void pci_determine_mem_io_space(struct p prom_halt(); } - printk("%s: PCI IO[%llx] MEM[%llx]\n", + printk("%s: PCI IO[%llx] MEM[%llx]", pbm->name, pbm->io_space.start, pbm->mem_space.start); + if (pbm->mem64_space.flags) + printk(" MEM64[%llx]", + pbm->mem64_space.start); + printk("\n"); pbm->io_space.name = pbm->mem_space.name = pbm->name; request_resource(&ioport_resource, &pbm->io_space); request_resource(&iomem_resource, &pbm->mem_space); + if (pbm->mem64_space.flags) + request_resource(&iomem_resource, &pbm->mem64_space); pci_register_legacy_regions(&pbm->io_space, &pbm->mem_space); Index: linux-2.6/arch/sparc/kernel/pci_impl.h === --- linux-2.6.orig/arch/sparc/kernel/pci_impl.h +++ linux-2.6/arch/sparc/kernel/pci_impl.h @@ -97,6 +97,7 @@ struct pci_pbm_info { /* PBM I/O and Memory space resources. */ struct resource io_space; struct resource mem_space; + struct resource mem64_space; struct resource busn; /* Base of PCI Config space, can be per-PBM or shared. */
Re: d63e2e1f3df breaks sparc/T5-8
On Fri, Mar 27, 2015 at 6:05 PM, Sam Ravnborg wrote: >> >> Index: linux-2.6/arch/alpha/Kconfig >> === >> --- linux-2.6.orig/arch/alpha/Kconfig >> +++ linux-2.6/arch/alpha/Kconfig >> @@ -66,6 +66,9 @@ config ZONE_DMA >> config ARCH_DMA_ADDR_T_64BIT >> def_bool y >> >> +config ARCH_PCI_BUS_ADDR_T_64BIT >> + def_bool y >> + > For alpha you define a new symbol with a default value. > >> config NEED_DMA_MAP_STATE >> def_bool y >> >> Index: linux-2.6/arch/arm/Kconfig >> === >> --- linux-2.6.orig/arch/arm/Kconfig >> +++ linux-2.6/arch/arm/Kconfig >> @@ -1779,6 +1779,7 @@ config XEN >> depends on !GENERIC_ATOMIC64 >> depends on MMU >> select ARCH_DMA_ADDR_T_64BIT >> + select ARCH_PCI_BUS_ADDR_T_64BIT >> select ARM_PSCI >> select SWIOTLB_XEN >> help > For arm you use select ... > > As the actual definition of the Kconfig symbol is already > present in mm/Kconfig please use select in all cases. > >> Index: linux-2.6/arch/arm/mm/Kconfig >> === >> --- linux-2.6.orig/arch/arm/mm/Kconfig >> +++ linux-2.6/arch/arm/mm/Kconfig >> @@ -630,6 +630,9 @@ config ARCH_PHYS_ADDR_T_64BIT >> config ARCH_DMA_ADDR_T_64BIT >> bool >> >> +config ARCH_PCI_BUS_ADDR_T_64BIT >> + bool >> + > Here it would have been nice with a comment that explains the use of ths > symbol. just duplicate the ARCH_DMA_ADDR_T_64BIT above > >> config ARM_THUMB >> bool "Support Thumb user binaries" if !CPU_THUMBONLY >> depends on CPU_ARM720T || CPU_ARM740T || CPU_ARM920T || CPU_ARM922T || >> \ >> Index: linux-2.6/arch/arm64/Kconfig >> === >> --- linux-2.6.orig/arch/arm64/Kconfig >> +++ linux-2.6/arch/arm64/Kconfig >> @@ -125,6 +125,9 @@ config HAVE_GENERIC_RCU_GUP >> config ARCH_DMA_ADDR_T_64BIT >> def_bool y >> >> +config ARCH_PCI_BUS_ADDR_T_64BIT >> + def_bool y >> + > Use select ... I was trying to keep the style to be consistent with old one in each Kconfig. So do we just need to make sure new added lines to use new style? > >> Index: linux-2.6/include/linux/types.h >> === >> --- linux-2.6.orig/include/linux/types.h >> +++ linux-2.6/include/linux/types.h >> @@ -146,6 +146,13 @@ typedef u64 dma_addr_t; >> typedef u32 dma_addr_t; >> #endif /* dma_addr_t */ >> >> +/* A pci_bus_addr_t can hold pci bus address for the platform */ >> +#ifdef CONFIG_ARCH_PCI_BUS_ADDR_T_64BIT >> +typedef u64 pci_bus_addr_t; >> +#else >> +typedef u32 pci_bus_addr_t; >> +#endif /* pci_bus_addr_t */ >> + > Looking att all the Kconfig logic I get the impression that > the type of pci_bus_addr_t equals unsigned long. > If this is correct then you can skip that and things > gets much simpler. For x86 32bit with PAE support, we could use 64bit for bus address. but unsigned long is 32bit. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
> > Index: linux-2.6/arch/alpha/Kconfig > === > --- linux-2.6.orig/arch/alpha/Kconfig > +++ linux-2.6/arch/alpha/Kconfig > @@ -66,6 +66,9 @@ config ZONE_DMA > config ARCH_DMA_ADDR_T_64BIT > def_bool y > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool y > + For alpha you define a new symbol with a default value. > config NEED_DMA_MAP_STATE > def_bool y > > Index: linux-2.6/arch/arm/Kconfig > === > --- linux-2.6.orig/arch/arm/Kconfig > +++ linux-2.6/arch/arm/Kconfig > @@ -1779,6 +1779,7 @@ config XEN > depends on !GENERIC_ATOMIC64 > depends on MMU > select ARCH_DMA_ADDR_T_64BIT > + select ARCH_PCI_BUS_ADDR_T_64BIT > select ARM_PSCI > select SWIOTLB_XEN > help For arm you use select ... As the actual definition of the Kconfig symbol is already present in mm/Kconfig please use select in all cases. > Index: linux-2.6/arch/arm/mm/Kconfig > === > --- linux-2.6.orig/arch/arm/mm/Kconfig > +++ linux-2.6/arch/arm/mm/Kconfig > @@ -630,6 +630,9 @@ config ARCH_PHYS_ADDR_T_64BIT > config ARCH_DMA_ADDR_T_64BIT > bool > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + bool > + Here it would have been nice with a comment that explains the use of ths symbol. > config ARM_THUMB > bool "Support Thumb user binaries" if !CPU_THUMBONLY > depends on CPU_ARM720T || CPU_ARM740T || CPU_ARM920T || CPU_ARM922T || \ > Index: linux-2.6/arch/arm64/Kconfig > === > --- linux-2.6.orig/arch/arm64/Kconfig > +++ linux-2.6/arch/arm64/Kconfig > @@ -125,6 +125,9 @@ config HAVE_GENERIC_RCU_GUP > config ARCH_DMA_ADDR_T_64BIT > def_bool y > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool y > + Use select > config NEED_DMA_MAP_STATE > def_bool y > > Index: linux-2.6/arch/ia64/Kconfig > === > --- linux-2.6.orig/arch/ia64/Kconfig > +++ linux-2.6/arch/ia64/Kconfig > @@ -74,6 +74,9 @@ config MMU > config ARCH_DMA_ADDR_T_64BIT > def_bool y > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool y > + Use select > config NEED_DMA_MAP_STATE > def_bool y > > Index: linux-2.6/arch/mips/Kconfig > === > --- linux-2.6.orig/arch/mips/Kconfig > +++ linux-2.6/arch/mips/Kconfig > @@ -984,6 +984,9 @@ config FW_CFE > config ARCH_DMA_ADDR_T_64BIT > def_bool (HIGHMEM && ARCH_PHYS_ADDR_T_64BIT) || 64BIT > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool (HIGHMEM && ARCH_PHYS_ADDR_T_64BIT) || 64BIT Use select ... if > === > --- linux-2.6.orig/arch/powerpc/Kconfig > +++ linux-2.6/arch/powerpc/Kconfig > @@ -23,6 +23,9 @@ config ARCH_PHYS_ADDR_T_64BIT > config ARCH_DMA_ADDR_T_64BIT > def_bool ARCH_PHYS_ADDR_T_64BIT > > +config ARCH_PCI_ADDR_T_64BIT > + def_bool ARCH_PHYS_ADDR_T_64BIT > + Use select > Index: linux-2.6/arch/sparc/Kconfig > === > --- linux-2.6.orig/arch/sparc/Kconfig > +++ linux-2.6/arch/sparc/Kconfig > @@ -143,6 +143,9 @@ config GENERIC_ISA_DMA > bool > default y if SPARC32 > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool y if SPARC64 > + Use select ... > config ARCH_SUPPORTS_DEBUG_PAGEALLOC > def_bool y if SPARC64 > > Index: linux-2.6/arch/tile/Kconfig > === > --- linux-2.6.orig/arch/tile/Kconfig > +++ linux-2.6/arch/tile/Kconfig > @@ -86,6 +86,9 @@ config ARCH_PHYS_ADDR_T_64BIT > config ARCH_DMA_ADDR_T_64BIT > def_bool y > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool y > + Use select ... > Index: linux-2.6/arch/x86/Kconfig > === > --- linux-2.6.orig/arch/x86/Kconfig > +++ linux-2.6/arch/x86/Kconfig > @@ -1295,6 +1295,10 @@ config ARCH_DMA_ADDR_T_64BIT > def_bool y > depends on X86_64 || HIGHMEM64G > > +config ARCH_PCI_BUS_ADDR_T_64BIT > + def_bool y > + depends on X86_64 || HIGHMEM64G Use select ... > + > config X86_DIRECT_GBPAGES > def_bool y > depends on X86_64 && !DEBUG_PAGEALLOC && !KMEMCHECK > Index: linux-2.6/include/linux/types.h > === > --- linux-2.6.orig/include/linux/types.h > +++ linux-2.6/include/linux/types.h > @@ -146,6 +146,13 @@ typedef u64 dma_addr_t; > typedef u32 dma_addr_t; > #endif /* dma_addr_t */ > > +/* A pci_bus_addr_t can hold pci bus address for the platform */ > +#ifdef CONFIG_ARCH_PCI_BUS_ADDR_T_64BIT > +typedef u64 pci_bus_addr_t; > +#else > +typedef u32 pci_bus_addr_t; > +#endif /* pci_bus_addr_t
Re: d63e2e1f3df breaks sparc/T5-8
On 3/27/15 6:32 PM, David Ahern wrote: On 3/27/15 5:57 PM, Yinghai Lu wrote: On Fri, Mar 27, 2015 at 2:01 PM, Yinghai Lu wrote: On Thu, Mar 26, 2015 at 4:27 PM, David Ahern wrote: Also please make sure your config have CONFIG_PCI_DEBUG=y and capture serial console with "debug ignore_loglevel", so we check if pci :00:01.0 really have resource assigned. Please check attached patch and send out boot log with above config. patched applied. PCI_DEBUG enabled. options added. log attached. This is with d63e2e1f3df reverted still. oops lied about that last one; I forgot to revert the patch this round. That the system booted with it applied is progress. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/27/15 5:57 PM, Yinghai Lu wrote: On Fri, Mar 27, 2015 at 2:01 PM, Yinghai Lu wrote: On Thu, Mar 26, 2015 at 4:27 PM, David Ahern wrote: Also please make sure your config have CONFIG_PCI_DEBUG=y and capture serial console with "debug ignore_loglevel", so we check if pci :00:01.0 really have resource assigned. Please check attached patch and send out boot log with above config. patched applied. PCI_DEBUG enabled. options added. log attached. This is with d63e2e1f3df reverted still. PROMLIB: Sun IEEE Boot Prom 'OBP 4.36.2 2014/10/24 08:15' PROMLIB: Root node compatible: sun4v Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Initializing cgroup subsys cpuacct Linux version 4.0.0-rc5+ (root@ca-qasparc24) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4.0.9) (GCC) ) #2 SMP Fri Mar 27 20:12:10 EDT 2015 debug: skip boot console de-registration. debug: ignoring loglevel setting. bootconsole [earlyprom0] enabled ARCH: SUN4V Ethernet address: 00:10:e0:35:15:f6 MM: PAGE_OFFSET is 0xfff8 (max_phys_bits == 47) MM: VMALLOC [0x0001 --> 0x0006] MM: VMEMMAP [0x0006 --> 0x000c] Kernel: Using 3 locked TLB entries for main kernel image. Remapping the kernel... done. OF stdout device is: /virtual-devices@100/console@1 PROM: Built device tree with 1478490 bytes of memory. MDESC: Size is 774368 bytes. PLATFORM: banner-name [SPARC T5-8] PLATFORM: name [ORCL,SPARC-T5-8] PLATFORM: hostid [863515f6] PLATFORM: serial# [003515f6] PLATFORM: stick-frequency [3b9aca00] PLATFORM: mac-address [10e03515f6] PLATFORM: watchdog-resolution [1000 ms] PLATFORM: watchdog-max-timeout [3153600 ms] PLATFORM: max-cpus [1024] Allocated 81920 bytes for kernel page tables. Zone ranges: DMA [mem 0x3040-0x] Normal [mem 0x-0x383fffd11fff] Movable zone start for each node Early memory node ranges node 0: [mem 0x3040-0x003fddee] node 0: [mem 0x003fddef6000-0x003fddef7fff] node 1: [mem 0x0800-0x083f] node 2: [mem 0x1000-0x103f] node 3: [mem 0x1800-0x183f] node 4: [mem 0x2000-0x203f] node 5: [mem 0x2800-0x283f] node 6: [mem 0x3000-0x303f] node 7: [mem 0x3800-0x383fffcddfff] node 7: [mem 0x383fffcee000-0x383fffd11fff] Initmem setup node 0 [mem 0x3040-0x003fddef7fff] On node 0 totalpages: 33385849 Normal zone: 293431 pages used for memmap Normal zone: 33385849 pages, LIFO batch:15 Initmem setup node 1 [mem 0x0800-0x083f] On node 1 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 2 [mem 0x1000-0x103f] On node 2 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 3 [mem 0x1800-0x183f] On node 3 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 4 [mem 0x2000-0x203f] On node 4 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 5 [mem 0x2800-0x283f] On node 5 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 6 [mem 0x3000-0x303f] On node 6 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 7 [mem 0x3800-0x383fffd11fff] On node 7 totalpages: 33554049 Normal zone: 294909 pages used for memmap Normal zone: 33554049 pages, LIFO batch:15 Booting Linux... CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit,fmaf,vis3] CPU CAPS: [hpc,ima,pause,cbcond,aes,des,kasumi,camellia] CPU CAPS: [md5,sha1,sha256,sha512,mpmul,montmul,montsqr,crc32c] PERCPU: Embedded 8 pages/cpu @fff8003f4d00 s27776 r8192 d29568 u65536 pcpu-alloc: s27776 r8192 d29568 u65536 alloc=1*4194304 pcpu-alloc: [0] 0001 0002 0003 0004 0005 0006 0007 0008 0009 0010 0011 0012 0013 0014 0015 0016 0017 0018 0019 0020 0021 0022 0023 0024 0025 0026 0027 0028 0029 0030 0031 0032 0033 0034 0035 0036 0037 0038 0039 0040 0041 0042 0043 0044 0045 0046 0047 0048 0049 0050 0051 0052 0053 0054 0055 0056 0057 0058 0059 0060 0061 0062 0063 pcpu-alloc: [0] 0064 0065 0066 0067 0068 0069 0070 0071 0072 0073 0074 0075 0076 0077 0078 0079 0080 0081 0082 0083 0084 0085 0086 0087 0088 0089 0090 0091 0092 0093 0094 0095 0096 0097 0098 0099 0100 0101 0102 0103 0104 0105 0106 0107 0108 0109 0110 0111 0112 0113 0114 0115 0116 0117 0118 0119
Re: d63e2e1f3df breaks sparc/T5-8
On Fri, Mar 27, 2015 at 2:01 PM, Yinghai Lu wrote: > On Thu, Mar 26, 2015 at 4:27 PM, David Ahern wrote: > Also please make sure your config have > > CONFIG_PCI_DEBUG=y > > and capture serial console with "debug ignore_loglevel", so we check if > pci :00:01.0 really have resource assigned. Please check attached patch and send out boot log with above config. Thanks Yinghai Subject: [RFC PATCH] PCI: Introduce pci_bus_addr_t David Ahern found commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows") broke sparc/T5-8. In the boot log, there is pci :06:00.0: reg 0x184: can't handle BAR above 4GB (bus address 0x110204000) but that only could happen when dma_addr_t is 32-bit. According to David Miller, all DMA occurs behind an IOMMU and these IOMMUs only support 32-bit addressing, therefore dma_addr_t is 32-bit on sparc64. Let's introduce pci_bus_addr instead of using dma_addr_t. Fixes: commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows") Fixes: commit 23b13bc76f35 ("PCI: Fail safely if we can't handle BARs larger than 4GB") Link: http://lkml.kernel.org/r/cae9fiqu1gjy1lyrxs+ma5lcteee4xmtjrg0axj9k_tsu+m9...@mail.gmail.com Reported-by: David Ahern Signed-off-by: Yinghai Lu --- arch/alpha/Kconfig |3 +++ arch/arm/Kconfig |1 + arch/arm/mach-axxia/Kconfig|1 + arch/arm/mach-exynos/Kconfig |1 + arch/arm/mach-highbank/Kconfig |1 + arch/arm/mach-shmobile/Kconfig |1 + arch/arm/mm/Kconfig|3 +++ arch/arm64/Kconfig |3 +++ arch/ia64/Kconfig |3 +++ arch/mips/Kconfig |3 +++ arch/powerpc/Kconfig |3 +++ arch/sparc/Kconfig |3 +++ arch/tile/Kconfig |3 +++ arch/x86/Kconfig |4 drivers/pci/bus.c |8 drivers/pci/probe.c| 12 ++-- include/linux/pci.h|6 +++--- include/linux/types.h |7 +++ 18 files changed, 53 insertions(+), 13 deletions(-) Index: linux-2.6/arch/alpha/Kconfig === --- linux-2.6.orig/arch/alpha/Kconfig +++ linux-2.6/arch/alpha/Kconfig @@ -66,6 +66,9 @@ config ZONE_DMA config ARCH_DMA_ADDR_T_64BIT def_bool y +config ARCH_PCI_BUS_ADDR_T_64BIT + def_bool y + config NEED_DMA_MAP_STATE def_bool y Index: linux-2.6/arch/arm/Kconfig === --- linux-2.6.orig/arch/arm/Kconfig +++ linux-2.6/arch/arm/Kconfig @@ -1779,6 +1779,7 @@ config XEN depends on !GENERIC_ATOMIC64 depends on MMU select ARCH_DMA_ADDR_T_64BIT + select ARCH_PCI_BUS_ADDR_T_64BIT select ARM_PSCI select SWIOTLB_XEN help Index: linux-2.6/arch/arm/mach-axxia/Kconfig === --- linux-2.6.orig/arch/arm/mach-axxia/Kconfig +++ linux-2.6/arch/arm/mach-axxia/Kconfig @@ -1,6 +1,7 @@ config ARCH_AXXIA bool "LSI Axxia platforms" if (ARCH_MULTI_V7 && ARM_LPAE) select ARCH_DMA_ADDR_T_64BIT + select ARCH_PCI_BUS_ADDR_T_64BIT select ARM_AMBA select ARM_GIC select ARM_TIMER_SP804 Index: linux-2.6/arch/arm/mach-exynos/Kconfig === --- linux-2.6.orig/arch/arm/mach-exynos/Kconfig +++ linux-2.6/arch/arm/mach-exynos/Kconfig @@ -106,6 +106,7 @@ config SOC_EXYNOS5440 default y depends on ARCH_EXYNOS5 select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE + select ARCH_PCI_BUS_ADDR_T_64BIT if ARM_LPAE select HAVE_ARM_ARCH_TIMER select AUTO_ZRELADDR select MIGHT_HAVE_PCI Index: linux-2.6/arch/arm/mach-highbank/Kconfig === --- linux-2.6.orig/arch/arm/mach-highbank/Kconfig +++ linux-2.6/arch/arm/mach-highbank/Kconfig @@ -1,6 +1,7 @@ config ARCH_HIGHBANK bool "Calxeda ECX-1000/2000 (Highbank/Midway)" if ARCH_MULTI_V7 select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE + select ARCH_PCI_BUS_ADDR_T_64BIT if ARM_LPAE select ARCH_HAS_HOLES_MEMORYMODEL select ARCH_SUPPORTS_BIG_ENDIAN select ARM_AMBA Index: linux-2.6/arch/arm/mach-shmobile/Kconfig === --- linux-2.6.orig/arch/arm/mach-shmobile/Kconfig +++ linux-2.6/arch/arm/mach-shmobile/Kconfig @@ -36,6 +36,7 @@ menuconfig ARCH_SHMOBILE_MULTI select HAVE_ARM_TWD if SMP select ARM_GIC select ARCH_DMA_ADDR_T_64BIT if ARM_LPAE + select ARCH_PCI_BUS_ADDR_T_64BIT if ARM_LPAE select NO_IOPORT_MAP select PINCTRL select ARCH_REQUIRE_GPIOLIB Index: linux-2.6/arch/arm/mm/Kconfig === --- linux-2.6.orig/arch/arm/mm/Kconfig +++ linux-2.6/arch/arm/mm/Kconfig @@ -630,6 +630,9 @@ config ARCH_PHYS_ADDR_T_64BIT config ARCH_DMA_ADDR_T_64BIT bool +config ARCH_PCI_BUS_ADDR_T_64BIT + bool + config ARM_THUMB bool "Support Thu
Re: d63e2e1f3df breaks sparc/T5-8
On Fri, Mar 27, 2015 at 2:50 PM, David Miller wrote: > All DMA occurs behind an IOMMU and these IOMMUs only > support 32-bit addressing, therefore dma_addr_t is > 32-bit on sparc64. > > If you want to represent PCI address in some way, you > absolutely cannot use dma_addr_t as your data type. Oh no, we missed that for a while. Hi Bjorn, I would suggest that we introduce pci_bus_addr_t or pci_addr_t. Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
From: Yinghai Lu Date: Fri, 27 Mar 2015 14:01:54 -0700 > On Thu, Mar 26, 2015 at 4:27 PM, David Ahern wrote: >> On 3/26/15 2:43 PM, Yinghai Lu wrote: >>> >>> Can you send out boot log with "debug ignore_loglevel"? >> >> >> attached > > So the kernel config is sparc32 or sparc64 ? > > pci :06:00.0: reg 0x184: can't handle BAR above 4GB (bus address > 0x110204000) > > > only could happen when dma_addr_t is 32bit. All DMA occurs behind an IOMMU and these IOMMUs only support 32-bit addressing, therefore dma_addr_t is 32-bit on sparc64. If you want to represent PCI address in some way, you absolutely cannot use dma_addr_t as your data type. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On Thu, Mar 26, 2015 at 4:27 PM, David Ahern wrote: > On 3/26/15 2:43 PM, Yinghai Lu wrote: >> >> Can you send out boot log with "debug ignore_loglevel"? > > > attached So the kernel config is sparc32 or sparc64 ? pci :06:00.0: reg 0x184: can't handle BAR above 4GB (bus address 0x110204000) only could happen when dma_addr_t is 32bit. in drivers/pci/probe.c::__pcie_read_base() if ((sizeof(dma_addr_t) < 8) && l) { /* Above 32-bit boundary; try to reallocate */ res->flags |= IORESOURCE_UNSET; res->start = 0; res->end = sz64; dev_info(&dev->dev, "reg 0x%x: can't handle BAR above 4GB (bus address %#010llx)\n", pos, (unsigned long long)l64); goto out; } also root bus has pci_sun4v f02dbcfc: PCI host bridge to bus :00 pci_bus :00: root bus resource [io 0x8040-0x80400fff] (bus address [0x-0xfff]) pci_bus :00: root bus resource [mem 0x8000-0x80007eff] (bus address [0x-0x7eff]) pci_bus :00: root bus resource [bus 00-77] and children device have pci :05:00.0: can't claim BAR 0 [mem 0x8001-0x80011fff]: no compatible bridge window pci :06:00.0: can't claim BAR 0 [mem 0x80011000-0x8001101f pref]: no compatible bridge window pci :06:00.0: can't claim BAR 4 [mem 0x80011020-0x800110203fff pref]: no compatible bridge window pci :06:00.1: can't claim BAR 0 [mem 0x80011060-0x8001107f pref]: no compatible bridge window pci :06:00.1: can't claim BAR 4 [mem 0x800110404000-0x800110407fff pref]: no compatible bridge window pci :07:00.0: can't claim BAR 1 [mem 0x80012000-0x80012000]: no compatible bridge window pci :07:00.0: can't claim BAR 3 [mem 0x80012004-0x80012007]: no compatible bridge window looks like some root bus resource get dumped because it is out of range. Also please make sure your config have CONFIG_PCI_DEBUG=y and capture serial console with "debug ignore_loglevel", so we check if pci :00:01.0 really have resource assigned. Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: d63e2e1f3df breaks sparc/T5-8
On 3/26/15 2:43 PM, Yinghai Lu wrote: Can you send out boot log with "debug ignore_loglevel"? attached PROMLIB: Sun IEEE Boot Prom 'OBP 4.36.2 2014/10/24 08:15' PROMLIB: Root node compatible: sun4v Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Initializing cgroup subsys cpuacct Linux version 4.0.0-rc5+ (root@ca-qasparc24) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-4.0.9) (GCC) ) #1 SMP Thu Mar 26 18:51:40 EDT 2015 debug: skip boot console de-registration. debug: ignoring loglevel setting. bootconsole [earlyprom0] enabled ARCH: SUN4V Ethernet address: 00:10:e0:35:15:f6 MM: PAGE_OFFSET is 0xfff8 (max_phys_bits == 47) MM: VMALLOC [0x0001 --> 0x0006] MM: VMEMMAP [0x0006 --> 0x000c] Kernel: Using 3 locked TLB entries for main kernel image. Remapping the kernel... done. OF stdout device is: /virtual-devices@100/console@1 PROM: Built device tree with 1478492 bytes of memory. MDESC: Size is 774368 bytes. PLATFORM: banner-name [SPARC T5-8] PLATFORM: name [ORCL,SPARC-T5-8] PLATFORM: hostid [863515f6] PLATFORM: serial# [003515f6] PLATFORM: stick-frequency [3b9aca00] PLATFORM: mac-address [10e03515f6] PLATFORM: watchdog-resolution [1000 ms] PLATFORM: watchdog-max-timeout [3153600 ms] PLATFORM: max-cpus [1024] Allocated 81920 bytes for kernel page tables. Zone ranges: DMA [mem 0x3040-0x] Normal [mem 0x-0x383fffd11fff] Movable zone start for each node Early memory node ranges node 0: [mem 0x3040-0x003fddee] node 0: [mem 0x003fddef6000-0x003fddef7fff] node 1: [mem 0x0800-0x083f] node 2: [mem 0x1000-0x103f] node 3: [mem 0x1800-0x183f] node 4: [mem 0x2000-0x203f] node 5: [mem 0x2800-0x283f] node 6: [mem 0x3000-0x303f] node 7: [mem 0x3800-0x383fffcddfff] node 7: [mem 0x383fffcee000-0x383fffd11fff] Initmem setup node 0 [mem 0x3040-0x003fddef7fff] On node 0 totalpages: 33385849 Normal zone: 293431 pages used for memmap Normal zone: 33385849 pages, LIFO batch:15 Initmem setup node 1 [mem 0x0800-0x083f] On node 1 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 2 [mem 0x1000-0x103f] On node 2 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 3 [mem 0x1800-0x183f] On node 3 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 4 [mem 0x2000-0x203f] On node 4 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 5 [mem 0x2800-0x283f] On node 5 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 6 [mem 0x3000-0x303f] On node 6 totalpages: 33554432 Normal zone: 294912 pages used for memmap Normal zone: 33554432 pages, LIFO batch:15 Initmem setup node 7 [mem 0x3800-0x383fffd11fff] On node 7 totalpages: 33554049 Normal zone: 294909 pages used for memmap Normal zone: 33554049 pages, LIFO batch:15 Booting Linux... CPU CAPS: [flush,stbar,swap,muldiv,v9,blkinit,n2,mul32] CPU CAPS: [div32,v8plus,popc,vis,vis2,ASIBlkInit,fmaf,vis3] CPU CAPS: [hpc,ima,pause,cbcond,aes,des,kasumi,camellia] CPU CAPS: [md5,sha1,sha256,sha512,mpmul,montmul,montsqr,crc32c] PERCPU: Embedded 8 pages/cpu @fff8003f4d00 s27776 r8192 d29568 u65536 pcpu-alloc: s27776 r8192 d29568 u65536 alloc=1*4194304 pcpu-alloc: [0] 0001 0002 0003 0004 0005 0006 0007 0008 0009 0010 0011 0012 0013 0014 0015 0016 0017 0018 0019 0020 0021 0022 0023 0024 0025 0026 0027 0028 0029 0030 0031 0032 0033 0034 0035 0036 0037 0038 0039 0040 0041 0042 0043 0044 0045 0046 0047 0048 0049 0050 0051 0052 0053 0054 0055 0056 0057 0058 0059 0060 0061 0062 0063 pcpu-alloc: [0] 0064 0065 0066 0067 0068 0069 0070 0071 0072 0073 0074 0075 0076 0077 0078 0079 0080 0081 0082 0083 0084 0085 0086 0087 0088 0089 0090 0091 0092 0093 0094 0095 0096 0097 0098 0099 0100 0101 0102 0103 0104 0105 0106 0107 0108 0109 0110 0111 0112 0113 0114 0115 0116 0117 0118 0119 0120 0121 0122 0123 0124 0125 0126 0127 pcpu-alloc: [1] 0128 0129 0130 0131 0132 0133 0134 0135 0136 0137 0138 0139 0140 0141 0142 0143 0144 0145 0146 0147 0148 0149 0150 0151 0152 0153 0154 0155 0156 0157 0158 0159 0160 0161 0162 0163 0164 0165 0166 0167 0168 0169 0170 0171 0172 0173 0174 0175 0176 0177 0178 0179 0180 0181 0182 0183 0184 0185 0186 0187 0188 0189 0190 0191 pcpu-alloc: [1] 0192
Re: d63e2e1f3df breaks sparc/T5-8
[+bjorn, pci list] On Thu, Mar 26, 2015 at 9:51 AM, David Ahern wrote: > Hi: > > Boot of an 8-socket T5 sparc system fails on top of tree. git bisect points > to this commit: > > commit 904bf6bd150bdafb42ddbb3257ea8 > Author: Yinghai Lu > Date: Thu Jan 15 16:21:51 2015 -0600 > > sparc/PCI: Clip bridge windows to fit in upstream windows > > Every PCI-PCI bridge window should fit inside an upstream bridge window > because orphaned address space is unreachable from the primary side of the > upstream bridge. If we inherit invalid bridge windows that overlap an > upstream window from firmware, clip them to fit and update the bridge > accordingly. > > > Sure enough if I revert it on top of tree the system boots fine. Can you send out boot log with "debug ignore_loglevel"? Thanks Yinghai -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
d63e2e1f3df breaks sparc/T5-8
Hi: Boot of an 8-socket T5 sparc system fails on top of tree. git bisect points to this commit: commit 904bf6bd150bdafb42ddbb3257ea8 Author: Yinghai Lu Date: Thu Jan 15 16:21:51 2015 -0600 sparc/PCI: Clip bridge windows to fit in upstream windows Every PCI-PCI bridge window should fit inside an upstream bridge window because orphaned address space is unreachable from the primary side of the upstream bridge. If we inherit invalid bridge windows that overlap an upstream window from firmware, clip them to fit and update the bridge accordingly. Sure enough if I revert it on top of tree the system boots fine. David -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/