Re: Why does BIOS assign memory to 16 byte BAR
Hi Bharat, On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote: > Hi, > > I'm observing that on x86 BIOS successfully assigns memory if an End point > requests > BAR of size 16byte. > > But as per Spec: > The minimum memory address range requested by a BAR is 128 bytes. Can you provide the spec reference for this? I don't see it in PCI r3.0. PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable, which would correspond to a minimum size of 16 bytes. > Why BIOS is successfully allocating region to 16 byte BAR requests? If you want to know why the BIOS does this, you'd have to ask the BIOS writer. If you were to ask about *Linux*, I'd say we should not gratuitously enforce things in the spec merely for the sake of being compliant with the letter of the spec. We should enforce things that are logically required or that are required for compatibility, security, etc. But if the spec contained an arbitrary restriction for no good reason, and real-world hardware violated that restriction, I'd say Linux should accommodate the hardware. Bjorn
Re: Why does BIOS assign memory to 16 byte BAR
On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote: > Hi Bharat, > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote: > > Hi, > > > > I'm observing that on x86 BIOS successfully assigns memory if an End point > > requests > > BAR of size 16byte. > > > > But as per Spec: > > The minimum memory address range requested by a BAR is 128 bytes. > > Can you provide the spec reference for this? I don't see it in PCI > r3.0. > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable, > which would correspond to a minimum size of 16 bytes. The reference above is to the conventional PCI spec. I happened to trip over a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the minimum memory address range requested by a BAR is 128 bytes." I don't think linux currently enforces this minimum. Bjorn
RE: Why does BIOS assign memory to 16 byte BAR
> Subject: Re: Why does BIOS assign memory to 16 byte BAR > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote: > > Hi Bharat, > > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote: > > > Hi, > > > > > > I'm observing that on x86 BIOS successfully assigns memory if an End > > > point requests BAR of size 16byte. > > > > > > But as per Spec: > > > The minimum memory address range requested by a BAR is 128 bytes. > > > > Can you provide the spec reference for this? I don't see it in PCI > > r3.0. > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable, > > which would correspond to a minimum size of 16 bytes. > > The reference above is to the conventional PCI spec. I happened to trip over > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the > minimum > memory address range requested by a BAR is 128 bytes." > > I don't think linux currently enforces this minimum. > Hi Bjorn Thanks for the reply. Here is what the issue we are seeing. We have total memory for BAR's on our SoC of 256 MB. When an End Point request individually 16 byte BAR's our root port assigns memory to BAR's successfully. But if I have an End point which has 4 BAR's each 32 bit and request as following: When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are running this as part of SIG compliance test case) 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not getting BAR's assigned. (Even though BAR space is available, since 1GB failed, We have 256 MB still) We have only one End point connected to our root port. Here is the log: [2.319289] nwl-pcie fd0e.pcie: Link is UP [2.319332] PCI host bridge /amba/pcie@fd0e ranges: [2.319349] No bus range found for /amba/pcie@fd0e, using [bus 00-ff] [2.319374]IO 0xe000..0xe000 -> 0x [2.319415] MEM 0xe010..0xefff -> 0xe010 [2.319431] MEM 0x6..0x7 -> 0x6 [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00 [2.319557] pci_bus :00: root bus resource [bus 00-ff] [2.319573] pci_bus :00: root bus resource [io 0x-0x] [2.319589] pci_bus :00: root bus resource [mem 0xe010-0xefff] [2.319606] pci_bus :00: root bus resource [mem 0x6-0x7 pref] [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same bus? [2.319861] iommu: Adding device :00:00.0 to group 1 [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same bus? [2.320258] iommu: Adding device :01:00.0 to group 1 [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000] [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 0x6000] [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-0xe01007ff pref] [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000] [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 0x4000] [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 64bit] [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 0x0010 64bit] [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010] [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 0x0010] [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010] [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 0x0010] [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c] Please let me know, what might might be the issue. Thanks & Regards, Bharat
RE: Why does BIOS assign memory to 16 byte BAR
> Subject: RE: Why does BIOS assign memory to 16 byte BAR > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR > > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote: > > > Hi Bharat, > > > > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote: > > > > Hi, > > > > > > > > I'm observing that on x86 BIOS successfully assigns memory if an End > > > > point requests BAR of size 16byte. > > > > > > > > But as per Spec: > > > > The minimum memory address range requested by a BAR is 128 bytes. > > > > > > Can you provide the spec reference for this? I don't see it in PCI > > > r3.0. > > > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable, > > > which would correspond to a minimum size of 16 bytes. > > > > The reference above is to the conventional PCI spec. I happened to trip > over > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the > minimum > > memory address range requested by a BAR is 128 bytes." > > > > I don't think linux currently enforces this minimum. > > > > Hi Bjorn Thanks for the reply. > > Here is what the issue we are seeing. > > We have total memory for BAR's on our SoC of 256 MB. > When an End Point request individually 16 byte BAR's our root port assigns > memory to BAR's successfully. > > But if I have an End point which has 4 BAR's each 32 bit and request as > following: > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are > running this as part of SIG compliance test case) > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not > getting BAR's assigned. (Even though BAR space is available, since 1GB failed, > We have 256 MB still) > > We have only one End point connected to our root port. > > Here is the log: > [2.319289] nwl-pcie fd0e.pcie: Link is UP > [2.319332] PCI host bridge /amba/pcie@fd0e ranges: > [2.319349] No bus range found for /amba/pcie@fd0e, using [bus 00- > ff] > [2.319374]IO 0xe000..0xe000 -> 0x > [2.319415] MEM 0xe010..0xefff -> 0xe010 > [2.319431] MEM 0x6..0x7 -> 0x6 > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00 > [2.319557] pci_bus :00: root bus resource [bus 00-ff] > [2.319573] pci_bus :00: root bus resource [io 0x-0x] > [2.319589] pci_bus :00: root bus resource [mem 0xe010-0xefff] > [2.319606] pci_bus :00: root bus resource [mem 0x6- > 0x7 pref] > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same bus? > [2.319861] iommu: Adding device :00:00.0 to group 1 > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same bus? > [2.320258] iommu: Adding device :01:00.0 to group 1 > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000] > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 0x6000] > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-0xe01007ff > pref] > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000] > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 0x4000] > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 > 64bit] > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 0x0010 > 64bit] > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010] > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 0x0010] > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010] > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 0x0010] > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c] > > Please let me know, what might might be the issue. > Adding to the above will kernel allocate other memory BARs to an End Point if one BAR assignment fails ? What if the End Point has multiple function and say first function BAR assignment failed, will the kernel assign BAR's to second function on same bus and device ? Thanks & Regards, Bharat
RE: Why does BIOS assign memory to 16 byte BAR
> Subject: RE: Why does BIOS assign memory to 16 byte BAR > > > Subject: RE: Why does BIOS assign memory to 16 byte BAR > > > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR > > > > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote: > > > > Hi Bharat, > > > > > > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote: > > > > > Hi, > > > > > > > > > > I'm observing that on x86 BIOS successfully assigns memory if an End > > > > > point requests BAR of size 16byte. > > > > > > > > > > But as per Spec: > > > > > The minimum memory address range requested by a BAR is 128 > bytes. > > > > > > > > Can you provide the spec reference for this? I don't see it in PCI > > > > r3.0. > > > > > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable, > > > > which would correspond to a minimum size of 16 bytes. > > > > > > The reference above is to the conventional PCI spec. I happened to trip > > over > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the > > minimum > > > memory address range requested by a BAR is 128 bytes." > > > > > > I don't think linux currently enforces this minimum. > > > > > > > Hi Bjorn Thanks for the reply. > > > > Here is what the issue we are seeing. > > > > We have total memory for BAR's on our SoC of 256 MB. > > When an End Point request individually 16 byte BAR's our root port assigns > > memory to BAR's successfully. > > > > But if I have an End point which has 4 BAR's each 32 bit and request as > > following: > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are > > running this as part of SIG compliance test case) > > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not > > getting BAR's assigned. (Even though BAR space is available, since 1GB > failed, > > We have 256 MB still) > > > > We have only one End point connected to our root port. > > > > Here is the log: > > [2.319289] nwl-pcie fd0e.pcie: Link is UP > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges: > > [2.319349] No bus range found for /amba/pcie@fd0e, using [bus > 00- > > ff] > > [2.319374]IO 0xe000..0xe000 -> 0x > > [2.319415] MEM 0xe010..0xefff -> 0xe010 > > [2.319431] MEM 0x6..0x7 -> 0x6 > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00 > > [2.319557] pci_bus :00: root bus resource [bus 00-ff] > > [2.319573] pci_bus :00: root bus resource [io 0x-0x] > > [2.319589] pci_bus :00: root bus resource [mem 0xe010- > 0xefff] > > [2.319606] pci_bus :00: root bus resource [mem 0x6- > > 0x7 pref] > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same > bus? > > [2.319861] iommu: Adding device :00:00.0 to group 1 > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same > bus? > > [2.320258] iommu: Adding device :01:00.0 to group 1 > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000] > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size > > 0x6000] > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010- > 0xe01007ff > > pref] > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000] > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size > > 0x4000] > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 > > 64bit] > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size > > 0x0010 > > 64bit] > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010] > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size > > 0x0010] > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010] > > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size > > 0x0010] > > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c] > > > > Please let me know, what might might be the issue. > > > > Adding to the above will kernel allocate other memory BARs to an End Point > if > one BAR assignment fails ? > > What if the End Point has multiple function and say first function BAR > assignment failed, > will the kernel assign BAR's to second function on same bus and device ? > After debugging in function pci_bus_alloc_from_region we have two resources, One which is taken as parameter (let's say resA), other is from pci_bus_for_each_resource (let's say resB). The resA contains different start address from resB, I see that resB is obtained from device resource from bus->resource, but im unable to understand how resA (start address & size) gets its data. Can any one help me understand this so that I might know reason why 16byte BAR allocation failing after 1GB request. Regards, Bharat
RE: Why does BIOS assign memory to 16 byte BAR
Adding yinghai lu. > > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR > > > > > > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote: > > > > > Hi Bharat, > > > > > > > > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote: > > > > > > Hi, > > > > > > > > > > > > I'm observing that on x86 BIOS successfully assigns memory if > > > > > > an End point requests BAR of size 16byte. > > > > > > > > > > > > But as per Spec: > > > > > > The minimum memory address range requested by a BAR is 128 > > bytes. > > > > > > > > > > Can you provide the spec reference for this? I don't see it in > > > > > PCI r3.0. > > > > > > > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as > > > > > writable, which would correspond to a minimum size of 16 bytes. > > > > > > > > The reference above is to the conventional PCI spec. I happened > > > > to trip > > > over > > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, > > > > "the > > > minimum > > > > memory address range requested by a BAR is 128 bytes." > > > > > > > > I don't think linux currently enforces this minimum. > > > > > > > > > > Hi Bjorn Thanks for the reply. > > > > > > Here is what the issue we are seeing. > > > > > > We have total memory for BAR's on our SoC of 256 MB. > > > When an End Point request individually 16 byte BAR's our root port > > > assigns memory to BAR's successfully. > > > > > > But if I have an End point which has 4 BAR's each 32 bit and request > > > as > > > following: > > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We > > > are running this as part of SIG compliance test case) 2nd BAR > > > requests 1MB and other 2 BAR's request 16byte, but these are not > > > getting BAR's assigned. (Even though BAR space is available, since > > > 1GB > > failed, > > > We have 256 MB still) > > > > > > We have only one End point connected to our root port. > > > > > > Here is the log: > > > [2.319289] nwl-pcie fd0e.pcie: Link is UP > > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges: > > > [2.319349] No bus range found for /amba/pcie@fd0e, using [bus > > 00- > > > ff] > > > [2.319374]IO 0xe000..0xe000 -> 0x > > > [2.319415] MEM 0xe010..0xefff -> 0xe010 > > > [2.319431] MEM 0x6..0x7 -> 0x6 > > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00 > > > [2.319557] pci_bus :00: root bus resource [bus 00-ff] > > > [2.319573] pci_bus :00: root bus resource [io 0x-0x] > > > [2.319589] pci_bus :00: root bus resource [mem 0xe010- > > 0xefff] > > > [2.319606] pci_bus :00: root bus resource [mem 0x6- > > > 0x7 pref] > > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same > > bus? > > > [2.319861] iommu: Adding device :00:00.0 to group 1 > > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same > > bus? > > > [2.320258] iommu: Adding device :01:00.0 to group 1 > > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000] > > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size > 0x6000] > > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010- > > 0xe01007ff > > > pref] > > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000] > > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size > 0x4000] > > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 > > > 64bit] > > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size > 0x0010 > > > 64bit] > > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010] > > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size > 0x0010] > > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010] > > >
Re: Why does BIOS assign memory to 16 byte BAR
On Tue, Jul 26, 2016 at 8:43 AM, Bharat Kumar Gogada wrote: >> > > We have only one End point connected to our root port. >> > > >> > > Here is the log: >> > > [2.319289] nwl-pcie fd0e.pcie: Link is UP >> > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges: >> > > [2.319349] No bus range found for /amba/pcie@fd0e, using [bus >> > 00- >> > > ff] >> > > [2.319374]IO 0xe000..0xe000 -> 0x >> > > [2.319415] MEM 0xe010..0xefff -> 0xe010 >> > > [2.319431] MEM 0x6..0x7 -> 0x6 >> > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00 >> > > [2.319557] pci_bus :00: root bus resource [bus 00-ff] >> > > [2.319573] pci_bus :00: root bus resource [io 0x-0x] >> > > [2.319589] pci_bus :00: root bus resource [mem 0xe010- >> > 0xefff] >> > > [2.319606] pci_bus :00: root bus resource [mem 0x6- >> > > 0x7 pref] >> > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same >> > bus? >> > > [2.319861] iommu: Adding device :00:00.0 to group 1 >> > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same >> > bus? >> > > [2.320258] iommu: Adding device :01:00.0 to group 1 >> > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size >> > > 0x6000] >> > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size >> 0x6000] >> > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010- >> > 0xe01007ff >> > > pref] >> > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size >> > > 0x4000] >> > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size >> 0x4000] >> > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 >> > > 64bit] >> > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size >> 0x0010 >> > > 64bit] >> > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size >> > > 0x0010] >> > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size >> 0x0010] >> > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size >> > > 0x0010] >> > > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size >> 0x0010] >> > > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c] >> > > >> > >> > What if the End Point has multiple function and say first function BAR >> > assignment failed, will the kernel assign BAR's to second function on >> > same bus and device ? Your system host bridge: has resource pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus :00: root bus resource [mem 0x6-0x7 pref] then one pci bridge: pci :00:00.0 then :01:00.0 have four bars: pci :01:00.0: BAR 0: [mem size 0x4000] pci :01:00.0: BAR 4: [mem size 0x0010 64bit] pci :01:00.0: BAR 2: [mem size 0x0010] pci :01:00.0: BAR 3: [mem size 0x0010] kernel need to get allocation for pci :00:00.0 at first but can not find big enough space. pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should come from [mem 0xe010-0xefff], and that is less 1.5G. so all children resource from pci :01:00.0 all fail. please check if you modify your FPGA code to make pci :01:00.0 BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio. or you can check if can increase root bus mmio range MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host bridge to bus :00 to have more than 1.5G. Thanks Yinghai
RE: Why does BIOS assign memory to 16 byte BAR
> Your system host bridge: has resource > pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus > :00: root bus resource [mem 0x6-0x7 pref] then one pci > bridge: > pci :00:00.0 > then :01:00.0 have four bars: > pci :01:00.0: BAR 0: [mem size 0x4000] pci :01:00.0: BAR 4: > [mem size 0x0010 64bit] pci :01:00.0: BAR 2: [mem size 0x0010] > pci :01:00.0: BAR 3: [mem size 0x0010] > > > kernel need to get allocation for pci :00:00.0 at first > > but can not find big enough space. > > pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should > come from [mem 0xe010-0xefff], and that is less 1.5G. > > so all children resource from pci :01:00.0 all fail. > > > please check if you modify your FPGA code to make pci :01:00.0 > > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio. > > or you can check if can increase root bus mmio range > > MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host > bridge to bus :00 > > to have more than 1.5G. > Thanks Yinghai Lu. We see that similar test is passing in x86 machine, where function one requesting 1GB BAR's is failing, but function two requesting BAR's with 16byte is getting assigned BAR's. To my knowledge on x86 BIOS assigns resources, or will kernel assign reosurces on x86 ? If kernel does is there any difference between x86 and arm64 resource assignment logic ? Thanks & Regards, Bharat
Re: Why does BIOS assign memory to 16 byte BAR
On Wed, Jul 27, 2016 at 06:33:29AM +, Bharat Kumar Gogada wrote: > > Your system host bridge: has resource > > pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus > > :00: root bus resource [mem 0x6-0x7 pref] then one pci > > bridge: > > pci :00:00.0 > > then :01:00.0 have four bars: > > pci :01:00.0: BAR 0: [mem size 0x4000] pci :01:00.0: BAR 4: > > [mem size 0x0010 64bit] pci :01:00.0: BAR 2: [mem size 0x0010] > > pci :01:00.0: BAR 3: [mem size 0x0010] > > > > > > kernel need to get allocation for pci :00:00.0 at first > > > > but can not find big enough space. > > > > pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should > > come from [mem 0xe010-0xefff], and that is less 1.5G. > > > > so all children resource from pci :01:00.0 all fail. > > > > > > please check if you modify your FPGA code to make pci :01:00.0 > > > > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio. > > > > or you can check if can increase root bus mmio range > > > > MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host > > bridge to bus :00 > > > > to have more than 1.5G. > > > Thanks Yinghai Lu. > We see that similar test is passing in x86 machine, where function one > requesting 1GB BAR's is failing, but function two requesting BAR's > with 16byte is getting assigned BAR's. > > To my knowledge on x86 BIOS assigns resources, or will kernel assign > reosurces on x86 ? If kernel does is there any difference between x86 > and arm64 resource assignment logic ? We can't answer your question if you do not provide a full log of x86 and ARM PCI configurations you are testing I am afraid. It is also unclear to me what "a similar test is passing in x86 machine" means, in particular in relation to the HW configuration you are testing on x86. Yes, there are differences between x86 and ARM resources assignments, x86 tries to claim PCI resources as set-up by BIOS and assign them iff the claiming fails whereas on ARM (and that's done in the host bridge driver) FW configuration is always discarded and the kernel reassigns the whole PCI resource hierarchy entirely, but to help we need more data as I said above. I suspect the issue you are facing has to do as Yinghai mentioned with the prefetchable memory window set-up, saying that "it works" on x86 does not really help unless you provide data to debug it. Thanks, Lorenzo
RE: Why does BIOS assign memory to 16 byte BAR
> > We see that similar test is passing in x86 machine, where function one > > requesting 1GB BAR's is failing, but function two requesting BAR's > > with 16byte is getting assigned BAR's. > > > > To my knowledge on x86 BIOS assigns resources, or will kernel assign > > reosurces on x86 ? If kernel does is there any difference between x86 > > and arm64 resource assignment logic ? > > We can't answer your question if you do not provide a full log of x86 and > ARM PCI configurations you are testing I am afraid. > > It is also unclear to me what "a similar test is passing in x86 machine" > means, > in particular in relation to the HW configuration you are testing on x86. > > Yes, there are differences between x86 and ARM resources assignments, > x86 tries to claim PCI resources as set-up by BIOS and assign them iff the > claiming fails whereas on ARM (and that's done in the host bridge driver) FW > configuration is always discarded and the kernel reassigns the whole PCI > resource hierarchy entirely, but to help we need more data as I said above. > Thanks Lorenzo. I will try to get x86 configuration details soon and post, as the setup is at different location. On ARM we using pcie-xilinx-nwl.c configurations where we have where we have 240MB BAR space as per device tree documentation. (The kernel log I posted for testing purpose is using different address spaces, if tests were successful, we will soon send patches for device tree) Similar test case means, on x86 where we have multifunction device, where 1st function requests 1GB memory BAR's and 2nd function requests 16byte memory BAR's, for 1st function BAR's are not assigned, but for 2nd function 16byte BAR assignment is successful. But on ARM similar test case fails to assign BAR's for both first function and second function. What is the reason on ARM we are handling resource assignment in hierarchy format? Because on ARM, in the above case due to lack of resources first function BAR assignment fails which is fine but even though we have enough resources for 16byte BAR allocations for second function, we are not assigning. Thanks & Regards, Bharat This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.