RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> > We see that similar test is passing in x86 machine, where function one
> > requesting 1GB BAR's is failing, but function two requesting BAR's
> > with 16byte is getting assigned BAR's.
> >
> > To my knowledge on x86 BIOS assigns resources, or will kernel assign
> > reosurces on x86 ?  If kernel does is there any difference between x86
> > and arm64 resource assignment logic ?
>
> We can't answer your question if you do not provide a full log of x86 and
> ARM PCI configurations you are testing I am afraid.
>
> It is also unclear to me what "a similar test is passing in x86 machine" 
> means,
> in particular in relation to the HW configuration you are testing on x86.
>
> Yes, there are differences between x86 and ARM resources assignments,
> x86 tries to claim PCI resources as set-up by BIOS and assign them iff the
> claiming fails whereas on ARM (and that's done in the host bridge driver) FW
> configuration is always discarded and the kernel reassigns the whole PCI
> resource hierarchy entirely, but to help we need more data as I said above.
>

Thanks Lorenzo.
I will try to get x86 configuration details soon and post, as the setup is at 
different location.
On ARM we using pcie-xilinx-nwl.c configurations where we have where we have 
240MB BAR space
as per device tree documentation. (The kernel log I posted for testing purpose 
is using different address spaces,
if tests were successful, we will soon send patches for device tree)

Similar test case means, on x86 where we have multifunction device, where 1st 
function requests 1GB memory BAR's
and 2nd function requests 16byte memory BAR's, for 1st function BAR's are not 
assigned, but for 2nd function 16byte BAR assignment
is successful.

But on ARM similar test case fails to assign BAR's for both first function and 
second function.

What is the reason on ARM we are handling resource assignment in hierarchy 
format?
Because on ARM, in the above case due to lack of resources first function BAR 
assignment fails  which is fine but
even though we have enough resources for 16byte BAR allocations for second 
function, we are not assigning.


Thanks & Regards,
Bharat


This email and any attachments are intended for the sole use of the named 
recipient(s) and contain(s) confidential information that may be proprietary, 
privileged or copyrighted under applicable law. If you are not the intended 
recipient, do not read, copy, or forward this email message or any attachments. 
Delete this email message and any attachments immediately.

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> > We see that similar test is passing in x86 machine, where function one
> > requesting 1GB BAR's is failing, but function two requesting BAR's
> > with 16byte is getting assigned BAR's.
> >
> > To my knowledge on x86 BIOS assigns resources, or will kernel assign
> > reosurces on x86 ?  If kernel does is there any difference between x86
> > and arm64 resource assignment logic ?
>
> We can't answer your question if you do not provide a full log of x86 and
> ARM PCI configurations you are testing I am afraid.
>
> It is also unclear to me what "a similar test is passing in x86 machine" 
> means,
> in particular in relation to the HW configuration you are testing on x86.
>
> Yes, there are differences between x86 and ARM resources assignments,
> x86 tries to claim PCI resources as set-up by BIOS and assign them iff the
> claiming fails whereas on ARM (and that's done in the host bridge driver) FW
> configuration is always discarded and the kernel reassigns the whole PCI
> resource hierarchy entirely, but to help we need more data as I said above.
>

Thanks Lorenzo.
I will try to get x86 configuration details soon and post, as the setup is at 
different location.
On ARM we using pcie-xilinx-nwl.c configurations where we have where we have 
240MB BAR space
as per device tree documentation. (The kernel log I posted for testing purpose 
is using different address spaces,
if tests were successful, we will soon send patches for device tree)

Similar test case means, on x86 where we have multifunction device, where 1st 
function requests 1GB memory BAR's
and 2nd function requests 16byte memory BAR's, for 1st function BAR's are not 
assigned, but for 2nd function 16byte BAR assignment
is successful.

But on ARM similar test case fails to assign BAR's for both first function and 
second function.

What is the reason on ARM we are handling resource assignment in hierarchy 
format?
Because on ARM, in the above case due to lack of resources first function BAR 
assignment fails  which is fine but
even though we have enough resources for 16byte BAR allocations for second 
function, we are not assigning.


Thanks & Regards,
Bharat


This email and any attachments are intended for the sole use of the named 
recipient(s) and contain(s) confidential information that may be proprietary, 
privileged or copyrighted under applicable law. If you are not the intended 
recipient, do not read, copy, or forward this email message or any attachments. 
Delete this email message and any attachments immediately.

Re: Why does BIOS assign memory to 16 byte BAR

[Lorenzo Pieralisi]

On Wed, Jul 27, 2016 at 06:33:29AM +, Bharat Kumar Gogada wrote:
> > Your system host bridge: has resource
> > pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus
> > :00: root bus resource [mem 0x6-0x7 pref] then one pci
> > bridge:
> > pci :00:00.0
> > then :01:00.0 have four bars:
> > pci :01:00.0: BAR 0:  [mem size 0x4000] pci :01:00.0: BAR 4:
> > [mem size 0x0010 64bit] pci :01:00.0: BAR 2:  [mem size 0x0010]
> > pci :01:00.0: BAR 3:  [mem size 0x0010]
> > 
> > 
> > kernel need to get allocation for pci :00:00.0 at first
> > 
> > but can not find big enough space.
> > 
> > pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should
> > come from [mem 0xe010-0xefff], and that is less 1.5G.
> > 
> > so all children resource from pci :01:00.0 all fail.
> > 
> > 
> > please check if you modify your FPGA code to make pci :01:00.0
> > 
> > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> > 
> > or you can check if can increase root bus mmio range
> > 
> >  MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host
> > bridge to bus :00
> > 
> > to have more than 1.5G.
> > 
> Thanks Yinghai Lu.
> We see that similar test is passing in x86 machine, where function one
> requesting 1GB BAR's is failing, but function two requesting BAR's
> with 16byte is getting assigned BAR's.
> 
> To my knowledge on x86 BIOS assigns resources, or will kernel assign
> reosurces on x86 ?  If kernel does is there any difference between x86
> and arm64 resource assignment logic ?

We can't answer your question if you do not provide a full log
of x86 and ARM PCI configurations you are testing I am afraid.

It is also unclear to me what "a similar test is passing in x86
machine" means, in particular in relation to the HW configuration
you are testing on x86.

Yes, there are differences between x86 and ARM resources assignments,
x86 tries to claim PCI resources as set-up by BIOS and assign them
iff the claiming fails whereas on ARM (and that's done in the host
bridge driver) FW configuration is always discarded and the kernel
reassigns the whole PCI resource hierarchy entirely, but to help we
need more data as I said above.

I suspect the issue you are facing has to do as Yinghai mentioned
with the prefetchable memory window set-up, saying that "it works"
on x86 does not really help unless you provide data to debug it.

Thanks,
Lorenzo

Re: Why does BIOS assign memory to 16 byte BAR

[Lorenzo Pieralisi]

On Wed, Jul 27, 2016 at 06:33:29AM +, Bharat Kumar Gogada wrote:
> > Your system host bridge: has resource
> > pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus
> > :00: root bus resource [mem 0x6-0x7 pref] then one pci
> > bridge:
> > pci :00:00.0
> > then :01:00.0 have four bars:
> > pci :01:00.0: BAR 0:  [mem size 0x4000] pci :01:00.0: BAR 4:
> > [mem size 0x0010 64bit] pci :01:00.0: BAR 2:  [mem size 0x0010]
> > pci :01:00.0: BAR 3:  [mem size 0x0010]
> > 
> > 
> > kernel need to get allocation for pci :00:00.0 at first
> > 
> > but can not find big enough space.
> > 
> > pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should
> > come from [mem 0xe010-0xefff], and that is less 1.5G.
> > 
> > so all children resource from pci :01:00.0 all fail.
> > 
> > 
> > please check if you modify your FPGA code to make pci :01:00.0
> > 
> > BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> > 
> > or you can check if can increase root bus mmio range
> > 
> >  MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host
> > bridge to bus :00
> > 
> > to have more than 1.5G.
> > 
> Thanks Yinghai Lu.
> We see that similar test is passing in x86 machine, where function one
> requesting 1GB BAR's is failing, but function two requesting BAR's
> with 16byte is getting assigned BAR's.
> 
> To my knowledge on x86 BIOS assigns resources, or will kernel assign
> reosurces on x86 ?  If kernel does is there any difference between x86
> and arm64 resource assignment logic ?

We can't answer your question if you do not provide a full log
of x86 and ARM PCI configurations you are testing I am afraid.

It is also unclear to me what "a similar test is passing in x86
machine" means, in particular in relation to the HW configuration
you are testing on x86.

Yes, there are differences between x86 and ARM resources assignments,
x86 tries to claim PCI resources as set-up by BIOS and assign them
iff the claiming fails whereas on ARM (and that's done in the host
bridge driver) FW configuration is always discarded and the kernel
reassigns the whole PCI resource hierarchy entirely, but to help we
need more data as I said above.

I suspect the issue you are facing has to do as Yinghai mentioned
with the prefetchable memory window set-up, saying that "it works"
on x86 does not really help unless you provide data to debug it.

Thanks,
Lorenzo

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Your system host bridge: has resource
> pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus
> :00: root bus resource [mem 0x6-0x7 pref] then one pci
> bridge:
> pci :00:00.0
> then :01:00.0 have four bars:
> pci :01:00.0: BAR 0:  [mem size 0x4000] pci :01:00.0: BAR 4:
> [mem size 0x0010 64bit] pci :01:00.0: BAR 2:  [mem size 0x0010]
> pci :01:00.0: BAR 3:  [mem size 0x0010]
> 
> 
> kernel need to get allocation for pci :00:00.0 at first
> 
> but can not find big enough space.
> 
> pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should
> come from [mem 0xe010-0xefff], and that is less 1.5G.
> 
> so all children resource from pci :01:00.0 all fail.
> 
> 
> please check if you modify your FPGA code to make pci :01:00.0
> 
> BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> 
> or you can check if can increase root bus mmio range
> 
>  MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host
> bridge to bus :00
> 
> to have more than 1.5G.
> 
Thanks Yinghai Lu.
We see that similar test is passing in x86 machine, where function one 
requesting 1GB BAR's is failing,
but function two requesting BAR's with 16byte is getting assigned BAR's.

To my knowledge on x86 BIOS assigns resources, or will kernel assign reosurces 
on x86 ?
If kernel does is there any difference between x86 and arm64 resource 
assignment logic ?

Thanks & Regards,
Bharat

  

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Your system host bridge: has resource
> pci_bus :00: root bus resource [mem 0xe010-0xefff] pci_bus
> :00: root bus resource [mem 0x6-0x7 pref] then one pci
> bridge:
> pci :00:00.0
> then :01:00.0 have four bars:
> pci :01:00.0: BAR 0:  [mem size 0x4000] pci :01:00.0: BAR 4:
> [mem size 0x0010 64bit] pci :01:00.0: BAR 2:  [mem size 0x0010]
> pci :01:00.0: BAR 3:  [mem size 0x0010]
> 
> 
> kernel need to get allocation for pci :00:00.0 at first
> 
> but can not find big enough space.
> 
> pci :00:00.0: BAR 8: no space for [mem size 0x6000] as it should
> come from [mem 0xe010-0xefff], and that is less 1.5G.
> 
> so all children resource from pci :01:00.0 all fail.
> 
> 
> please check if you modify your FPGA code to make pci :01:00.0
> 
> BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.
> 
> or you can check if can increase root bus mmio range
> 
>  MEM 0xe010..0xefff -> 0xe010 nwl-pcie fd0e.pcie: PCI host
> bridge to bus :00
> 
> to have more than 1.5G.
> 
Thanks Yinghai Lu.
We see that similar test is passing in x86 machine, where function one 
requesting 1GB BAR's is failing,
but function two requesting BAR's with 16byte is getting assigned BAR's.

To my knowledge on x86 BIOS assigns resources, or will kernel assign reosurces 
on x86 ?
If kernel does is there any difference between x86 and arm64 resource 
assignment logic ?

Thanks & Regards,
Bharat

  

Re: Why does BIOS assign memory to 16 byte BAR

[Yinghai Lu]

On Tue, Jul 26, 2016 at 8:43 AM, Bharat Kumar Gogada
 wrote:
>> > > We have only one End point connected to our root port.
>> > >
>> > > Here is the log:
>> > > [2.319289] nwl-pcie fd0e.pcie: Link is UP
>> > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
>> > > [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus
>> > 00-
>> > > ff]
>> > > [2.319374]IO 0xe000..0xe000 -> 0x
>> > > [2.319415]   MEM 0xe010..0xefff -> 0xe010
>> > > [2.319431]   MEM 0x6..0x7 -> 0x6
>> > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
>> > > [2.319557] pci_bus :00: root bus resource [bus 00-ff]
>> > > [2.319573] pci_bus :00: root bus resource [io  0x-0x]
>> > > [2.319589] pci_bus :00: root bus resource [mem 0xe010-
>> > 0xefff]
>> > > [2.319606] pci_bus :00: root bus resource [mem 0x6-
>> > > 0x7 pref]
>> > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [2.319861] iommu: Adding device :00:00.0 to group 1
>> > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [2.320258] iommu: Adding device :01:00.0 to group 1
>> > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 
>> > > 0x6000]
>> > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size
>> 0x6000]
>> > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-
>> > 0xe01007ff
>> > > pref]
>> > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 
>> > > 0x4000]
>> > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size
>> 0x4000]
>> > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
>> > > 64bit]
>> > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size
>> 0x0010
>> > > 64bit]
>> > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 
>> > > 0x0010]
>> > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size
>> 0x0010]
>> > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 
>> > > 0x0010]
>> > > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size
>> 0x0010]
>> > > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]
>> > >
>> >
>> > What if the End Point has multiple function and say first function BAR
>> > assignment failed, will the kernel assign BAR's to second function on
>> > same bus and device ?

Your system host bridge: has resource
pci_bus :00: root bus resource [mem 0xe010-0xefff]
pci_bus :00: root bus resource [mem 0x6-0x7 pref]
then one pci bridge:
pci :00:00.0
then :01:00.0 have four bars:
pci :01:00.0: BAR 0:  [mem size 0x4000]
pci :01:00.0: BAR 4:  [mem size 0x0010 64bit]
pci :01:00.0: BAR 2:  [mem size 0x0010]
pci :01:00.0: BAR 3:  [mem size 0x0010]


kernel need to get allocation for pci :00:00.0 at first

but can not find big enough space.

pci :00:00.0: BAR 8: no space for [mem size 0x6000]
as it should come from [mem 0xe010-0xefff], and that is
less 1.5G.

so all children resource from pci :01:00.0 all fail.


please check if you modify your FPGA code to make pci :01:00.0

BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.

or you can check if can increase root bus mmio range

 MEM 0xe010..0xefff -> 0xe010
nwl-pcie fd0e.pcie: PCI host bridge to bus :00

to have more than 1.5G.

Thanks

Yinghai

Re: Why does BIOS assign memory to 16 byte BAR

[Yinghai Lu]

On Tue, Jul 26, 2016 at 8:43 AM, Bharat Kumar Gogada
 wrote:
>> > > We have only one End point connected to our root port.
>> > >
>> > > Here is the log:
>> > > [2.319289] nwl-pcie fd0e.pcie: Link is UP
>> > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
>> > > [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus
>> > 00-
>> > > ff]
>> > > [2.319374]IO 0xe000..0xe000 -> 0x
>> > > [2.319415]   MEM 0xe010..0xefff -> 0xe010
>> > > [2.319431]   MEM 0x6..0x7 -> 0x6
>> > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
>> > > [2.319557] pci_bus :00: root bus resource [bus 00-ff]
>> > > [2.319573] pci_bus :00: root bus resource [io  0x-0x]
>> > > [2.319589] pci_bus :00: root bus resource [mem 0xe010-
>> > 0xefff]
>> > > [2.319606] pci_bus :00: root bus resource [mem 0x6-
>> > > 0x7 pref]
>> > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [2.319861] iommu: Adding device :00:00.0 to group 1
>> > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same
>> > bus?
>> > > [2.320258] iommu: Adding device :01:00.0 to group 1
>> > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 
>> > > 0x6000]
>> > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size
>> 0x6000]
>> > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-
>> > 0xe01007ff
>> > > pref]
>> > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 
>> > > 0x4000]
>> > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size
>> 0x4000]
>> > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
>> > > 64bit]
>> > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size
>> 0x0010
>> > > 64bit]
>> > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 
>> > > 0x0010]
>> > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size
>> 0x0010]
>> > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 
>> > > 0x0010]
>> > > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size
>> 0x0010]
>> > > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]
>> > >
>> >
>> > What if the End Point has multiple function and say first function BAR
>> > assignment failed, will the kernel assign BAR's to second function on
>> > same bus and device ?

Your system host bridge: has resource
pci_bus :00: root bus resource [mem 0xe010-0xefff]
pci_bus :00: root bus resource [mem 0x6-0x7 pref]
then one pci bridge:
pci :00:00.0
then :01:00.0 have four bars:
pci :01:00.0: BAR 0:  [mem size 0x4000]
pci :01:00.0: BAR 4:  [mem size 0x0010 64bit]
pci :01:00.0: BAR 2:  [mem size 0x0010]
pci :01:00.0: BAR 3:  [mem size 0x0010]


kernel need to get allocation for pci :00:00.0 at first

but can not find big enough space.

pci :00:00.0: BAR 8: no space for [mem size 0x6000]
as it should come from [mem 0xe010-0xefff], and that is
less 1.5G.

so all children resource from pci :01:00.0 all fail.


please check if you modify your FPGA code to make pci :01:00.0

BAR 0, and BAR 4 to use 64bit pref instead non-pref mmio.

or you can check if can increase root bus mmio range

 MEM 0xe010..0xefff -> 0xe010
nwl-pcie fd0e.pcie: PCI host bridge to bus :00

to have more than 1.5G.

Thanks

Yinghai

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Adding yinghai lu.

> > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > > >
> > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > > Hi Bharat,
> > > > >
> > > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm observing that on x86 BIOS successfully assigns memory if
> > > > > > an End point requests BAR of size 16byte.
> > > > > >
> > > > > > But as per Spec:
> > > > > > The minimum memory address range requested by a BAR is 128
> > bytes.
> > > > >
> > > > > Can you provide the spec reference for this?  I don't see it in
> > > > > PCI r3.0.
> > > > >
> > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as
> > > > > writable, which would correspond to a minimum size of 16 bytes.
> > > >
> > > > The reference above is to the conventional PCI spec.  I happened
> > > > to trip
> > > over
> > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint,
> > > > "the
> > > minimum
> > > > memory address range requested by a BAR is 128 bytes."
> > > >
> > > > I don't think linux currently enforces this minimum.
> > > >
> > >
> > > Hi Bjorn Thanks for the reply.
> > >
> > > Here is what the issue we are seeing.
> > >
> > > We have total memory for BAR's on our SoC of 256 MB.
> > > When an End Point request individually 16 byte BAR's our root port
> > > assigns memory to BAR's successfully.
> > >
> > > But if I have an End point which has 4 BAR's each 32 bit and request
> > > as
> > > following:
> > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We
> > > are running this as part of SIG compliance test case) 2nd BAR
> > > requests 1MB and other 2 BAR's request 16byte, but these are not
> > > getting BAR's assigned. (Even though BAR space is available, since
> > > 1GB
> > failed,
> > > We have 256 MB still)
> > >
> > > We have only one End point connected to our root port.
> > >
> > > Here is the log:
> > > [2.319289] nwl-pcie fd0e.pcie: Link is UP
> > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
> > > [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus
> > 00-
> > > ff]
> > > [2.319374]IO 0xe000..0xe000 -> 0x
> > > [2.319415]   MEM 0xe010..0xefff -> 0xe010
> > > [2.319431]   MEM 0x6..0x7 -> 0x6
> > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
> > > [2.319557] pci_bus :00: root bus resource [bus 00-ff]
> > > [2.319573] pci_bus :00: root bus resource [io  0x-0x]
> > > [2.319589] pci_bus :00: root bus resource [mem 0xe010-
> > 0xefff]
> > > [2.319606] pci_bus :00: root bus resource [mem 0x6-
> > > 0x7 pref]
> > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [2.319861] iommu: Adding device :00:00.0 to group 1
> > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [2.320258] iommu: Adding device :01:00.0 to group 1
> > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
> > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size
> 0x6000]
> > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-
> > 0xe01007ff
> > > pref]
> > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
> > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size
> 0x4000]
> > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
> > > 64bit]
> > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size
> 0x0010
> > > 64bit]
> > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
> > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size
> 0x0010]
> > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
> > > [2.320493] pci :01:00.0: BAR 3: fai

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Adding yinghai lu.

> > > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > > >
> > > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > > Hi Bharat,
> > > > >
> > > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm observing that on x86 BIOS successfully assigns memory if
> > > > > > an End point requests BAR of size 16byte.
> > > > > >
> > > > > > But as per Spec:
> > > > > > The minimum memory address range requested by a BAR is 128
> > bytes.
> > > > >
> > > > > Can you provide the spec reference for this?  I don't see it in
> > > > > PCI r3.0.
> > > > >
> > > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as
> > > > > writable, which would correspond to a minimum size of 16 bytes.
> > > >
> > > > The reference above is to the conventional PCI spec.  I happened
> > > > to trip
> > > over
> > > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint,
> > > > "the
> > > minimum
> > > > memory address range requested by a BAR is 128 bytes."
> > > >
> > > > I don't think linux currently enforces this minimum.
> > > >
> > >
> > > Hi Bjorn Thanks for the reply.
> > >
> > > Here is what the issue we are seeing.
> > >
> > > We have total memory for BAR's on our SoC of 256 MB.
> > > When an End Point request individually 16 byte BAR's our root port
> > > assigns memory to BAR's successfully.
> > >
> > > But if I have an End point which has 4 BAR's each 32 bit and request
> > > as
> > > following:
> > > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We
> > > are running this as part of SIG compliance test case) 2nd BAR
> > > requests 1MB and other 2 BAR's request 16byte, but these are not
> > > getting BAR's assigned. (Even though BAR space is available, since
> > > 1GB
> > failed,
> > > We have 256 MB still)
> > >
> > > We have only one End point connected to our root port.
> > >
> > > Here is the log:
> > > [2.319289] nwl-pcie fd0e.pcie: Link is UP
> > > [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
> > > [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus
> > 00-
> > > ff]
> > > [2.319374]IO 0xe000..0xe000 -> 0x
> > > [2.319415]   MEM 0xe010..0xefff -> 0xe010
> > > [2.319431]   MEM 0x6..0x7 -> 0x6
> > > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
> > > [2.319557] pci_bus :00: root bus resource [bus 00-ff]
> > > [2.319573] pci_bus :00: root bus resource [io  0x-0x]
> > > [2.319589] pci_bus :00: root bus resource [mem 0xe010-
> > 0xefff]
> > > [2.319606] pci_bus :00: root bus resource [mem 0x6-
> > > 0x7 pref]
> > > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [2.319861] iommu: Adding device :00:00.0 to group 1
> > > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same
> > bus?
> > > [2.320258] iommu: Adding device :01:00.0 to group 1
> > > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
> > > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size
> 0x6000]
> > > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-
> > 0xe01007ff
> > > pref]
> > > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
> > > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size
> 0x4000]
> > > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
> > > 64bit]
> > > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size
> 0x0010
> > > 64bit]
> > > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
> > > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size
> 0x0010]
> > > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
> > > [2.320493] pci :01:00.0: BAR 3: fai

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: RE: Why does BIOS assign memory to 16 byte BAR
> >
> > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > >
> > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > Hi Bharat,
> > > >
> > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > > > Hi,
> > > > >
> > > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > > point requests BAR of size 16byte.
> > > > >
> > > > > But as per Spec:
> > > > > The minimum memory address range requested by a BAR is 128
> bytes.
> > > >
> > > > Can you provide the spec reference for this?  I don't see it in PCI
> > > > r3.0.
> > > >
> > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > > which would correspond to a minimum size of 16 bytes.
> > >
> > > The reference above is to the conventional PCI spec.  I happened to trip
> > over
> > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> > minimum
> > > memory address range requested by a BAR is 128 bytes."
> > >
> > > I don't think linux currently enforces this minimum.
> > >
> >
> > Hi Bjorn Thanks for the reply.
> >
> > Here is what the issue we are seeing.
> >
> > We have total memory for BAR's on our SoC of 256 MB.
> > When an End Point request individually 16 byte BAR's our root port assigns
> > memory to BAR's successfully.
> >
> > But if I have an End point which has 4 BAR's each 32 bit and request as
> > following:
> > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> > running this as part of SIG compliance test case)
> > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> > getting BAR's assigned. (Even though BAR space is available, since 1GB
> failed,
> > We have 256 MB still)
> >
> > We have only one End point connected to our root port.
> >
> > Here is the log:
> > [2.319289] nwl-pcie fd0e.pcie: Link is UP
> > [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
> > [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus
> 00-
> > ff]
> > [2.319374]IO 0xe000..0xe000 -> 0x
> > [2.319415]   MEM 0xe010..0xefff -> 0xe010
> > [2.319431]   MEM 0x6..0x7 -> 0x6
> > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
> > [2.319557] pci_bus :00: root bus resource [bus 00-ff]
> > [2.319573] pci_bus :00: root bus resource [io  0x-0x]
> > [2.319589] pci_bus :00: root bus resource [mem 0xe010-
> 0xefff]
> > [2.319606] pci_bus :00: root bus resource [mem 0x6-
> > 0x7 pref]
> > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [2.319861] iommu: Adding device :00:00.0 to group 1
> > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [2.320258] iommu: Adding device :01:00.0 to group 1
> > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
> > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 
> > 0x6000]
> > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-
> 0xe01007ff
> > pref]
> > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
> > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 
> > 0x4000]
> > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
> > 64bit]
> > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 
> > 0x0010
> > 64bit]
> > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
> > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 
> > 0x0010]
> > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
> > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 
> > 0x0010]
> > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]
> >
> > Please let me know, what might might be the issue.
> >
> 
> Adding to the above will kernel allocate other memory BARs to an End Point
> if
> one BAR assignment fails ?
> 
> What if the End Point has multiple function and say first function BAR
> assignment failed,
> will the kernel assign BAR's to second function on same bus and device ?
> 
After debugging in function pci_bus_alloc_from_region we have two resources,
One which is taken as parameter (let's say resA), other is from  
pci_bus_for_each_resource (let's say resB).

The resA contains different start address from resB, I see that resB is 
obtained from device resource 
from bus->resource, but im unable to understand how resA (start address & size) 
gets its data. 

Can any one help me understand this so that I might know reason why 16byte BAR 
allocation failing after
1GB request.

Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: RE: Why does BIOS assign memory to 16 byte BAR
> >
> > > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> > >
> > > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > > Hi Bharat,
> > > >
> > > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > > > Hi,
> > > > >
> > > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > > point requests BAR of size 16byte.
> > > > >
> > > > > But as per Spec:
> > > > > The minimum memory address range requested by a BAR is 128
> bytes.
> > > >
> > > > Can you provide the spec reference for this?  I don't see it in PCI
> > > > r3.0.
> > > >
> > > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > > which would correspond to a minimum size of 16 bytes.
> > >
> > > The reference above is to the conventional PCI spec.  I happened to trip
> > over
> > > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> > minimum
> > > memory address range requested by a BAR is 128 bytes."
> > >
> > > I don't think linux currently enforces this minimum.
> > >
> >
> > Hi Bjorn Thanks for the reply.
> >
> > Here is what the issue we are seeing.
> >
> > We have total memory for BAR's on our SoC of 256 MB.
> > When an End Point request individually 16 byte BAR's our root port assigns
> > memory to BAR's successfully.
> >
> > But if I have an End point which has 4 BAR's each 32 bit and request as
> > following:
> > When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> > running this as part of SIG compliance test case)
> > 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> > getting BAR's assigned. (Even though BAR space is available, since 1GB
> failed,
> > We have 256 MB still)
> >
> > We have only one End point connected to our root port.
> >
> > Here is the log:
> > [2.319289] nwl-pcie fd0e.pcie: Link is UP
> > [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
> > [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus
> 00-
> > ff]
> > [2.319374]IO 0xe000..0xe000 -> 0x
> > [2.319415]   MEM 0xe010..0xefff -> 0xe010
> > [2.319431]   MEM 0x6..0x7 -> 0x6
> > [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
> > [2.319557] pci_bus :00: root bus resource [bus 00-ff]
> > [2.319573] pci_bus :00: root bus resource [io  0x-0x]
> > [2.319589] pci_bus :00: root bus resource [mem 0xe010-
> 0xefff]
> > [2.319606] pci_bus :00: root bus resource [mem 0x6-
> > 0x7 pref]
> > [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [2.319861] iommu: Adding device :00:00.0 to group 1
> > [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same
> bus?
> > [2.320258] iommu: Adding device :01:00.0 to group 1
> > [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
> > [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 
> > 0x6000]
> > [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-
> 0xe01007ff
> > pref]
> > [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
> > [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 
> > 0x4000]
> > [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
> > 64bit]
> > [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 
> > 0x0010
> > 64bit]
> > [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
> > [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 
> > 0x0010]
> > [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
> > [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 
> > 0x0010]
> > [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]
> >
> > Please let me know, what might might be the issue.
> >
> 
> Adding to the above will kernel allocate other memory BARs to an End Point
> if
> one BAR assignment fails ?
> 
> What if the End Point has multiple function and say first function BAR
> assignment failed,
> will the kernel assign BAR's to second function on same bus and device ?
> 
After debugging in function pci_bus_alloc_from_region we have two resources,
One which is taken as parameter (let's say resA), other is from  
pci_bus_for_each_resource (let's say resB).

The resA contains different start address from resB, I see that resB is 
obtained from device resource 
from bus->resource, but im unable to understand how resA (start address & size) 
gets its data. 

Can any one help me understand this so that I might know reason why 16byte BAR 
allocation failing after
1GB request.

Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> >
> > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > Hi Bharat,
> > >
> > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > > Hi,
> > > >
> > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > point requests BAR of size 16byte.
> > > >
> > > > But as per Spec:
> > > > The minimum memory address range requested by a BAR is 128 bytes.
> > >
> > > Can you provide the spec reference for this?  I don't see it in PCI
> > > r3.0.
> > >
> > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > which would correspond to a minimum size of 16 bytes.
> >
> > The reference above is to the conventional PCI spec.  I happened to trip
> over
> > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> minimum
> > memory address range requested by a BAR is 128 bytes."
> >
> > I don't think linux currently enforces this minimum.
> >
> 
> Hi Bjorn Thanks for the reply.
> 
> Here is what the issue we are seeing.
> 
> We have total memory for BAR's on our SoC of 256 MB.
> When an End Point request individually 16 byte BAR's our root port assigns
> memory to BAR's successfully.
> 
> But if I have an End point which has 4 BAR's each 32 bit and request as
> following:
> When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> running this as part of SIG compliance test case)
> 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
> We have 256 MB still)
> 
> We have only one End point connected to our root port.
> 
> Here is the log:
> [2.319289] nwl-pcie fd0e.pcie: Link is UP
> [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
> [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus 00-
> ff]
> [2.319374]IO 0xe000..0xe000 -> 0x
> [2.319415]   MEM 0xe010..0xefff -> 0xe010
> [2.319431]   MEM 0x6..0x7 -> 0x6
> [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
> [2.319557] pci_bus :00: root bus resource [bus 00-ff]
> [2.319573] pci_bus :00: root bus resource [io  0x-0x]
> [2.319589] pci_bus :00: root bus resource [mem 0xe010-0xefff]
> [2.319606] pci_bus :00: root bus resource [mem 0x6-
> 0x7 pref]
> [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same bus?
> [2.319861] iommu: Adding device :00:00.0 to group 1
> [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same bus?
> [2.320258] iommu: Adding device :01:00.0 to group 1
> [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
> [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 0x6000]
> [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-0xe01007ff
> pref]
> [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
> [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 0x4000]
> [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
> 64bit]
> [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 0x0010
> 64bit]
> [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
> [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 0x0010]
> [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
> [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 0x0010]
> [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]
> 
> Please let me know, what might might be the issue.
> 

Adding to the above will kernel allocate other memory BARs to an End Point if 
one BAR assignment fails ?

What if the End Point has multiple function and say first function BAR 
assignment failed,
will the kernel assign BAR's to second function on same bus and device ?  

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: RE: Why does BIOS assign memory to 16 byte BAR
> 
> > Subject: Re: Why does BIOS assign memory to 16 byte BAR
> >
> > On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > > Hi Bharat,
> > >
> > > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > > Hi,
> > > >
> > > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > > point requests BAR of size 16byte.
> > > >
> > > > But as per Spec:
> > > > The minimum memory address range requested by a BAR is 128 bytes.
> > >
> > > Can you provide the spec reference for this?  I don't see it in PCI
> > > r3.0.
> > >
> > > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > > which would correspond to a minimum size of 16 bytes.
> >
> > The reference above is to the conventional PCI spec.  I happened to trip
> over
> > a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the
> minimum
> > memory address range requested by a BAR is 128 bytes."
> >
> > I don't think linux currently enforces this minimum.
> >
> 
> Hi Bjorn Thanks for the reply.
> 
> Here is what the issue we are seeing.
> 
> We have total memory for BAR's on our SoC of 256 MB.
> When an End Point request individually 16 byte BAR's our root port assigns
> memory to BAR's successfully.
> 
> But if I have an End point which has 4 BAR's each 32 bit and request as
> following:
> When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are
> running this as part of SIG compliance test case)
> 2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not
> getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
> We have 256 MB still)
> 
> We have only one End point connected to our root port.
> 
> Here is the log:
> [2.319289] nwl-pcie fd0e.pcie: Link is UP
> [2.319332] PCI host bridge /amba/pcie@fd0e ranges:
> [2.319349]   No bus range found for /amba/pcie@fd0e, using [bus 00-
> ff]
> [2.319374]IO 0xe000..0xe000 -> 0x
> [2.319415]   MEM 0xe010..0xefff -> 0xe010
> [2.319431]   MEM 0x6..0x7 -> 0x6
> [2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
> [2.319557] pci_bus :00: root bus resource [bus 00-ff]
> [2.319573] pci_bus :00: root bus resource [io  0x-0x]
> [2.319589] pci_bus :00: root bus resource [mem 0xe010-0xefff]
> [2.319606] pci_bus :00: root bus resource [mem 0x6-
> 0x7 pref]
> [2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same bus?
> [2.319861] iommu: Adding device :00:00.0 to group 1
> [2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same bus?
> [2.320258] iommu: Adding device :01:00.0 to group 1
> [2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
> [2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 0x6000]
> [2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-0xe01007ff
> pref]
> [2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
> [2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 0x4000]
> [2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010
> 64bit]
> [2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 0x0010
> 64bit]
> [2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
> [2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 0x0010]
> [2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
> [2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 0x0010]
> [2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]
> 
> Please let me know, what might might be the issue.
> 

Adding to the above will kernel allocate other memory BARs to an End Point if 
one BAR assignment fails ?

What if the End Point has multiple function and say first function BAR 
assignment failed,
will the kernel assign BAR's to second function on same bus and device ?  

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: Re: Why does BIOS assign memory to 16 byte BAR
> 
> On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > Hi Bharat,
> >
> > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > Hi,
> > >
> > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > point requests BAR of size 16byte.
> > >
> > > But as per Spec:
> > > The minimum memory address range requested by a BAR is 128 bytes.
> >
> > Can you provide the spec reference for this?  I don't see it in PCI
> > r3.0.
> >
> > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > which would correspond to a minimum size of 16 bytes.
> 
> The reference above is to the conventional PCI spec.  I happened to trip over
> a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the 
> minimum
> memory address range requested by a BAR is 128 bytes."
> 
> I don't think linux currently enforces this minimum.
> 

Hi Bjorn Thanks for the reply.

Here is what the issue we are seeing.

We have total memory for BAR's on our SoC of 256 MB.
When an End Point request individually 16 byte BAR's our root port assigns 
memory to BAR's successfully.

But if I have an End point which has 4 BAR's each 32 bit and request as 
following:
When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are running 
this as part of SIG compliance test case)
2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not 
getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
We have 256 MB still)

We have only one End point connected to our root port.

Here is the log:
[2.319289] nwl-pcie fd0e.pcie: Link is UP
[2.319332] PCI host bridge /amba/pcie@fd0e ranges:
[2.319349]   No bus range found for /amba/pcie@fd0e, using [bus 00-ff]
[2.319374]IO 0xe000..0xe000 -> 0x
[2.319415]   MEM 0xe010..0xefff -> 0xe010
[2.319431]   MEM 0x6..0x7 -> 0x6
[2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
[2.319557] pci_bus :00: root bus resource [bus 00-ff]
[2.319573] pci_bus :00: root bus resource [io  0x-0x]
[2.319589] pci_bus :00: root bus resource [mem 0xe010-0xefff]
[2.319606] pci_bus :00: root bus resource [mem 0x6-0x7 
pref]
[2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same bus?
[2.319861] iommu: Adding device :00:00.0 to group 1
[2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same bus?
[2.320258] iommu: Adding device :01:00.0 to group 1
[2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
[2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 0x6000]
[2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-0xe01007ff 
pref]
[2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
[2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 0x4000]
[2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 64bit]
[2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 0x0010 
64bit]
[2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
[2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 0x0010]
[2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
[2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 0x0010]
[2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]

Please let me know, what might might be the issue.

Thanks & Regards,
Bharat

RE: Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

> Subject: Re: Why does BIOS assign memory to 16 byte BAR
> 
> On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> > Hi Bharat,
> >
> > On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > > Hi,
> > >
> > > I'm observing that on x86 BIOS successfully assigns memory if an End
> > > point requests BAR of size 16byte.
> > >
> > > But as per Spec:
> > > The minimum memory address range requested by a BAR is 128 bytes.
> >
> > Can you provide the spec reference for this?  I don't see it in PCI
> > r3.0.
> >
> > PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> > which would correspond to a minimum size of 16 bytes.
> 
> The reference above is to the conventional PCI spec.  I happened to trip over
> a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express endpoint, "the 
> minimum
> memory address range requested by a BAR is 128 bytes."
> 
> I don't think linux currently enforces this minimum.
> 

Hi Bjorn Thanks for the reply.

Here is what the issue we are seeing.

We have total memory for BAR's on our SoC of 256 MB.
When an End Point request individually 16 byte BAR's our root port assigns 
memory to BAR's successfully.

But if I have an End point which has 4 BAR's each 32 bit and request as 
following:
When 1st BAR requests 1GB BAR it fails due to lack of memory. (We are running 
this as part of SIG compliance test case)
2nd BAR requests 1MB and other 2 BAR's request 16byte, but these are not 
getting BAR's assigned. (Even though BAR space is available, since 1GB failed,
We have 256 MB still)

We have only one End point connected to our root port.

Here is the log:
[2.319289] nwl-pcie fd0e.pcie: Link is UP
[2.319332] PCI host bridge /amba/pcie@fd0e ranges:
[2.319349]   No bus range found for /amba/pcie@fd0e, using [bus 00-ff]
[2.319374]IO 0xe000..0xe000 -> 0x
[2.319415]   MEM 0xe010..0xefff -> 0xe010
[2.319431]   MEM 0x6..0x7 -> 0x6
[2.319539] nwl-pcie fd0e.pcie: PCI host bridge to bus :00
[2.319557] pci_bus :00: root bus resource [bus 00-ff]
[2.319573] pci_bus :00: root bus resource [io  0x-0x]
[2.319589] pci_bus :00: root bus resource [mem 0xe010-0xefff]
[2.319606] pci_bus :00: root bus resource [mem 0x6-0x7 
pref]
[2.319845] pci :00:00.0: cannot attach to SMMU, is it on the same bus?
[2.319861] iommu: Adding device :00:00.0 to group 1
[2.320243] pci :01:00.0: cannot attach to SMMU, is it on the same bus?
[2.320258] iommu: Adding device :01:00.0 to group 1
[2.320313] pci :00:00.0: BAR 8: no space for [mem size 0x6000]
[2.320331] pci :00:00.0: BAR 8: failed to assign [mem size 0x6000]
[2.320349] pci :00:00.0: BAR 6: assigned [mem 0xe010-0xe01007ff 
pref]
[2.320374] pci :01:00.0: BAR 0: no space for [mem size 0x4000]
[2.320390] pci :01:00.0: BAR 0: failed to assign [mem size 0x4000]
[2.320407] pci :01:00.0: BAR 4: no space for [mem size 0x0010 64bit]
[2.320423] pci :01:00.0: BAR 4: failed to assign [mem size 0x0010 
64bit]
[2.320446] pci :01:00.0: BAR 2: no space for [mem size 0x0010]
[2.320461] pci :01:00.0: BAR 2: failed to assign [mem size 0x0010]
[2.320477] pci :01:00.0: BAR 3: no space for [mem size 0x0010]
[2.320493] pci :01:00.0: BAR 3: failed to assign [mem size 0x0010]
[2.320509] pci :00:00.0: PCI bridge to [bus 01-0c]

Please let me know, what might might be the issue.

Thanks & Regards,
Bharat

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> Hi Bharat,
> 
> On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > Hi,
> > 
> > I'm observing that on x86 BIOS successfully assigns memory if an End point 
> > requests
> > BAR of size 16byte.
> > 
> > But as per Spec:
> > The minimum memory address range requested by a BAR is 128 bytes.
> 
> Can you provide the spec reference for this?  I don't see it in PCI
> r3.0.
>
> PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> which would correspond to a minimum size of 16 bytes.

The reference above is to the conventional PCI spec.  I happened to
trip over a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express
endpoint, "the minimum memory address range requested by a BAR is 128
bytes."

I don't think linux currently enforces this minimum.

Bjorn

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

On Fri, Jul 22, 2016 at 10:15:46AM -0500, Bjorn Helgaas wrote:
> Hi Bharat,
> 
> On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> > Hi,
> > 
> > I'm observing that on x86 BIOS successfully assigns memory if an End point 
> > requests
> > BAR of size 16byte.
> > 
> > But as per Spec:
> > The minimum memory address range requested by a BAR is 128 bytes.
> 
> Can you provide the spec reference for this?  I don't see it in PCI
> r3.0.
>
> PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
> which would correspond to a minimum size of 16 bytes.

The reference above is to the conventional PCI spec.  I happened to
trip over a note in PCIe r3.0, sec 1.3.2.2, that for a PCI Express
endpoint, "the minimum memory address range requested by a BAR is 128
bytes."

I don't think linux currently enforces this minimum.

Bjorn

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

Hi Bharat,

On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> Hi,
> 
> I'm observing that on x86 BIOS successfully assigns memory if an End point 
> requests
> BAR of size 16byte.
> 
> But as per Spec:
> The minimum memory address range requested by a BAR is 128 bytes.

Can you provide the spec reference for this?  I don't see it in PCI
r3.0.

PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
which would correspond to a minimum size of 16 bytes.

> Why BIOS is successfully allocating region to 16 byte BAR requests?

If you want to know why the BIOS does this, you'd have to ask the BIOS
writer.

If you were to ask about *Linux*, I'd say we should not gratuitously
enforce things in the spec merely for the sake of being compliant with
the letter of the spec.  We should enforce things that are logically
required or that are required for compatibility, security, etc.  But
if the spec contained an arbitrary restriction  for no good reason,
and real-world hardware violated that restriction, I'd say Linux
should accommodate the hardware.

Bjorn

Re: Why does BIOS assign memory to 16 byte BAR

[Bjorn Helgaas]

Hi Bharat,

On Fri, Jul 22, 2016 at 09:24:22AM +, Bharat Kumar Gogada wrote:
> Hi,
> 
> I'm observing that on x86 BIOS successfully assigns memory if an End point 
> requests
> BAR of size 16byte.
> 
> But as per Spec:
> The minimum memory address range requested by a BAR is 128 bytes.

Can you provide the spec reference for this?  I don't see it in PCI
r3.0.

PCI r3.0, sec 6.2.5.1, shows bits 4-31 of a memory BAR as writable,
which would correspond to a minimum size of 16 bytes.

> Why BIOS is successfully allocating region to 16 byte BAR requests?

If you want to know why the BIOS does this, you'd have to ask the BIOS
writer.

If you were to ask about *Linux*, I'd say we should not gratuitously
enforce things in the spec merely for the sake of being compliant with
the letter of the spec.  We should enforce things that are logically
required or that are required for compatibility, security, etc.  But
if the spec contained an arbitrary restriction  for no good reason,
and real-world hardware violated that restriction, I'd say Linux
should accommodate the hardware.

Bjorn

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Hi,

I'm observing that on x86 BIOS successfully assigns memory if an End point 
requests
BAR of size 16byte.

But as per Spec:
The minimum memory address range requested by a BAR is 128 bytes.

Why BIOS is successfully allocating region to 16 byte BAR requests?

Please let me know if my observation is wrong.

Regards,
Bharat



This email and any attachments are intended for the sole use of the named 
recipient(s) and contain(s) confidential information that may be proprietary, 
privileged or copyrighted under applicable law. If you are not the intended 
recipient, do not read, copy, or forward this email message or any attachments. 
Delete this email message and any attachments immediately.

Why does BIOS assign memory to 16 byte BAR

[Bharat Kumar Gogada]

Hi,

I'm observing that on x86 BIOS successfully assigns memory if an End point 
requests
BAR of size 16byte.

But as per Spec:
The minimum memory address range requested by a BAR is 128 bytes.

Why BIOS is successfully allocating region to 16 byte BAR requests?

Please let me know if my observation is wrong.

Regards,
Bharat



This email and any attachments are intended for the sole use of the named 
recipient(s) and contain(s) confidential information that may be proprietary, 
privileged or copyrighted under applicable law. If you are not the intended 
recipient, do not read, copy, or forward this email message or any attachments. 
Delete this email message and any attachments immediately.