RE: Neophyte questions about PCIe
From: Mason > Sent: 14 March 2017 12:06 > On 14/03/2017 11:23, David Laight wrote: > > > Mason wrote: > > > >> I'd like to push support for this PCIe controller upstream. > >> > >> Is the code I posted on the right track? > >> Maybe I can post a RFC patch tomorrow? > > > > I think you need to resolve the problem of config space (and IO) cycles > > before the driver can be deemed usable. > > You're alluding to the (unfortunate) muxing of config and mem spaces > on my controller, where concurrent accesses by two different threads > would blow the system up. > > You've suggested sending IPIs in the config space accessor, in order > to prevent other CPUs from starting a mem access. But this doesn't > help if a mem access is already in flight, AFAIU. > > I fear there is nothing that can be done in SW, short of rewriting > drivers such that mem space accesses are handled by a driver-specific > call-back which could take care of all required locking. > > AFAICT, my only (reasonable) option is putting a big fat warning > in the code, and pray that concurrent accesses never happen. > (I'll test with a storage stress test on a USB3 drive.) Since this is a host controller and you want to be able to use standard PCIe cards (with their standard drivers) you need to fix the hardware or do some kind of software workaround that is only in the config space code. You cannot assume than config space cycles won't happen at the same time as other accesses. While most driver accesses should use either the iowrite32() or writel() families of functions I'm pretty sure they don't have inter-cpu locking. (They might have annoying, excessive barriers.) But it is also legal for drivers to allow pcie memory space be mmapped directly into a processes address space. You have no control over the bus cycles that generates. This isn't common, but is done. We use it for bit-banging serial eeproms. David -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 14/03/2017 11:23, David Laight wrote: > Mason wrote: > >> I'd like to push support for this PCIe controller upstream. >> >> Is the code I posted on the right track? >> Maybe I can post a RFC patch tomorrow? > > I think you need to resolve the problem of config space (and IO) cycles > before the driver can be deemed usable. You're alluding to the (unfortunate) muxing of config and mem spaces on my controller, where concurrent accesses by two different threads would blow the system up. You've suggested sending IPIs in the config space accessor, in order to prevent other CPUs from starting a mem access. But this doesn't help if a mem access is already in flight, AFAIU. I fear there is nothing that can be done in SW, short of rewriting drivers such that mem space accesses are handled by a driver-specific call-back which could take care of all required locking. AFAICT, my only (reasonable) option is putting a big fat warning in the code, and pray that concurrent accesses never happen. (I'll test with a storage stress test on a USB3 drive.) In parallel, I'm trying to convince management that the HW needs fixing ASAP. Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Neophyte questions about PCIe
From: Mason > Sent: 13 March 2017 21:58 ... > I'd like to push support for this PCIe controller upstream. > > Is the code I posted on the right track? > Maybe I can post a RFC patch tomorrow? I think you need to resolve the problem of config space (and IO) cycles before the driver can be deemed usable. David -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On Mon, Mar 13, 2017 at 10:57:48PM +0100, Mason wrote:
> On 13/03/2017 22:40, Bjorn Helgaas wrote:
>
> > On Sat, Mar 11, 2017 at 11:57:56AM +0100, Mason wrote:
> >
> >> On 10/03/2017 18:49, Mason wrote:
> >>
> >>> static void tango_pcie_bar_quirk(struct pci_dev *dev)
> >>> {
> >>> struct pci_bus *bus = dev->bus;
> >>>
> >>> printk("%s: bus=%d devfn=%d\n", __func__, bus->number, dev->devfn);
> >>>
> >>> pci_write_config_dword(dev, PCI_BASE_ADDRESS_0, 0x8004);
> >>> }
> >>> DECLARE_PCI_FIXUP_FINAL(0x1105, PCI_ANY_ID, tango_pcie_bar_quirk);
> >>
> >> And this is where the elusive "black magic" happens.
> >>
> >> Is it "safe" to configure a BAR behind Linux's back?
> >
> > No. Linux maintains a struct resource for every BAR. This quirk
> > makes the BAR out of sync with the resource, so Linux no longer has an
> > accurate idea of what bus address space is consumed and what is
> > available.
>
> Even when Linux is not able to map the BAR, since it's too
> large to fit in the mem window?
I don't think there's much point in advertising a BAR that isn't
really a BAR and making assumptions about how Linux will handle it.
So my answer remains "No, I don't think it's a good idea to change a
BAR behind the back of the PCI core. It might work now, but there's
no guarantee it will keep working."
> > Normally a BAR is for mapping device registers into PCI bus address
> > space. If this BAR controls how the RC forwards PCI DMA transactions
> > to RAM, then it's not really a BAR and you should prevent Linux from
> > seeing it as a BAR. You could do this by special-casing it in the
> > config accessor so reads return 0 and writes are dropped. Then you
> > could write the register in your host bridge driver safely because the
> > PCI core would think the BAR is not implemented.
>
> In fact, that's what I used to do in a previous version :-)
>
> I'd like to push support for this PCIe controller upstream.
>
> Is the code I posted on the right track?
> Maybe I can post a RFC patch tomorrow?
No need to ask before posting a patch :)
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 13/03/2017 22:40, Bjorn Helgaas wrote:
> On Sat, Mar 11, 2017 at 11:57:56AM +0100, Mason wrote:
>
>> On 10/03/2017 18:49, Mason wrote:
>>
>>> static void tango_pcie_bar_quirk(struct pci_dev *dev)
>>> {
>>> struct pci_bus *bus = dev->bus;
>>>
>>> printk("%s: bus=%d devfn=%d\n", __func__, bus->number, dev->devfn);
>>>
>>> pci_write_config_dword(dev, PCI_BASE_ADDRESS_0, 0x8004);
>>> }
>>> DECLARE_PCI_FIXUP_FINAL(0x1105, PCI_ANY_ID, tango_pcie_bar_quirk);
>>
>> And this is where the elusive "black magic" happens.
>>
>> Is it "safe" to configure a BAR behind Linux's back?
>
> No. Linux maintains a struct resource for every BAR. This quirk
> makes the BAR out of sync with the resource, so Linux no longer has an
> accurate idea of what bus address space is consumed and what is
> available.
Even when Linux is not able to map the BAR, since it's too
large to fit in the mem window?
> Normally a BAR is for mapping device registers into PCI bus address
> space. If this BAR controls how the RC forwards PCI DMA transactions
> to RAM, then it's not really a BAR and you should prevent Linux from
> seeing it as a BAR. You could do this by special-casing it in the
> config accessor so reads return 0 and writes are dropped. Then you
> could write the register in your host bridge driver safely because the
> PCI core would think the BAR is not implemented.
In fact, that's what I used to do in a previous version :-)
I'd like to push support for this PCIe controller upstream.
Is the code I posted on the right track?
Maybe I can post a RFC patch tomorrow?
Regards.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On Sat, Mar 11, 2017 at 11:57:56AM +0100, Mason wrote:
> On 10/03/2017 18:49, Mason wrote:
> > static void tango_pcie_bar_quirk(struct pci_dev *dev)
> > {
> > struct pci_bus *bus = dev->bus;
> >
> > printk("%s: bus=%d devfn=%d\n", __func__, bus->number, dev->devfn);
> >
> > pci_write_config_dword(dev, PCI_BASE_ADDRESS_0, 0x8004);
> > }
> > DECLARE_PCI_FIXUP_FINAL(0x1105, PCI_ANY_ID, tango_pcie_bar_quirk);
>
> And this is where the elusive "black magic" happens.
>
> Is it "safe" to configure a BAR behind Linux's back?
No. Linux maintains a struct resource for every BAR. This quirk
makes the BAR out of sync with the resource, so Linux no longer has an
accurate idea of what bus address space is consumed and what is
available.
Normally a BAR is for mapping device registers into PCI bus address
space. If this BAR controls how the RC forwards PCI DMA transactions
to RAM, then it's not really a BAR and you should prevent Linux from
seeing it as a BAR. You could do this by special-casing it in the
config accessor so reads return 0 and writes are dropped. Then you
could write the register in your host bridge driver safely because the
PCI core would think the BAR is not implemented.
Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 10/03/2017 18:49, Mason wrote:
> And my current code, to work-around the silicon bugs:
>
> #include
> #include
> #include
> #include
> #include
> #include
> #include
>
> //#define DEBUG_CONFIG
>
> static int tango_config_read(struct pci_bus *bus, unsigned int devfn,
> int where, int size, u32 *val)
> {
> int ret;
> void __iomem *pci_conf = (void *)0xf002e048;
>
> #ifdef DEBUG_CONFIG
> if (where == PCI_BASE_ADDRESS_0)
> dump_stack();
> #endif
>
> writel(1, pci_conf);
This sets the config/mem mux to CONFIG SPACE.
> if (devfn != 0) {
> *val = ~0;
> return PCIBIOS_DEVICE_NOT_FOUND;
> }
This works around a silicon bug, where accesses to devices or
functions not 0 return garbage.
> ret = pci_generic_config_read(bus, devfn, where, size, val);
>
> writel(0, pci_conf);
This resets the config/mem mux back to MEM SPACE.
If anything tries to access MEM in that time frame, we're toast.
> #ifdef DEBUG_CONFIG
> printk("%s: bus=%d where=%d size=%d val=0x%x\n",
> __func__, bus->number, where, size, *val);
> #endif
>
> return ret;
> }
>
> static int tango_config_write(struct pci_bus *bus, unsigned int devfn,
>int where, int size, u32 val)
> {
> int ret;
> void __iomem *pci_conf = (void *)0xf002e048;
>
> #ifdef DEBUG_CONFIG
> if (where == PCI_BASE_ADDRESS_0)
> dump_stack();
> #endif
>
> #ifdef DEBUG_CONFIG
> printk("%s: bus=%d where=%d size=%d val=0x%x\n",
> __func__, bus->number, where, size, val);
> #endif
>
> writel(1, pci_conf);
>
> ret = pci_generic_config_write(bus, devfn, where, size, val);
>
> writel(0, pci_conf);
>
> return ret;
> }
>
> static struct pci_ecam_ops tango_pci_ops = {
> .bus_shift = 20,
> .pci_ops= {
> .map_bus= pci_ecam_map_bus,
> .read = tango_config_read,
> .write = tango_config_write,
> }
> };
>
> static const struct of_device_id tango_pci_ids[] = {
> { .compatible = "sigma,smp8759-pcie" },
> { /* sentinel */ },
> };
>
> static int tango_pci_probe(struct platform_device *pdev)
> {
> return pci_host_common_probe(pdev, _pci_ops);
> }
>
> static struct platform_driver tango_pci_driver = {
> .probe = tango_pci_probe,
> .driver = {
> .name = KBUILD_MODNAME,
> .of_match_table = tango_pci_ids,
> },
> };
>
> builtin_platform_driver(tango_pci_driver);
>
> #define RIESLING_B 0x24
>
> /* Root complex reports incorrect device class */
> static void tango_pcie_fixup_class(struct pci_dev *dev)
> {
> dev->class = PCI_CLASS_BRIDGE_PCI << 8;
> }
> DECLARE_PCI_FIXUP_EARLY(0x1105, RIESLING_B, tango_pcie_fixup_class);
This works around another silicon bug.
> static void tango_pcie_bar_quirk(struct pci_dev *dev)
> {
> struct pci_bus *bus = dev->bus;
>
> printk("%s: bus=%d devfn=%d\n", __func__, bus->number, dev->devfn);
>
> pci_write_config_dword(dev, PCI_BASE_ADDRESS_0, 0x8004);
> }
> DECLARE_PCI_FIXUP_FINAL(0x1105, PCI_ANY_ID, tango_pcie_bar_quirk);
And this is where the elusive "black magic" happens.
Is it "safe" to configure a BAR behind Linux's back?
Basically, there seems to be an identity map between RAM and PCI space.
(Is that, perhaps, some kind of default? I would think that the default
would have been defined by the "ranges" prop in the pci DT node.)
So PCI address 0x8000_ maps to CPU address 0x8000_, i.e. the
start of system RAM. And when dev 1 accesses RAM, the RC correctly
forwards the packet to the memory bus.
However, RC BAR0 is limited to 1 GB (split across 8 x 128 MB "region").
Thus, to properly set this up, I need to account for what memory
Linux is managing, i.e. the mem= command line argument.
(I don't know how to access that at run-time.)
For example, if we have 2 x 512 MB of RAM.
DRAM0 is at [0x8000_, 0xa000_[
DRAM1 is at [0xc000_, 0xe000_[
But a different situation is 1 x 1 GB of RAM.
DRAM0 is at [0x8000_, 0xc000_[
I need to program different region targets.
How to do that in a way that is acceptable upstream?
Regards.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On Fri, Mar 10, 2017 at 04:05:50PM +0100, Mason wrote:
> On 10/03/2017 15:06, David Laight wrote:
>
> > Robin Murphy wrote:
> >
> >> On 09/03/17 23:43, Mason wrote:
> >>
> >>> I think I'm making progress, in that I now have a better
> >>> idea of what I don't understand. So I'm able to ask
> >>> (hopefully) less vague questions.
> >>>
> >>> Take the USB3 PCIe adapter I've been testing with. At some
> >>> point during init, the XHCI driver request some memory
> >>> (via kmalloc?) in order to exchange data with the host, right?
> >>>
> >>> On my SoC, the RAM used by Linux lives at physical range
> >>> [0x8000_, 0x8800_[ => 128 MB
> >>>
> >>> How does the XHCI driver make the adapter aware of where
> >>> it can scribble data? The XHCI driver has no notion that
> >>> the device is behind a bus, does it?
> >>>
> >>> At some point, the physical addresses must be converted
> >>> to PCI bus addresses, right? Is it computed subtracting
> >>> the offset defined in the DT?
> >
> > The driver should call dma_alloc_coherent() which returns both the
> > kernel virtual address and the device (xhci controller) has
> > to use to access it.
> > The cpu physical address is irrelevant (although it might be
> > calculated in the middle somewhere).
>
> Thank you for that missing piece of the puzzle.
> I see some relevant action in drivers/usb/host/xhci-mem.c
>
> And I now see this log:
>
> [2.499320] xhci_hcd :01:00.0: // Device context base array address =
> 0x8e07e000 (DMA), d0855000 (virt)
> [2.509156] xhci_hcd :01:00.0: Allocated command ring at cfb04200
> [2.515640] xhci_hcd :01:00.0: First segment DMA is 0x8e07f000
> [2.521863] xhci_hcd :01:00.0: // Setting command ring address to 0x20
> [2.528786] xhci_hcd :01:00.0: // xHC command ring deq ptr low bits +
> flags = @
> [2.537188] xhci_hcd :01:00.0: // xHC command ring deq ptr high bits =
> @
> [2.545002] xhci_hcd :01:00.0: // Doorbell array is located at offset
> 0x800 from cap regs base addr
> [2.554455] xhci_hcd :01:00.0: // xHCI capability registers at
> d0852000:
> [2.561550] xhci_hcd :01:00.0: // @d0852000 = 0x120 (CAPLENGTH AND
> HCIVERSION)
>
> I believe 0x8e07e000 is a CPU address, not a PCI bus address.
Sounds like you've made good progress since this email, but I think
0x8e07e000 is a PCI bus address, not a CPU address. The code in
xhci_mem_init() is this:
xhci->dcbaa = dma_alloc_coherent(dev, sizeof(*xhci->dcbaa), , flags);
xhci->dcbaa->dma = dma;
xhci_dbg_trace("Device context base array address = 0x%llx (DMA), %p (virt)",
(unsigned long long)xhci->dcbaa->dma, xhci->dcbaa);
dma_alloc_coherent() allocates a buffer and returns two values: the
CPU virtual address ("xhci->dcbaa" here) for use by the driver and the
corresponding DMA address the device can use to reach the buffer
("dma").
Documentation/DMA-API-HOWTO.txt has more details that might be useful.
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 10/03/2017 17:45, Mason wrote: > Time to clean up a million hacks to be able to discuss the finer points. Here is my current boot log: [1.133895] OF: PCI: host bridge /soc/pcie@5000 ranges: [1.139607] pci_add_resource_offset: res=[bus 00-0f] offset=0x0 [1.145659] OF: PCI: Parsing ranges property... [1.150316] OF: PCI: MEM 0x5400..0x5fff -> 0x0400 [1.156364] pci_add_resource_offset: res=[mem 0x5400-0x5fff] offset=0x5000 [1.164628] pci_tango 5000.pcie: ECAM at [mem 0x5000-0x50ff] for [bus 00-0f] [1.173033] pci_tango 5000.pcie: PCI host bridge to bus :00 [1.179440] pci_bus :00: root bus resource [bus 00-0f] [1.185056] pci_bus :00: root bus resource [mem 0x5400-0x5fff] (bus address [0x0400-0x0fff]) [1.195386] pci_bus :00: scanning bus [1.199539] pci :00:00.0: [1105:0024] type 01 class 0x048000 [1.205691] pci :00:00.0: calling tango_pcie_fixup_class+0x0/0x10 [1.212277] pci :00:00.0: reg 0x10: [mem 0x-0x3fff 64bit] [1.219220] pci :00:00.0: calling pci_fixup_ide_bases+0x0/0x40 [1.225570] pci :00:00.0: supports D1 D2 [1.229957] pci :00:00.0: PME# supported from D0 D1 D2 D3hot [1.236092] pci :00:00.0: PME# disabled [1.240576] pci_bus :00: fixups for bus [1.244886] PCI: bus0: Fast back to back transfers disabled [1.250587] pci :00:00.0: scanning [bus 00-00] behind bridge, pass 0 [1.257420] pci :00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring [1.265567] pci :00:00.0: scanning [bus 00-00] behind bridge, pass 1 [1.272517] pci_bus :01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f]) [1.283462] pci_bus :01: scanning bus [1.287623] pci :01:00.0: [1912:0014] type 00 class 0x0c0330 [1.293799] pci :01:00.0: reg 0x10: [mem 0x-0x1fff 64bit] [1.300799] pci :01:00.0: calling pci_fixup_ide_bases+0x0/0x40 [1.307223] pci :01:00.0: PME# supported from D0 D3hot D3cold [1.313446] pci :01:00.0: PME# disabled [1.318053] pci_bus :01: fixups for bus [1.322362] PCI: bus1: Fast back to back transfers disabled [1.328060] pci_bus :01: bus scan returning with max=01 [1.333759] pci_bus :01: busn_res: [bus 01-ff] end is updated to 01 [1.340506] pci_bus :00: bus scan returning with max=01 [1.346205] pci :00:00.0: fixup irq: got 0 [1.350765] pci :00:00.0: assigning IRQ 00 [1.355332] pci :01:00.0: fixup irq: got 0 [1.359892] pci :01:00.0: assigning IRQ 00 [1.364479] pci :00:00.0: BAR 0: no space for [mem size 0x4000 64bit] [1.371748] pci :00:00.0: BAR 0: failed to assign [mem size 0x4000 64bit] [1.379369] pci :00:00.0: BAR 8: assigned [mem 0x5400-0x540f] [1.386291] pci :01:00.0: BAR 0: assigned [mem 0x5400-0x54001fff 64bit] [1.393747] pci :00:00.0: PCI bridge to [bus 01] [1.398833] pci :00:00.0: bridge window [mem 0x5400-0x540f] [1.405767] pci :00:00.0: calling tango_pcie_bar_quirk+0x0/0x40 [1.412160] tango_pcie_bar_quirk: bus=0 devfn=0 [1.416843] pcieport :00:00.0: enabling device (0140 -> 0142) [1.423074] pcieport :00:00.0: enabling bus mastering [1.428652] altera_irq_domain_alloc: ENTER [1.432876] FOO-msi 2e080.msi: msi#0 address_hi 0x0 address_lo 0x9002e07c [1.440007] FOO-msi 2e080.msi: msi#0 address_hi 0x0 address_lo 0x9002e07c [1.446972] aer :00:00.0:pcie002: service driver aer loaded [1.453157] pci :01:00.0: calling quirk_usb_early_handoff+0x0/0x7e0 [1.459913] pci :01:00.0: enabling device (0140 -> 0142) [1.465709] quirk_usb_handoff_xhci: ioremap(0x5400, 8192) [1.471589] xhci_find_next_ext_cap: offset=0x500 [1.476325] val = 0x1000401 ... [1.624093] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [1.630675] ehci-pci: EHCI PCI platform driver [1.635338] xhci_hcd :01:00.0: enabling bus mastering [1.640789] xhci_hcd :01:00.0: xHCI Host Controller [1.646071] xhci_hcd :01:00.0: new USB bus registered, assigned bus number 1 [1.659065] xhci_find_next_ext_cap: offset=0x500 [1.663714] val = 0x1000401 [1.666526] xhci_find_next_ext_cap: offset=0x510 [1.671171] val = 0x3000502 [1.673984] xhci_find_next_ext_cap: offset=0x510 [1.678632] val = 0x3000502 [1.681433] xhci_find_next_ext_cap: offset=0x524 [1.686079] val = 0x2000702 [1.68] xhci_find_next_ext_cap: offset=0x524 [1.693533] val = 0x2000702 [1.696343] xhci_find_next_ext_cap: offset=0x540 [1.700987] val = 0x4c0 [1.703446] xhci_find_next_ext_cap: offset=0x550 [1.708091] val = 0xa [1.710382] xhci_find_next_ext_cap: offset=0x510 [1.715028] val = 0x3000502 [1.717837] xhci_find_next_ext_cap: offset=0x524 [
Re: Neophyte questions about PCIe
On 10/03/2017 00:43, Mason wrote: > I think I'm making progress [...] Yes! I was able to plug a USB3 Flash drive, mount it, and read its contents. A million thanks, my head was starting to hurt from too much banging. Time to clean up a million hacks to be able to discuss the finer points. Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 10/03/17 15:35, David Laight wrote:
> From: Robin Murphy
>> Sent: 10 March 2017 15:23
> ...
So you have 128MB (max) of system memory that has cpu physical
addresses 0x8000 upwards.
I'd expect it all to be accessible from any PCIe card at some PCIe
address, it might be at address 0, 0x8000 or any other offset.
I don't know which DT entry controls that offset.
>>>
>>> This is a crucial point, I think.
>>
>> The appropriate DT property would be "dma-ranges", i.e.
>>
>> pci@... {
>> ...
>> dma-ranges = <(PCI bus address) (CPU phys address) (size)>;
>> }
>
> Isn't that just saying which physical addresses the cpu can assign
> for buffers for those devices?
> There is also an offset between the 'cpu physical address' and the
> 'dma address'.
That offset is inherent in what "dma-ranges" describes. Say (for ease of
calculation) that BAR0 has been put at a mem space address of 0x2000
and maps the first 1GB of physical DRAM. That would give us:
dma-ranges = <0x2000 0x8000 0x4000>;
Then a "virt = dma_alloc_coherent(..., , ...)", borrowing the
numbers from earlier in the thread, would automatically end up with:
virt == 0xd0855000;
handle == 0x2e07e000;
(with the physical address of 0x8e07e000 in between being irrelevant to
the consuming driver)
It is true that the device's DMA mask assignment is also part and parcel
of this, whereby we will limit what physical addresses the kernel
considers valid for DMA involving devices behind this range to the lower
3GB (i.e. 0x8000 + 0x4000 - 1). With a bit of luck,
CONFIG_DMABOUNCE should do the rest of the job of working around that
where necessary.
Robin.
> This might be implicit in the 'BAR0' base address register.
>
>> The fun part is that that will only actually match the hardware once the
>> magic BAR has actually been programmed with (bus address), so you end up
>> with this part of your DT being more of a prophecy than a property :)
>
> The BAR0 values could easily be programmed into the cpld/fpga - so
> not need writing by the cpu at all.
>
> David
>
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Neophyte questions about PCIe
From: Robin Murphy
> Sent: 10 March 2017 15:23
...
> >> So you have 128MB (max) of system memory that has cpu physical
> >> addresses 0x8000 upwards.
> >> I'd expect it all to be accessible from any PCIe card at some PCIe
> >> address, it might be at address 0, 0x8000 or any other offset.
> >>
> >> I don't know which DT entry controls that offset.
> >
> > This is a crucial point, I think.
>
> The appropriate DT property would be "dma-ranges", i.e.
>
> pci@... {
> ...
> dma-ranges = <(PCI bus address) (CPU phys address) (size)>;
> }
Isn't that just saying which physical addresses the cpu can assign
for buffers for those devices?
There is also an offset between the 'cpu physical address' and the
'dma address'.
This might be implicit in the 'BAR0' base address register.
> The fun part is that that will only actually match the hardware once the
> magic BAR has actually been programmed with (bus address), so you end up
> with this part of your DT being more of a prophecy than a property :)
The BAR0 values could easily be programmed into the cpld/fpga - so
not need writing by the cpu at all.
David
--
To unsubscribe from this list: send the line "unsubscribe linux-usb" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 10/03/2017 16:14, David Laight wrote: > Mason wrote: > >> My RC drops packets not targeting its BAR0. > > I suspect the fpga/cpld logic supports RC and endpoint modes > and is using much the same names for the registers (and logic > implementation). Your guess is spot on. In the controller's MMIO registers, the so-called core_conf_0 register has the following field: chip_is_root: 1 means tango is root port, 0 means tango is endpoint. > If your cpu support more than 1GB of memory but only part is > PCIe accessible you'll have to ensure that all the memory > definitions are set correctly and 'bounce buffers' used for > some operations. Do you mean I would have to "fix" something in the XHCI driver? Hopefully, no customer plans to give Linux more than 1 GB. (Although the latest systems do support 4 GB... A lot of it is used for video buffers, handled outside Linux.) Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 10/03/17 15:05, Mason wrote: > On 10/03/2017 15:06, David Laight wrote: > >> Robin Murphy wrote: >> >>> On 09/03/17 23:43, Mason wrote: >>> I think I'm making progress, in that I now have a better idea of what I don't understand. So I'm able to ask (hopefully) less vague questions. Take the USB3 PCIe adapter I've been testing with. At some point during init, the XHCI driver request some memory (via kmalloc?) in order to exchange data with the host, right? On my SoC, the RAM used by Linux lives at physical range [0x8000_, 0x8800_[ => 128 MB How does the XHCI driver make the adapter aware of where it can scribble data? The XHCI driver has no notion that the device is behind a bus, does it? At some point, the physical addresses must be converted to PCI bus addresses, right? Is it computed subtracting the offset defined in the DT? >> >> The driver should call dma_alloc_coherent() which returns both the >> kernel virtual address and the device (xhci controller) has >> to use to access it. >> The cpu physical address is irrelevant (although it might be >> calculated in the middle somewhere). > > Thank you for that missing piece of the puzzle. > I see some relevant action in drivers/usb/host/xhci-mem.c > > And I now see this log: > > [2.499320] xhci_hcd :01:00.0: // Device context base array address = > 0x8e07e000 (DMA), d0855000 (virt) > [2.509156] xhci_hcd :01:00.0: Allocated command ring at cfb04200 > [2.515640] xhci_hcd :01:00.0: First segment DMA is 0x8e07f000 > [2.521863] xhci_hcd :01:00.0: // Setting command ring address to 0x20 > [2.528786] xhci_hcd :01:00.0: // xHC command ring deq ptr low bits + > flags = @ > [2.537188] xhci_hcd :01:00.0: // xHC command ring deq ptr high bits = > @ > [2.545002] xhci_hcd :01:00.0: // Doorbell array is located at offset > 0x800 from cap regs base addr > [2.554455] xhci_hcd :01:00.0: // xHCI capability registers at > d0852000: > [2.561550] xhci_hcd :01:00.0: // @d0852000 = 0x120 (CAPLENGTH AND > HCIVERSION) > > I believe 0x8e07e000 is a CPU address, not a PCI bus address. > > Then suppose the USB3 card wants to write to an address in RAM. It sends a packet on the PCIe bus, targeting the PCI bus address of that RAM, right? Is this address supposed to be in BAR0 of the root complex? I guess not, since Bjorn said that it was unusual for a RC to have a BAR at all. So I'll hand-wave, and decree that, by some protocol magic, the packet arrives at the PCIe controller. And this controller knows to forward this write request over the memory bus. Does that look about right? >>> >>> Generally, yes - if an area of memory space *is* claimed by a BAR, then >>> another PCI device accessing that would be treated as peer-to-peer DMA, >>> which may or may not be allowed (or supported at all). >> >> So PCIe addresses that refer to the host memory addresses are >> just forwarded to the memory subsystem. >> In practise this is almost everything. > > My RC drops packets not targeting its BAR0. OK, so it does sound like you're in a particularly awkward position that rules out using a sane 1:1 mapping between mem space and the system address map. >> The only other PCIe writes the host will see are likely to be associated >> with MIS and MSI-X interrupt support. > > Rev 1 of the PCIe controller is supposed to forward MSI doorbell > writes over the global bus to the PCIe controller's MMIO register. > >> Some PCIe root complex support peer-to-peer writes but not reads. >> Write are normally 'posted' (so are 'fire and forget') reads need the >> completion TLP (containing the data) sent back - all hard and difficult. >> >>> For mem space >>> which isn't claimed by BARs, it's up to the RC to decide what to do. As >>> a concrete example (which might possibly be relevant) the PLDA XR3-AXI >>> IP which we have in the ARM Juno SoC has the ATR_PCIE_WINx registers in >>> its root port configuration block that control what ranges of mem space >>> are mapped to the external AXI master interface and how. >>> My problem is that, in the current implementation of the PCIe controller, the USB device that wants to write to memory is supposed to target BAR0 of the RC. >>> >>> That doesn't sound right at all. If the RC has a BAR, I'd expect it to >>> be for poking the guts of the RC device itself (since this prompted me >>> to go and compare, I see the Juno RC does indeed have it own enigmatic >>> 16KB BAR, which reads as ever-changing random junk; no idea what that's >>> about). >>> Since my mem space is limited to 256 MB, then BAR0 is limited to 256 MB (or even 128 MB, since I also need to mapthe device's BAR into the same mem space). >>> >>> Your window into mem space *from the CPU's point of view* is limited to >>> 256MB. The
RE: Neophyte questions about PCIe
From: Mason > Sent: 10 March 2017 15:06 ... > My RC drops packets not targeting its BAR0. I suspect the fpga/cpld logic supports RC and endpoint modes and is using much the same names for the registers (and logic implementation). If your cpu support more than 1GB of memory but only part is PCIe accessible you'll have to ensure that all the memory definitions are set correctly and 'bounce buffers' used for some operations. David -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 10/03/2017 15:06, David Laight wrote: > Robin Murphy wrote: > >> On 09/03/17 23:43, Mason wrote: >> >>> I think I'm making progress, in that I now have a better >>> idea of what I don't understand. So I'm able to ask >>> (hopefully) less vague questions. >>> >>> Take the USB3 PCIe adapter I've been testing with. At some >>> point during init, the XHCI driver request some memory >>> (via kmalloc?) in order to exchange data with the host, right? >>> >>> On my SoC, the RAM used by Linux lives at physical range >>> [0x8000_, 0x8800_[ => 128 MB >>> >>> How does the XHCI driver make the adapter aware of where >>> it can scribble data? The XHCI driver has no notion that >>> the device is behind a bus, does it? >>> >>> At some point, the physical addresses must be converted >>> to PCI bus addresses, right? Is it computed subtracting >>> the offset defined in the DT? > > The driver should call dma_alloc_coherent() which returns both the > kernel virtual address and the device (xhci controller) has > to use to access it. > The cpu physical address is irrelevant (although it might be > calculated in the middle somewhere). Thank you for that missing piece of the puzzle. I see some relevant action in drivers/usb/host/xhci-mem.c And I now see this log: [2.499320] xhci_hcd :01:00.0: // Device context base array address = 0x8e07e000 (DMA), d0855000 (virt) [2.509156] xhci_hcd :01:00.0: Allocated command ring at cfb04200 [2.515640] xhci_hcd :01:00.0: First segment DMA is 0x8e07f000 [2.521863] xhci_hcd :01:00.0: // Setting command ring address to 0x20 [2.528786] xhci_hcd :01:00.0: // xHC command ring deq ptr low bits + flags = @ [2.537188] xhci_hcd :01:00.0: // xHC command ring deq ptr high bits = @ [2.545002] xhci_hcd :01:00.0: // Doorbell array is located at offset 0x800 from cap regs base addr [2.554455] xhci_hcd :01:00.0: // xHCI capability registers at d0852000: [2.561550] xhci_hcd :01:00.0: // @d0852000 = 0x120 (CAPLENGTH AND HCIVERSION) I believe 0x8e07e000 is a CPU address, not a PCI bus address. >>> Then suppose the USB3 card wants to write to an address >>> in RAM. It sends a packet on the PCIe bus, targeting >>> the PCI bus address of that RAM, right? Is this address >>> supposed to be in BAR0 of the root complex? I guess not, >>> since Bjorn said that it was unusual for a RC to have >>> a BAR at all. So I'll hand-wave, and decree that, by some >>> protocol magic, the packet arrives at the PCIe controller. >>> And this controller knows to forward this write request >>> over the memory bus. Does that look about right? >> >> Generally, yes - if an area of memory space *is* claimed by a BAR, then >> another PCI device accessing that would be treated as peer-to-peer DMA, >> which may or may not be allowed (or supported at all). > > So PCIe addresses that refer to the host memory addresses are > just forwarded to the memory subsystem. > In practise this is almost everything. My RC drops packets not targeting its BAR0. > The only other PCIe writes the host will see are likely to be associated > with MIS and MSI-X interrupt support. Rev 1 of the PCIe controller is supposed to forward MSI doorbell writes over the global bus to the PCIe controller's MMIO register. > Some PCIe root complex support peer-to-peer writes but not reads. > Write are normally 'posted' (so are 'fire and forget') reads need the > completion TLP (containing the data) sent back - all hard and difficult. > >> For mem space >> which isn't claimed by BARs, it's up to the RC to decide what to do. As >> a concrete example (which might possibly be relevant) the PLDA XR3-AXI >> IP which we have in the ARM Juno SoC has the ATR_PCIE_WINx registers in >> its root port configuration block that control what ranges of mem space >> are mapped to the external AXI master interface and how. >> >>> My problem is that, in the current implementation of the >>> PCIe controller, the USB device that wants to write to >>> memory is supposed to target BAR0 of the RC. >> >> That doesn't sound right at all. If the RC has a BAR, I'd expect it to >> be for poking the guts of the RC device itself (since this prompted me >> to go and compare, I see the Juno RC does indeed have it own enigmatic >> 16KB BAR, which reads as ever-changing random junk; no idea what that's >> about). >> >>> Since my mem space is limited to 256 MB, then BAR0 is >>> limited to 256 MB (or even 128 MB, since I also need >>> to mapthe device's BAR into the same mem space). >> >> Your window into mem space *from the CPU's point of view* is limited to >> 256MB. The relationship between mem space and the system (AXI) memory >> map from the point of view of PCI devices is a separate issue; if it's >> configurable at all, it probably makes sense to have the firmware set an >> outbound window to at least cover DRAM 1:1, then forget about it (this >> is essentially what Juno UEFI does, for
Re: Neophyte questions about PCIe
On 10/03/2017 14:15, Robin Murphy wrote: > On 09/03/17 23:43, Mason wrote: >> On 08/03/2017 16:17, Bjorn Helgaas wrote: >> [snip excellent in-depth overview] >> >> I think I'm making progress, in that I now have a better >> idea of what I don't understand. So I'm able to ask >> (hopefully) less vague questions. >> >> Take the USB3 PCIe adapter I've been testing with. At some >> point during init, the XHCI driver request some memory >> (via kmalloc?) in order to exchange data with the host, right? >> >> On my SoC, the RAM used by Linux lives at physical range >> [0x8000_, 0x8800_[ => 128 MB >> >> How does the XHCI driver make the adapter aware of where >> it can scribble data? The XHCI driver has no notion that >> the device is behind a bus, does it? >> >> At some point, the physical addresses must be converted >> to PCI bus addresses, right? Is it computed subtracting >> the offset defined in the DT? >> >> Then suppose the USB3 card wants to write to an address >> in RAM. It sends a packet on the PCIe bus, targeting >> the PCI bus address of that RAM, right? Is this address >> supposed to be in BAR0 of the root complex? I guess not, >> since Bjorn said that it was unusual for a RC to have >> a BAR at all. So I'll hand-wave, and decree that, by some >> protocol magic, the packet arrives at the PCIe controller. >> And this controller knows to forward this write request >> over the memory bus. Does that look about right? > > Generally, yes - if an area of memory space *is* claimed by a BAR, then > another PCI device accessing that would be treated as peer-to-peer DMA, > which may or may not be allowed (or supported at all). For mem space > which isn't claimed by BARs, it's up to the RC to decide what to do. As > a concrete example (which might possibly be relevant) the PLDA XR3-AXI > IP which we have in the ARM Juno SoC has the ATR_PCIE_WINx registers in > its root port configuration block that control what ranges of mem space > are mapped to the external AXI master interface and how. The HW dev told me that the Verilog code for the RC considers packets not targeted at RC BAR0 an error, and drops it. >> My problem is that, in the current implementation of the >> PCIe controller, the USB device that wants to write to >> memory is supposed to target BAR0 of the RC. > > That doesn't sound right at all. If the RC has a BAR, I'd expect it to > be for poking the guts of the RC device itself (since this prompted me > to go and compare, I see the Juno RC does indeed have it own enigmatic > 16KB BAR, which reads as ever-changing random junk; no idea what that's > about). That's not how our RC works. If I want to poke its guts, I have some MMIO addresses on the global bus. RC BAR0 is strictly used as a window to the global bus. >> Since my mem space is limited to 256 MB, then BAR0 is >> limited to 256 MB (or even 128 MB, since I also need >> to mapthe device's BAR into the same mem space). > > Your window into mem space *from the CPU's point of view* is limited to > 256MB. The relationship between mem space and the system (AXI) memory > map from the point of view of PCI devices is a separate issue; if it's > configurable at all, it probably makes sense to have the firmware set an > outbound window to at least cover DRAM 1:1, then forget about it (this > is essentially what Juno UEFI does, for example). The size of RC BAR0 is limited to 1 GB, so best case I can map 1 GB back to the system RAM. Well, actually best case is 896 MB since 1/8 of the window must map the MSI doorbell region. I'll see what I can come up with. Thanks a lot for your comments. Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Neophyte questions about PCIe
From: Robin Murphy > Sent: 10 March 2017 13:16 > On 09/03/17 23:43, Mason wrote: > > On 08/03/2017 16:17, Bjorn Helgaas wrote: > > [snip excellent in-depth overview] > > > > I think I'm making progress, in that I now have a better > > idea of what I don't understand. So I'm able to ask > > (hopefully) less vague questions. > > > > Take the USB3 PCIe adapter I've been testing with. At some > > point during init, the XHCI driver request some memory > > (via kmalloc?) in order to exchange data with the host, right? > > > > On my SoC, the RAM used by Linux lives at physical range > > [0x8000_, 0x8800_[ => 128 MB > > > > How does the XHCI driver make the adapter aware of where > > it can scribble data? The XHCI driver has no notion that > > the device is behind a bus, does it? > > > > At some point, the physical addresses must be converted > > to PCI bus addresses, right? Is it computed subtracting > > the offset defined in the DT? The driver should call dma_alloc_coherent() which returns both the kernel virtual address and the device (xhci controller) has to use to access it. The cpu physical address is irrelevant (although it might be calculated in the middle somewhere). > > Then suppose the USB3 card wants to write to an address > > in RAM. It sends a packet on the PCIe bus, targeting > > the PCI bus address of that RAM, right? Is this address > > supposed to be in BAR0 of the root complex? I guess not, > > since Bjorn said that it was unusual for a RC to have > > a BAR at all. So I'll hand-wave, and decree that, by some > > protocol magic, the packet arrives at the PCIe controller. > > And this controller knows to forward this write request > > over the memory bus. Does that look about right? > > Generally, yes - if an area of memory space *is* claimed by a BAR, then > another PCI device accessing that would be treated as peer-to-peer DMA, > which may or may not be allowed (or supported at all). So PCIe addresses that refer to the host memory addresses are just forwarded to the memory subsystem. In practise this is almost everything. The only other PCIe writes the host will see are likely to be associated with MIS and MSI-X interrupt support. Some PCIe root complex support peer-to-peer writes but not reads. Write are normally 'posted' (so are 'fire and forget') reads need the completion TLP (containing the data) sent back - all hard and difficult. > For mem space > which isn't claimed by BARs, it's up to the RC to decide what to do. As > a concrete example (which might possibly be relevant) the PLDA XR3-AXI > IP which we have in the ARM Juno SoC has the ATR_PCIE_WINx registers in > its root port configuration block that control what ranges of mem space > are mapped to the external AXI master interface and how. > > > My problem is that, in the current implementation of the > > PCIe controller, the USB device that wants to write to > > memory is supposed to target BAR0 of the RC. > > That doesn't sound right at all. If the RC has a BAR, I'd expect it to > be for poking the guts of the RC device itself (since this prompted me > to go and compare, I see the Juno RC does indeed have it own enigmatic > 16KB BAR, which reads as ever-changing random junk; no idea what that's > about). > > > Since my mem space is limited to 256 MB, then BAR0 is > > limited to 256 MB (or even 128 MB, since I also need > > to mapthe device's BAR into the same mem space). > > Your window into mem space *from the CPU's point of view* is limited to > 256MB. The relationship between mem space and the system (AXI) memory > map from the point of view of PCI devices is a separate issue; if it's > configurable at all, it probably makes sense to have the firmware set an > outbound window to at least cover DRAM 1:1, then forget about it (this > is essentially what Juno UEFI does, for example). So you have 128MB (max) of system memory that has cpu physical addresses 0x8000 upwards. I'd expect it all to be accessible from any PCIe card at some PCIe address, it might be at address 0, 0x8000 or any other offset. I don't know which DT entry controls that offset. David -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 09/03/17 23:43, Mason wrote: > On 08/03/2017 16:17, Bjorn Helgaas wrote: > [snip excellent in-depth overview] > > I think I'm making progress, in that I now have a better > idea of what I don't understand. So I'm able to ask > (hopefully) less vague questions. > > Take the USB3 PCIe adapter I've been testing with. At some > point during init, the XHCI driver request some memory > (via kmalloc?) in order to exchange data with the host, right? > > On my SoC, the RAM used by Linux lives at physical range > [0x8000_, 0x8800_[ => 128 MB > > How does the XHCI driver make the adapter aware of where > it can scribble data? The XHCI driver has no notion that > the device is behind a bus, does it? > > At some point, the physical addresses must be converted > to PCI bus addresses, right? Is it computed subtracting > the offset defined in the DT? > > Then suppose the USB3 card wants to write to an address > in RAM. It sends a packet on the PCIe bus, targeting > the PCI bus address of that RAM, right? Is this address > supposed to be in BAR0 of the root complex? I guess not, > since Bjorn said that it was unusual for a RC to have > a BAR at all. So I'll hand-wave, and decree that, by some > protocol magic, the packet arrives at the PCIe controller. > And this controller knows to forward this write request > over the memory bus. Does that look about right? Generally, yes - if an area of memory space *is* claimed by a BAR, then another PCI device accessing that would be treated as peer-to-peer DMA, which may or may not be allowed (or supported at all). For mem space which isn't claimed by BARs, it's up to the RC to decide what to do. As a concrete example (which might possibly be relevant) the PLDA XR3-AXI IP which we have in the ARM Juno SoC has the ATR_PCIE_WINx registers in its root port configuration block that control what ranges of mem space are mapped to the external AXI master interface and how. > My problem is that, in the current implementation of the > PCIe controller, the USB device that wants to write to > memory is supposed to target BAR0 of the RC. That doesn't sound right at all. If the RC has a BAR, I'd expect it to be for poking the guts of the RC device itself (since this prompted me to go and compare, I see the Juno RC does indeed have it own enigmatic 16KB BAR, which reads as ever-changing random junk; no idea what that's about). > Since my mem space is limited to 256 MB, then BAR0 is > limited to 256 MB (or even 128 MB, since I also need > to mapthe device's BAR into the same mem space). Your window into mem space *from the CPU's point of view* is limited to 256MB. The relationship between mem space and the system (AXI) memory map from the point of view of PCI devices is a separate issue; if it's configurable at all, it probably makes sense to have the firmware set an outbound window to at least cover DRAM 1:1, then forget about it (this is essentially what Juno UEFI does, for example). Robin. > So, if I understand correctly (which, at this point, > is quite unlikely) PCIe will work correctly for me > only if Linux manages 128 MB or less... > > How does it work on systems where the RC has no BAR? > I suppose devices are able to access all of RAM... > because the controller forwards everything? (This may > be where an IOMMU comes handy?) > > Is there a way to know, at run-time, where and how big > Linux's dynamic memory pool is? Perhaps the memory pool > itself remains smaller than 128 MB? > > I realize that I've asked a million questions. Feel free > to ignore most of them, if you can help with just one, > it would be a tremendous help already. > > Regards. > > ___ > linux-arm-kernel mailing list > linux-arm-ker...@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel > -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Neophyte questions about PCIe
On 08/03/2017 16:17, Bjorn Helgaas wrote: [snip excellent in-depth overview] I think I'm making progress, in that I now have a better idea of what I don't understand. So I'm able to ask (hopefully) less vague questions. Take the USB3 PCIe adapter I've been testing with. At some point during init, the XHCI driver request some memory (via kmalloc?) in order to exchange data with the host, right? On my SoC, the RAM used by Linux lives at physical range [0x8000_, 0x8800_[ => 128 MB How does the XHCI driver make the adapter aware of where it can scribble data? The XHCI driver has no notion that the device is behind a bus, does it? At some point, the physical addresses must be converted to PCI bus addresses, right? Is it computed subtracting the offset defined in the DT? Then suppose the USB3 card wants to write to an address in RAM. It sends a packet on the PCIe bus, targeting the PCI bus address of that RAM, right? Is this address supposed to be in BAR0 of the root complex? I guess not, since Bjorn said that it was unusual for a RC to have a BAR at all. So I'll hand-wave, and decree that, by some protocol magic, the packet arrives at the PCIe controller. And this controller knows to forward this write request over the memory bus. Does that look about right? My problem is that, in the current implementation of the PCIe controller, the USB device that wants to write to memory is supposed to target BAR0 of the RC. Since my mem space is limited to 256 MB, then BAR0 is limited to 256 MB (or even 128 MB, since I also need to mapthe device's BAR into the same mem space). So, if I understand correctly (which, at this point, is quite unlikely) PCIe will work correctly for me only if Linux manages 128 MB or less... How does it work on systems where the RC has no BAR? I suppose devices are able to access all of RAM... because the controller forwards everything? (This may be where an IOMMU comes handy?) Is there a way to know, at run-time, where and how big Linux's dynamic memory pool is? Perhaps the memory pool itself remains smaller than 128 MB? I realize that I've asked a million questions. Feel free to ignore most of them, if you can help with just one, it would be a tremendous help already. Regards. -- To unsubscribe from this list: send the line "unsubscribe linux-usb" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
