Hi Pali, On Mon, Nov 8, 2021 at 3:11 AM Pali Rohár <p...@kernel.org> wrote: > > Hello! > > On Sunday 07 November 2021 18:08:56 Tony Dinh wrote: > > Hi Pali, > > > > On Sun, Nov 7, 2021 at 4:58 PM Tony Dinh <mibo...@gmail.com> wrote: > > > > > > Hi Pali, > > > > > > Looks like it is working! But in a weird way. Please see my responses > > > in between, and also in the test log below. > > > > > > On Sun, Nov 7, 2021 at 3:45 PM Pali Rohár <p...@kernel.org> wrote: > > > > > > > > Hello! > > > > > > > > On Sunday 07 November 2021 12:56:37 Tony Dinh wrote: > > > > > Hi Pali, > > > > > > > > > > I've applied your patch, and ran some tests. Looks like we got to the > > > > > bind, but it was not probing (or the probe died silently). Later, I > > > > > tried pci enum and got some errors during probing. > > > > > > > > It looks like that arch/arm/mach-kirkwood/cpu.c already setup PCIe > > > > windows. Therefore when pci_mvebu.c tries it too, it fails on error > > > > "conflicts with another window" which you see in the log. > > > > > > > > Could you try to comment calling of "mvebu_mbus_add_window_by_id" > > > > function in pci_mvebu.c? As PCIe window setup should be done exactly > > > > once. > > > > > > > > Also try to call 'pci 0' and 'pci 1' after the 'pci enum'. > > > > > ... > > > > > > > > > > - These are relevant configs used in the build (not sure if I need to > > > > > use CONFIG_DM_PCI_COMPAT?) > > > > > > > > > > CONFIG_DM=y > > > > > CONFIG_CMD_PCI=y > > > > > CONFIG_PCI=y > > > > > CONFIG_PCI_MVEBU=y > > > > > CONFIG_PCI_ENDPOINT=y > > > > > CONFIG_DM_PCI_COMPAT=y > > > > > CONFIG_USB_XHCI_HCD=y > > > > > CONFIG_USB_XHCI_PCI=y > > > > > CONFIG_USB_XHCI_MVEBU=y > > > > > > > > Following options should be enough: > > > > > > > > CONFIG_CMD_PCI=y > > > > CONFIG_PCI=y > > > > # CONFIG_DM_PCI_COMPAT is not set > > > > CONFIG_PCI_PNP=y > > > > CONFIG_PCI_MVEBU=y > > > > > > Since the PCI bus in this box is used for USB 3.0, I've included > > > various XHCI configs to see if I can probe this USB 3.0 flash drive > > > also (that's the ultimate test). > > OK! > > > > > > > > > Option CONFIG_PCI_PNP=y is required to see endpoint cards. Without it > > > > U-Boot would see only PCIe Root Ports. But for first basic tests it > > > > should be enough. > > > > > > > > > BTW, the topic is no longer kwboot, should we move this to another new > > > > > thread? i.e. Testing PCI MVEBU with Kirkwood SoCs. > > > > > > > > Changed :-) > > > > > > Thanks :) > > > > > > Here is the log of a successful spin up for the PCI bus and USB 3.0 > > > drive. Please see my commentary ### in between the logs. > > > > > > U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 16:03:46 -0800) > > > Pogoplug V4 > > > > > > SoC: Kirkwood 88F6281_A1 > > > DRAM: 128 MiB > > > device_probe: > > > device_probe: exit 0 > > > > > > > > > ### The device_probe above seems to have nothing happening. > > > > > > > > > mvebu_pcie_bind: begin > > > mvebu_pcie_bind: got an available node > > > Bound device to pcie@82000000 > > > mvebu_pcie_bind: bound > > > mvebu_pcie_bind: exit 0 > > > Bound device pcie@82000000 to mbus@f1000000 > > > Bound device mbus@f1000000 to root_driver > > > Bound device ehci@50000 to ocp@f1000000 > > > Bound device ethernet-controller@72000 to ocp@f1000000 > > > Bound device sata@80000 to ocp@f1000000 > > > Bound device mvs...@90000.blk to mvsdio@90000 > > > Bound device mvsdio@90000 to ocp@f1000000 > > > Bound device ocp@f1000000 to root_driver > > > NAND: 128 MiB > > > MMC: device_probe: > > > device_probe: > > > device_probe: > > > device_probe: exit 0 > > > device_probe: exit 0 > > > device_probe: > > > device_probe: > > > mvsdio@90000: 0 > > > Loading Environment from NAND... *** Warning - bad CRC, using default > > > environment > > > > > > In: serial > > > Out: serial > > > Err: serial > > > Net: device_probe: > > > device_probe: > > > > > > Warning: ethernet-controller@72000 (eth0) using random MAC address - > > > 86:b8:84:ee:a9:a5 > > > device_probe: exit 0 > > > eth0: ethernet-controller@72000 > > > Hit any key to stop autoboot: 0 > > > Pogo_V4> usb reset > > > resetting USB... > > > Bus ehci@50000: device_probe: > > > device_probe: > > > USB EHCI 1.00 > > > device_probe: exit 0 > > > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000 > > > device_probe: > > > device_probe: > > > Bound device usb_hub to usb_hub > > > device_probe: > > > device_probe: > > > device_probe: exit 0 > > > Bound device usb_mass_storage to usb_hub > > > device_probe: > > > device_probe: > > > Bound device usb_mass_storage.lun0 to usb_mass_storage > > > device_probe: exit 0 > > > device_probe: exit 0 > > > 3 USB Device(s) found > > > scanning usb for storage devices... 1 Storage Device(s) found > > > > > > > > > ### Try to bring up both USB drives (2.0 and 3.0). But only the USB > > > 2.0 was up as shown above. > > > > > > > > > Pogo_V4> pci enum > > > device_probe: > > > device_probe: > > > device_probe: > > > device_probe: > > > device_probe: exit 0 > > > device_probe: exit 0 > > > mvebu_pcie_probe: > > > Cannot add window '4:e8', conflicts with another window > > > PCIe unable to add mbus window for mem at 90000000+10000000 > > > Cannot add window '4:e0', conflicts with another window > > > PCIe unable to add mbus window for IO at c0000000+00010000 > > Error is still there... > > Have you tried to comment and disable calling > "mvebu_mbus_add_window_by_id" function from pci_mvebu.c? > > > > mvebu_pcie_probe: exit 0 > > > Bound device pci_0:0.0 to pcie0.0 > > > device_probe: > > > device_probe: > > > Bound device xhci_pci to pci_0:0.0 > > > device_probe: exit 0 > > > device_probe: exit 0 > > > device_probe: > > > > > > ### So I did the pci enum above to get the XHCI bound to PCIe. > > > > > > Pogo_V4> usb reset > > > resetting USB... > > > Bus ehci@50000: device_probe: > > > device_probe: > > > USB EHCI 1.00 > > > device_probe: exit 0 > > > Bus xhci_pci: device_probe: > > > device_probe: > > > Register 400081f NbrPorts 4 > > > Starting the controller > > > USB XHCI 1.00 > > > device_probe: exit 0 > > > scanning bus ehci@50000 for devices... Bound device usb_hub to ehci@50000 > > > device_probe: > > > device_probe: > > > Bound device usb_hub to usb_hub > > > device_probe: > > > device_probe: > > > device_probe: exit 0 > > > Bound device usb_mass_storage to usb_hub > > > device_probe: > > > device_probe: > > > Bound device usb_mass_storage.lun0 to usb_mass_storage > > > device_probe: exit 0 > > > device_probe: exit 0 > > > 3 USB Device(s) found > > > scanning bus xhci_pci for devices... Bound device usb_hub to xhci_pci > > > device_probe: > > > device_probe: > > > Bound device usb_mass_storage to usb_hub > > > device_probe: > > > device_probe: > > > Bound device usb_mass_storage.lun0 to usb_mass_storage > > > device_probe: exit 0 > > > device_probe: exit 0 > > > 2 USB Device(s) found > > > scanning usb for storage devices... 2 Storage Device(s) found > > > > > > > > > ### Now both drives are up! > > > > > > > > > Pogo_V4> pci 0 > > > device_probe: > > > Scanning PCI devices on bus 0 > > > BusDevFun VendorId DeviceId Device Class Sub-Class > > > _____________________________________________________________ > > > 00.00.00 0x11ab 0x6281 Bridge device 0x04 > > > Pogo_V4> pci 1 > > > device_probe: > > > Scanning PCI devices on bus 1 > > > BusDevFun VendorId DeviceId Device Class Sub-Class > > > _____________________________________________________________ > > > 01.00.00 0x1b73 0x1009 Serial bus controller 0x03 > > > > > > > > > ### And the pci command has meaningful output at this point as shown > > > above. > > > > > > > > > Pogo_V4> usb tree > > > USB device tree: > > > 1 Hub (480 Mb/s, 0mA) > > > | u-boot EHCI Host Controller > > > | > > > +-2 Hub (480 Mb/s, 100mA) > > > | GenesysLogic USB2.0 Hub > > > | > > > +-3 Mass Storage (480 Mb/s, 224mA) > > > SanDisk Ultra Fit 4C531001560827107320 > > > > > > 1 Hub (5 Gb/s, 0mA) > > > | U-Boot XHCI Host Controller > > > | > > > +-2 Mass Storage (5 Gb/s, 224mA) > > > SanDisk Cruzer Glide 3.0 4C530000130418116112 > > > > > > Pogo_V4> usb info > > > 1: Hub, USB Revision 2.0 > > > - u-boot EHCI Host Controller > > > - Class: Hub > > > - PacketSize: 64 Configurations: 1 > > > - Vendor: 0x0000 Product 0x0000 Version 1.0 > > > Configuration: 1 > > > - Interfaces: 1 Self Powered 0mA > > > Interface: 0 > > > - Alternate Setting 0, Endpoints: 1 > > > - Class Hub > > > - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms > > > > > > 2: Hub, USB Revision 2.0 > > > - GenesysLogic USB2.0 Hub > > > - Class: Hub > > > - PacketSize: 64 Configurations: 1 > > > - Vendor: 0x05e3 Product 0x0610 Version 144.48 > > > Configuration: 1 > > > - Interfaces: 1 Self Powered Remote Wakeup 100mA > > > Interface: 0 > > > - Alternate Setting 0, Endpoints: 1 > > > - Class Hub > > > - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms > > > - Endpoint 1 In Interrupt MaxPacket 1 Interval 12ms > > > > > > 3: Mass Storage, USB Revision 2.10 > > > - SanDisk Ultra Fit 4C531001560827107320 > > > - Class: (from Interface) Mass Storage > > > - PacketSize: 64 Configurations: 1 > > > - Vendor: 0x0781 Product 0x5583 Version 1.0 > > > Configuration: 1 > > > - Interfaces: 1 Bus Powered 224mA > > > Interface: 0 > > > - Alternate Setting 0, Endpoints: 2 > > > - Class Mass Storage, Transp. SCSI, Bulk only > > > - Endpoint 1 In Bulk MaxPacket 512 > > > - Endpoint 2 Out Bulk MaxPacket 512 > > > > > > 1: Hub, USB Revision 3.0 > > > - U-Boot XHCI Host Controller > > > - Class: Hub > > > - PacketSize: 512 Configurations: 1 > > > - Vendor: 0x0000 Product 0x0000 Version 1.0 > > > Configuration: 1 > > > - Interfaces: 1 Self Powered 0mA > > > Interface: 0 > > > - Alternate Setting 0, Endpoints: 1 > > > - Class Hub > > > - Endpoint 1 In Interrupt MaxPacket 8 Interval 255ms > > > > > > 2: Mass Storage, USB Revision 3.0 > > > - SanDisk Cruzer Glide 3.0 4C530000130418116112 > > > - Class: (from Interface) Mass Storage > > > - PacketSize: 512 Configurations: 1 > > > - Vendor: 0x0781 Product 0x5597 Version 1.0 > > > Configuration: 1 > > > - Interfaces: 1 Bus Powered 224mA > > > Interface: 0 > > > - Alternate Setting 0, Endpoints: 2 > > > - Class Mass Storage, Transp. SCSI, Bulk only > > > - Endpoint 1 In Bulk MaxPacket 1024 > > > - Endpoint 2 Out Bulk MaxPacket 1024 > > > > > > ### The XHCI Host Controller and the SanDisk Cruzer Glide are now > > > active as shown above. > > > > > > ### So list the files on this SanDisk Cruzer Glide to make sure we can > > > read it. > > > > > > > > > Pogo_V4> ext2ls usb 1:1 / > > > device_probe: > > > device_probe: > > > device_probe: exit 0 > > > <DIR> 4096 . > > > <DIR> 4096 .. > > > 522666 uboot.2021.07-tld-1.pogo_v4.bin > > > 524288 uboot.2021.07-tld-1.pogo_v4.mtd0.kwb > > > > > > Great works Pali! > > > > > > I think some more investigation is needed. Why did we need to do "pci > > > enum", and then "usb reset", in that order, to get the PCI bus and the > > > XHCI controller probed? there must be something missing in the process > > > somewhere between the Device uclass, the PCI uclass, and the pci_mvebu > > > uclass? > > > > I think I can see the order of enumeration. PCI must be enumerated > > first, and then XHCI being the controller on this host bus would come > > alive? I think we can live with 'pci enum' and 'usb reset' to get the > > USB 3.0 drives enumerated. However, it seems just a little bit > > unintuitive. > > 'pci enum' should be called internally by U-Boot during loading. So only > 'usb start' would be needed. > > But from your boot log it looks like that PCI enumaration was not done > and so calling 'pci enum' manually is needed. > > I will look into U-Boot code why it happens... > > Anyway, based on your test, PCIe must work correctly :)
Indeed, it's working perfectly :) I've also commented out the 2 calls to mvebu_mbus_add_window_by_id(), and no longer see the conflicts error. - if (mvebu_mbus_add_window_by_id(pcie->mem_target, pcie->mem_attr, - (phys_addr_t)pcie->mem.start, - PCIE_MEM_SIZE)) { - if (mvebu_mbus_add_window_by_id(pcie->io_target, pcie->io_attr, - (phys_addr_t)pcie->io.start, - MBUS_PCI_IO_SIZE)) { > Could you send config space dump of PCIe Root Port? Following U-Boot > command prints it on terminal: 'pci display.b 0.0.0 0 200' Sure, here is the log with the dump. U-Boot 2022.01-rc1-00054-g52207514ba-dirty (Nov 07 2021 - 22:52:05 -0800) Pogoplug V4 SoC: Kirkwood 88F6281_A1 DRAM: 128 MiB NAND: 128 MiB MMC: mvsdio@90000: 0 Loading Environment from NAND... *** Warning - bad CRC, using default environment In: serial Out: serial Err: serial Net: Warning: ethernet-controller@72000 (eth0) using random MAC address - 56:f0:8e:db:76:2f eth0: ethernet-controller@72000 Hit any key to stop autoboot: 0 Pogo_V4> pci enum Pogo_V4> pci 0 Scanning PCI devices on bus 0 BusDevFun VendorId DeviceId Device Class Sub-Class _____________________________________________________________ 00.00.00 0x11ab 0x6281 Bridge device 0x04 Pogo_V4> pci 1 Scanning PCI devices on bus 1 BusDevFun VendorId DeviceId Device Class Sub-Class _____________________________________________________________ 01.00.00 0x1b73 0x1009 Serial bus controller 0x03 Pogo_V4> pci display.b 0.0.0 0 200 00000000: ab 11 81 62 07 00 10 00 03 00 04 06 00 00 01 00 00000010: 00 00 00 00 00 00 00 00 00 01 01 00 01 f1 00 00 00000020: 00 90 00 90 01 10 01 00 00 00 00 00 00 00 00 00 00000030: 00 c0 ff bf 40 00 00 00 00 00 00 00 00 01 00 00 00000040: 01 50 03 06 00 00 00 00 00 00 00 00 00 00 00 00 00000050: 05 60 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00000060: 10 00 41 00 80 80 00 00 00 20 00 00 11 ac 07 00 00000070: 08 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00 00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000000f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000100: 01 00 01 00 00 00 00 00 00 00 00 00 10 00 06 00 00000110: 00 00 00 00 00 20 00 00 00 00 00 00 01 00 00 4a 00000120: 04 00 00 01 00 00 08 01 0c 03 30 02 00 00 00 00 00000130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00000190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000001a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000001b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000001c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000001d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000001e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 000001f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Pogo_V4> Thanks, Tony