Re: PCIe NVME drives not detected on Dell R6515
On Mon, May 4, 2020 at 11:12 AM Miroslav Lachman <000.f...@quip.cz> wrote: > > On 2020-04-27 08:02, Miroslav Lachman wrote: > > I don't know what is with Scott. I hope he is well. > > Is there somebody else who can help me with this issue? > > Scott wrote there are hotplug PCIe buses not probed during boot process. > > I am not a developer so I cannot move forward alone. > > The problem is with PCIe Hot Plug. > Hot Plug bus was not enumerated thus no NVME detected. I may have just been bitten by this as well when running FreeBSD under qemu. The q35 machine type with PCIe emulation enables PCIe hot plug on all the root ports, but I am not seeing any downstream devices (either emulated like e1000 or passed through by the host) because of a check in pcib_hotplug_present(): /* * Require the Electromechanical Interlock to be engaged if * present. */ if (sc->pcie_slot_cap & PCIEM_SLOT_CAP_EIP && (sc->pcie_slot_sta & PCIEM_SLOT_STA_EIS) == 0) return (0); Under qemu, the slot indicates an Electromechanical Interlock is Present in the capabilities register, but it does not set the Electromechanical Interlock Status bit. This causes the PCI driver to not probe any children. Commenting out the above code made both emulated PCIe devices as well as host devices passed through appear in FreeBSD. As a data point, I'm not seeing similar checks in the Linux kernel. Miroslav, would it be possible to comment out/delete the above code in your kernel and retest to see if that helps your case as well? --chuck ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
On 2020-04-27 08:02, Miroslav Lachman wrote: I don't know what is with Scott. I hope he is well. Is there somebody else who can help me with this issue? Scott wrote there are hotplug PCIe buses not probed during boot process. I am not a developer so I cannot move forward alone. The problem is with PCIe Hot Plug. Hot Plug bus was not enumerated thus no NVME detected. Dan Lukes suggested (privately) to disable hot plugging by this at second stage loader prompt: set hw.pci.enable_pcie_hp=0 Then I was able to boot FreeBSD installer ISO in BIOS mode (I don't know why but this machine is not able to boot FreeBSD ISO media in UEFI mode). Installer sees both NVME disks and installation was successful but it cannot boot - Dell R6515 in BIOS mode does not show NVME drives. Switching to UEFI boot shows disks but they didn't contained EFI partition boot code. When I modified the partitions layout (remove swap, ad efi partition and swap again) it is now able to boot FreeBSD 11.3 amd64 in UEFI mode from NVME disks with Hot Plug disabled in loader.conf. Can somebody look on to it why the bus is not probed when Hot Plug is enabled? I have a few days to run some tests on this HW before it will go in to production. Kind regards Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
I don't know what is with Scott. I hope he is well. Is there somebody else who can help me with this issue? Scott wrote there are hotplug PCIe buses not probed during boot process. I am not a developer so I cannot move forward alone. Kind regards Miroslav Lachman On 2020-04-17 23:30, Scott Long wrote: On Apr 17, 2020, at 3:07 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Scott Long wrote on 04/17/2020 23:04: On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Scott Long wrote on 04/17/2020 22:17: On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Kurt Jaeger wrote on 04/17/2020 21:44: Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? Here it is https://ibb.co/c1dZrTf Miroslav Lachman Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed. The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures. Over 3000 lines long but I finally managed to make SOL work so I have it as text! https://pastebin.pl/view/90fdaafb This helped a lot, thanks. It looks like these PCIe buses are marked as being hotplug, and for some reason we’re not probing them. At this point, I’d need to feed you some kernel patches that will dump out more info, but you’d have to compile them and get them onto your boot media. Is that a possibility? Currently I have all machines on 11.3 (where I can rebuild kernel without problem) If CURRENT is required I would need to setup some CURRENT VM in VirtualBox. Can you send me some link to documentation who should I create new ISO after rebuild? I don’t know of any docs for doing custom releases, and it looks like it’s harder than it used to be to insert custom patches. That said, I recommend doing the following on your 11.x build system: 1. Do a clean `make buildworld` with an up-to-date tree 2. change into the `release` directory that you just did the buildworld from 3. `sudo make release NOPORTS= NODOC= CHROOTDIR=/usr/tmp/release SRCBRANCH="base/stable/11@rHEAD”` You can set CHROOTDIR to whatever you want that has a few GB of space, but remember where you’ve set it for later steps. This will build a release with stock sources. Let it complete, both to prepare for the next step and to ensure that it works. It’ll take an hour or two depending on your machine speed 4. Take the patch that I’ll send you shortly and apply it to $CHROOTDIR/usr/src 5. `sudo make memstick NOPORTS= NODOC= SRC_UPDATE_SKIP= CHROOTDIR=/usr/tmp/release` Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 18:17: You are correct about Intel vs AMD. Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD. Can you send the full output of `lspci -tvv` from linux? Sorry for my late reply. Booting the Linux SystemRescueCd is too slow. lspci -tvv output is attached I tried to connect to SOL by SSH but it shows black screen only. lspci shows drives: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] Kind regards Miroslav Lachman On Apr 17, 2020, at 9:54 AM, Scott Long wrote: Would that be the Intel VMD/VROC stuff? If so, there’s a driver for FreeBSD, but it’s not well tested yet. Will have to dig in further. Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 23:30: Can you send me some link to documentation who should I create new ISO after rebuild? I don’t know of any docs for doing custom releases, and it looks like it’s harder than it used to be to insert custom patches. That said, I recommend doing the following on your 11.x build system: 1. Do a clean `make buildworld` with an up-to-date tree 2. change into the `release` directory that you just did the buildworld from 3. `sudo make release NOPORTS= NODOC= CHROOTDIR=/usr/tmp/release SRCBRANCH="base/stable/11@rHEAD”` You can set CHROOTDIR to whatever you want that has a few GB of space, but remember where you’ve set it for later steps. This will build a release with stock sources. Let it complete, both to prepare for the next step and to ensure that it works. It’ll take an hour or two depending on your machine speed 4. Take the patch that I’ll send you shortly and apply it to $CHROOTDIR/usr/src 5. `sudo make memstick NOPORTS= NODOC= SRC_UPDATE_SKIP= CHROOTDIR=/usr/tmp/release` I did frech checkout of 11 stable in to /usr/src113/ setenv MAKEOBJDIRPREFIX /usr/obj113 cd /usr/src113 make buildworld make buildkernel KERNCONF=GENERIC cd release/ mkdir /vol0/release make release NOPORTS= NODOC= CHROOTDIR=/vol0/release SRCBRANCH="base/stable/11@rHEAD" /vol0/release is empty, it was created in /usr/obj113/usr/src113/release # ll /usr/obj113/usr/src113/release/ total 1696255 -rw-r--r-- 1 root wheel 1051 Apr 18 08:28 MANIFEST -rw-r--r-- 1 root wheel 56066772 Apr 18 07:51 base-dbg.txz -rw-r--r-- 1 root wheel 118984236 Apr 18 07:51 base.txz drwxr-xr-x 18 root wheel 22 Apr 18 08:32 bootonly -rw-r--r-- 1 root wheel 333000704 Apr 18 08:33 bootonly.iso drwxr-xr-x 18 root wheel 22 Apr 18 08:30 disc1 -rw-r--r-- 1 root wheel 692711424 Apr 18 08:31 disc1.iso drwxr-xr-x 8 root wheel 8 Apr 18 08:25 dist -rw-r--r-- 1 root wheel1428564 Apr 18 07:51 doc.txz drwxr-xr-x 2 root wheel 12 Apr 18 08:28 ftp -rw-r--r-- 1 root wheel116 Apr 18 08:25 kernel-dbg.txz -rw-r--r-- 1 root wheel 40103140 Apr 18 08:25 kernel.txz -rw-r--r-- 1 root wheel 14682736 Apr 18 07:51 lib32-dbg.txz -rw-r--r-- 1 root wheel 24179800 Apr 18 07:51 lib32.txz -rw-r--r-- 1 root wheel 762520064 Apr 18 08:33 memstick.img -rw-r--r-- 1 root wheel 373301760 Apr 18 08:34 mini-memstick.img -rw-r--r-- 1 root wheel 0 Apr 18 08:28 packagesystem -rw-r--r-- 1 root wheel 0 Apr 18 08:51 release -rw-r--r-- 1 root wheel 159327100 Apr 18 08:27 src.txz -rw-r--r-- 1 root wheel4941816 Apr 18 07:51 tests.txz What I did wrong? Anyway I am waiting for your patches. Kind regards Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
On Fri, 17 Apr 2020 23:07:38 +0200 Miroslav Lachman 000.f...@quip.cz said Scott Long wrote on 04/17/2020 23:04: > > >> On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <000.f...@quip.cz> wrote: >> >> Scott Long wrote on 04/17/2020 22:17: On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Kurt Jaeger wrote on 04/17/2020 21:44: > Hi! >>> pciconf -lBc pcib12 >>> pciconf -lBc pcib13 >> >> Printscreen attached. > Attachments are stripped from the list -- can you put them somewhere > online ? Here it is https://ibb.co/c1dZrTf Miroslav Lachman >>> Ok, the bridges know about their downstream bus numbers, but I see nothing > that suggests that they’re being probed. The next step would be bootverbose, > but that’s going to be a lot of output to collect in screen captures. >> >> Over 3000 lines long but I finally managed to make SOL work so I have it as > text! >> >> https://pastebin.pl/view/90fdaafb >> > > This helped a lot, thanks. It looks like these PCIe buses are marked as > being hotplug, and for some reason we’re not probing them. At this point, > I’d need to feed you some kernel patches that will dump out more info, but > you’d have to compile them and get them onto your boot media. Is that a > possibility? Currently I have all machines on 11.3 (where I can rebuild kernel without problem) If CURRENT is required I would need to setup some CURRENT VM in VirtualBox. Can you send me some link to documentation who should I create new ISO after rebuild? Here's what I do After building world && kernel: # cd /usr/src/ # make installworld DESTDIR=/to/path/with-2Gig-space # make distribution DESTDIR=/to/path/with-2Gig-space (you need a slice with ~2G space available) Then: # mkisofs -b boot/cdboot -no-emul-boot -r -J -V "FreeBSD__Install" -publisher "" -o /path/to/put/NEW_INSTALL.iso /to/path/with-2Gig-space Hope it makes sense to you. :) --Chris Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote: On Apr 17, 2020, at 3:07 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Scott Long wrote on 04/17/2020 23:04: On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Scott Long wrote on 04/17/2020 22:17: On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Kurt Jaeger wrote on 04/17/2020 21:44: Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? Here it is https://ibb.co/c1dZrTf Miroslav Lachman Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed. The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures. Over 3000 lines long but I finally managed to make SOL work so I have it as text! https://pastebin.pl/view/90fdaafb This helped a lot, thanks. It looks like these PCIe buses are marked as being hotplug, and for some reason we’re not probing them. At this point, I’d need to feed you some kernel patches that will dump out more info, but you’d have to compile them and get them onto your boot media. Is that a possibility? Currently I have all machines on 11.3 (where I can rebuild kernel without problem) If CURRENT is required I would need to setup some CURRENT VM in VirtualBox. Can you send me some link to documentation who should I create new ISO after rebuild? I don’t know of any docs for doing custom releases, and it looks like it’s harder than it used to be to insert custom patches. That said, I recommend doing the following on your 11.x build system: 1. Do a clean `make buildworld` with an up-to-date tree 2. change into the `release` directory that you just did the buildworld from 3. `sudo make release NOPORTS= NODOC= CHROOTDIR=/usr/tmp/release SRCBRANCH="base/stable/11@rHEAD”` You can set CHROOTDIR to whatever you want that has a few GB of space, but remember where you’ve set it for later steps. This will build a release with stock sources. Let it complete, both to prepare for the next step and to ensure that it works. It’ll take an hour or two depending on your machine speed 4. Take the patch that I’ll send you shortly and apply it to $CHROOTDIR/usr/src 5. `sudo make memstick NOPORTS= NODOC= SRC_UPDATE_SKIP= CHROOTDIR=/usr/tmp/release` Just a thought - it could possibly be easier to build just the patched kernel and try booting it (11.3 userland should work with CURRENT kernel, right?), doing the kernel-toolchaing target first, and using INSTKERNNAME with the installkernel target to temporary select it from loader menu. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
> On Apr 17, 2020, at 3:07 PM, Miroslav Lachman <000.f...@quip.cz> wrote: > > Scott Long wrote on 04/17/2020 23:04: >>> On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <000.f...@quip.cz> wrote: >>> >>> Scott Long wrote on 04/17/2020 22:17: > On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: > > Kurt Jaeger wrote on 04/17/2020 21:44: >> Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 >>> >>> Printscreen attached. >> Attachments are stripped from the list -- can you put them somewhere >> online ? > > Here it is https://ibb.co/c1dZrTf > > Miroslav Lachman > Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed. The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures. >>> >>> Over 3000 lines long but I finally managed to make SOL work so I have it as >>> text! >>> >>> https://pastebin.pl/view/90fdaafb >>> >> This helped a lot, thanks. It looks like these PCIe buses are marked as >> being hotplug, and for some reason we’re not probing them. At this point, >> I’d need to feed you some kernel patches that will dump out more info, but >> you’d have to compile them and get them onto your boot media. Is that a >> possibility? > > Currently I have all machines on 11.3 (where I can rebuild kernel without > problem) > If CURRENT is required I would need to setup some CURRENT VM in VirtualBox. > > Can you send me some link to documentation who should I create new ISO after > rebuild? > I don’t know of any docs for doing custom releases, and it looks like it’s harder than it used to be to insert custom patches. That said, I recommend doing the following on your 11.x build system: 1. Do a clean `make buildworld` with an up-to-date tree 2. change into the `release` directory that you just did the buildworld from 3. `sudo make release NOPORTS= NODOC= CHROOTDIR=/usr/tmp/release SRCBRANCH="base/stable/11@rHEAD”` You can set CHROOTDIR to whatever you want that has a few GB of space, but remember where you’ve set it for later steps. This will build a release with stock sources. Let it complete, both to prepare for the next step and to ensure that it works. It’ll take an hour or two depending on your machine speed 4. Take the patch that I’ll send you shortly and apply it to $CHROOTDIR/usr/src 5. `sudo make memstick NOPORTS= NODOC= SRC_UPDATE_SKIP= CHROOTDIR=/usr/tmp/release` Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
> On Apr 17, 2020, at 3:13 PM, Mel Pilgrim > wrote: > > On 2020-04-17 8:50, Warner Losh wrote: >> On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: >>> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or >>> 13-CURRENT? Also, can you send me the output of ‘dmesg’? >>> >> There was another thread that said there was a raid card in the way... It >> would be cool to find a way to get it out of the way... :) > On the R6515, NVME drives are supported through a PERC S150 mini card. > There's no m.2 slots or U.2 ports on the mainboard itself. The S150 supports > non-RAID mode, but that still leaves supporting a software-based PERC card as > a PCIe switch for the 8 or 10 NVME bays behind it. This doesn’t seem to be the case for Miroslav’s system. There’s no special handling of the PCIe bridge/switch in linux where it does work, and there’s growing evidence that this is a case of FreeBSD not handling hotplug ports correctly. Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
> On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <000.f...@quip.cz> wrote: > > Scott Long wrote on 04/17/2020 22:17: >>> On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: >>> >>> Kurt Jaeger wrote on 04/17/2020 21:44: Hi! >> pciconf -lBc pcib12 >> pciconf -lBc pcib13 > > Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? >>> >>> Here it is https://ibb.co/c1dZrTf >>> >>> Miroslav Lachman >>> >> Ok, the bridges know about their downstream bus numbers, but I see nothing >> that suggests that they’re being probed. The next step would be >> bootverbose, but that’s going to be a lot of output to collect in screen >> captures. > > Over 3000 lines long but I finally managed to make SOL work so I have it as > text! > > https://pastebin.pl/view/90fdaafb > This helped a lot, thanks. It looks like these PCIe buses are marked as being hotplug, and for some reason we’re not probing them. At this point, I’d need to feed you some kernel patches that will dump out more info, but you’d have to compile them and get them onto your boot media. Is that a possibility? Thanks, Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
On 2020-04-17 8:50, Warner Losh wrote: On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT? Also, can you send me the output of ‘dmesg’? There was another thread that said there was a raid card in the way... It would be cool to find a way to get it out of the way... :) On the R6515, NVME drives are supported through a PERC S150 mini card. There's no m.2 slots or U.2 ports on the mainboard itself. The S150 supports non-RAID mode, but that still leaves supporting a software-based PERC card as a PCIe switch for the 8 or 10 NVME bays behind it. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 23:04: On Apr 17, 2020, at 2:45 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Scott Long wrote on 04/17/2020 22:17: On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Kurt Jaeger wrote on 04/17/2020 21:44: Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? Here it is https://ibb.co/c1dZrTf Miroslav Lachman Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed. The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures. Over 3000 lines long but I finally managed to make SOL work so I have it as text! https://pastebin.pl/view/90fdaafb This helped a lot, thanks. It looks like these PCIe buses are marked as being hotplug, and for some reason we’re not probing them. At this point, I’d need to feed you some kernel patches that will dump out more info, but you’d have to compile them and get them onto your boot media. Is that a possibility? Currently I have all machines on 11.3 (where I can rebuild kernel without problem) If CURRENT is required I would need to setup some CURRENT VM in VirtualBox. Can you send me some link to documentation who should I create new ISO after rebuild? Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 22:17: On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: Kurt Jaeger wrote on 04/17/2020 21:44: Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? Here it is https://ibb.co/c1dZrTf Miroslav Lachman Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed. The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures. Over 3000 lines long but I finally managed to make SOL work so I have it as text! https://pastebin.pl/view/90fdaafb Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
> On Apr 17, 2020, at 1:47 PM, Miroslav Lachman <000.f...@quip.cz> wrote: > > Kurt Jaeger wrote on 04/17/2020 21:44: >> Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 >>> >>> Printscreen attached. >> Attachments are stripped from the list -- can you put them somewhere >> online ? > > Here it is https://ibb.co/c1dZrTf > > Miroslav Lachman > Ok, the bridges know about their downstream bus numbers, but I see nothing that suggests that they’re being probed. The next step would be bootverbose, but that’s going to be a lot of output to collect in screen captures. Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Kurt Jaeger wrote on 04/17/2020 21:44: Hi! pciconf -lBc pcib12 pciconf -lBc pcib13 Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? Here it is https://ibb.co/c1dZrTf Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Hi! > > pciconf -lBc pcib12 > > pciconf -lBc pcib13 > > Printscreen attached. Attachments are stripped from the list -- can you put them somewhere online ? -- p...@opsec.eu+49 171 3101372Now what ? ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 21:18: On Apr 17, 2020, at 11:46 AM, Miroslav Lachman <000.f...@quip.cz> wrote: Scott Long wrote on 04/17/2020 18:17: You are correct about Intel vs AMD. Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD. Can you send the full output of `lspci -tvv` from linux? Sorry for my late reply. Booting the Linux SystemRescueCd is too slow. lspci -tvv output is attached I tried to connect to SOL by SSH but it shows black screen only. lspci shows drives: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] Kind regards Miroslav Lachman It looks like pcib12 and pcib13 in FreeBSD should be the bridges that have the NVMe devices behind them, but those devices aren’t showing up. I don’t see any obvious code in linux to handle those bridges, so there must be something subtle that we’re missing in FreeBSD. Maybe we’re having trouble enumerating above PCI bus 128? I think that was a problem a few years ago but I also thought it was fixed. Can you do the following in FreeBSD: pciconf -lBc pcib12 pciconf -lBc pcib13 Printscreen attached. Anything else I can provide? Thank you Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
> On Apr 17, 2020, at 11:46 AM, Miroslav Lachman <000.f...@quip.cz> wrote: > > Scott Long wrote on 04/17/2020 18:17: >> You are correct about Intel vs AMD. Comparing the full output of pciconf >> from FreeBSD with the fragment of lspci from Linux suggests that there’s at >> least one set of a PCIe switch and child devices that is not being >> enumerated by FreeBSD. Can you send the full output of `lspci -tvv` from >> linux? > > Sorry for my late reply. Booting the Linux SystemRescueCd is too slow. > > lspci -tvv output is attached > > I tried to connect to SOL by SSH but it shows black screen only. > > lspci shows drives: > > Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] > > Kind regards > Miroslav Lachman > It looks like pcib12 and pcib13 in FreeBSD should be the bridges that have the NVMe devices behind them, but those devices aren’t showing up. I don’t see any obvious code in linux to handle those bridges, so there must be something subtle that we’re missing in FreeBSD. Maybe we’re having trouble enumerating above PCI bus 128? I think that was a problem a few years ago but I also thought it was fixed. Can you do the following in FreeBSD: pciconf -lBc pcib12 pciconf -lBc pcib13 Thanks, Scott ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
On Fri, Apr 17, 2020 at 05:40:55PM +0200, Miroslav Lachman wrote: > Scott Long wrote on 04/17/2020 17:38: > > Can you send me the output of ???pciconf -llv???, either in 12-STABLE or > > 13-CURRENT? Also, can you send me the output of ???dmesg > > I have only iDRAC access to this machine, I cannot copy or export text. Can > I send you printscreen images? You used to be able to ssh to the iDRAC IP and there was a magic command that let you get to the serial console via SSH. I don't have access to Dell hardware any more to dig out the magic command, sorry Regards, Gary ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Warner Losh wrote on 04/17/2020 18:15: No. It was some kind of extra card (Perteq?) managing things in a Dell server. It may be the VMD/VROC stuff, and that driver may be worth a shot, but I'm thinking not. It looks like Doug's code for that is in the tree, though https://reviews.freebsd.org/rS353380. Or are other changes needed? I've been blessed with either being able to turn this off, or not having to deal... vmd is in the kernel of 13-CURRENT installer I tried but I don't see any vmd device. Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
You are correct about Intel vs AMD. Comparing the full output of pciconf from FreeBSD with the fragment of lspci from Linux suggests that there’s at least one set of a PCIe switch and child devices that is not being enumerated by FreeBSD. Can you send the full output of `lspci -tvv` from linux? Thanks, Scott > On Apr 17, 2020, at 9:54 AM, Scott Long wrote: > > Would that be the Intel VMD/VROC stuff? If so, there’s a driver for FreeBSD, > but it’s not well tested yet. Will have to dig in further. > > Scott > > >> On Apr 17, 2020, at 9:50 AM, Warner Losh wrote: >> >> >> >> On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: >> Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or >> 13-CURRENT? Also, can you send me the output of ‘dmesg’? >> >> There was another thread that said there was a raid card in the way... It >> would be cool to find a way to get it out of the way... :) >> >> Warner >> >> Thanks, >> Scott >> >> >>> On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <000.f...@quip.cz> wrote: >>> >>> I already asked on stable@ but as I tried it on 13-CURRENT with the same >>> result I am trying to ask for help here. >>> >>> I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only. >>> There are 2 NVME drives which I wanted to use as ZFS root pool. >>> >>> They are shown in iDRAC >>> >>> Device Description: PCIe SSD in Slot 1 in Bay 1 >>> Device Protocol: NVMe-MI1.0 >>> Model:Dell Express Flash NVMe P4510 1TB SFF >>> Bus: 130 >>> Manufacturer: INTEL >>> Product ID: a54 >>> Revision: VDV1DP23 >>> Enclosure:PCIe SSD Backplane 1 >>> >>> >>> pciconf -l show many things, some of them are named "noneN@pci..." but none >>> "nvme" >>> >>> The is printscreen (12.1 but 13-CURRENT is the same) >>> >>> https://ibb.co/tPnymL7 >>> >>> But I booted Linux SystemRescueCd and nvme devices are there visible in >>> /dev/ >>> https://ibb.co/sj22Nwg >>> >>> There is verbose output of Linux lspci https://ibb.co/dPZTwV1 >>> >>> Linux dmesg contains: >>> nvme nvme0: pci function :81:00.0 >>> nvme nvme1: pci function :82:00.0 >>> nvme nvme0: 32/0/0 default/read/poll queues >>> nvme nvme1: 32/0/0 default/read/poll queues >>> >>> >>> The machine is Dell PowerEdge R6515 with AMD EPYC 7302P >>> >>> >>> More details extracted from iDRAC web interface >>> >>> I found this informations >>> >>> PCIe SSD in Slot 1 in Bay 1 >>> Bus: 82 >>> BusProtocol: PCIE >>> Device: 0 >>> DeviceDescription: PCIe SSD in Slot 1 in Bay 1 >>> DeviceProtocol: NVMe-MI1.0 >>> DeviceType: PCIeSSD >>> DriveFormFactor: 2.5 inch >>> FailurePredicted: NO >>> FQDD: Disk.Bay.1:Enclosure.Internal.0-1 >>> FreeSizeInBytes: Information Not Available >>> Function: 0 >>> HotSpareStatus: Information Not Available >>> InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 >>> Manufacturer: INTEL >>> MaximumCapableSpeed: 8 GT/s >>> MediaType: Solid State Drive >>> Model: Dell Express Flash NVMe P4510 1TB SFF >>> NegotiatedSpeed: 8 GT/s >>> PCIeCapableLinkWidth: x4 >>> PCIeNegotiatedLinkWidth: x4 >>> PrimaryStatus: Ok >>> ProductID: a54 >>> RaidStatus: Information Not Available >>> RAIDType: Unknown >>> RemainingRatedWriteEndurance: 100 % >>> Revision: VDV1DP23 >>> SerialNumber: PHLJxxWF1PN >>> SizeInBytes: 1000204886016 >>> Slot: 1 >>> State: Ready >>> SystemEraseCapability: CryptographicErasePD >>> >>> PCIe SSD in Slot 1 in Bay 1 - PCI Device >>> BusNumber: 130 >>> DataBusWidth: 4x or x4 >>> Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) >>> DeviceDescription: PCIe SSD in Slot 1 in Bay 1 >>> DeviceNumber: 0 >>> DeviceType: PCIDevice >>> FQDD: Disk.Bay.1:Enclosure.Internal.0-1 >>> FunctionNumber: 0 >>> InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 >>> LastSystemInventoryTime: 2020-04-17T03:21:39 >>> LastUpdateTime: 2020-03-31T13:55:06 >>> Manufacturer: Intel Corporation >>> PCIDeviceID: 0A54 >>> PCISubDeviceID: 2003 >>> PCISubVendorID: 1028 >>> PCIVendorID: 8086 >>> SlotLength: 2.5 Inch Drive Form Factor >>> SlotType: PCI Express Gen 3 SFF-8639 >>> >>> >>> Can anybody shed some light what the real problem is? >>> >>> Is the hardware not properly detected or is the driver completely missing? >>> >>> NVME PCIe architecture is out of my knowledge. >>> >>> I really appreciate any help. >>> >>> Kind regards >>> Miroslav Lachman >>> ___ >>> freebsd-current@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" >> >> ___ >> freebsd-current@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > > ___ > fre
Re: PCIe NVME drives not detected on Dell R6515
No. It was some kind of extra card (Perteq?) managing things in a Dell server. It may be the VMD/VROC stuff, and that driver may be worth a shot, but I'm thinking not. It looks like Doug's code for that is in the tree, though https://reviews.freebsd.org/rS353380. Or are other changes needed? I've been blessed with either being able to turn this off, or not having to deal... Warner On Fri, Apr 17, 2020 at 9:54 AM Scott Long wrote: > Would that be the Intel VMD/VROC stuff? If so, there’s a driver for > FreeBSD, but it’s not well tested yet. Will have to dig in further. > > Scott > > > > On Apr 17, 2020, at 9:50 AM, Warner Losh wrote: > > > > > > > > On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: > > Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or > 13-CURRENT? Also, can you send me the output of ‘dmesg’? > > > > There was another thread that said there was a raid card in the way... > It would be cool to find a way to get it out of the way... :) > > > > Warner > > > > Thanks, > > Scott > > > > > > > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <000.f...@quip.cz> > wrote: > > > > > > I already asked on stable@ but as I tried it on 13-CURRENT with the > same result I am trying to ask for help here. > > > > > > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access > only. > > > There are 2 NVME drives which I wanted to use as ZFS root pool. > > > > > > They are shown in iDRAC > > > > > > Device Description: PCIe SSD in Slot 1 in Bay 1 > > > Device Protocol: NVMe-MI1.0 > > > Model:Dell Express Flash NVMe P4510 1TB SFF > > > Bus: 130 > > > Manufacturer: INTEL > > > Product ID: a54 > > > Revision: VDV1DP23 > > > Enclosure:PCIe SSD Backplane 1 > > > > > > > > > pciconf -l show many things, some of them are named "noneN@pci..." > but none "nvme" > > > > > > The is printscreen (12.1 but 13-CURRENT is the same) > > > > > > https://ibb.co/tPnymL7 > > > > > > But I booted Linux SystemRescueCd and nvme devices are there visible > in /dev/ > > > https://ibb.co/sj22Nwg > > > > > > There is verbose output of Linux lspci https://ibb.co/dPZTwV1 > > > > > > Linux dmesg contains: > > > nvme nvme0: pci function :81:00.0 > > > nvme nvme1: pci function :82:00.0 > > > nvme nvme0: 32/0/0 default/read/poll queues > > > nvme nvme1: 32/0/0 default/read/poll queues > > > > > > > > > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P > > > > > > > > > More details extracted from iDRAC web interface > > > > > > I found this informations > > > > > > PCIe SSD in Slot 1 in Bay 1 > > > Bus: 82 > > > BusProtocol: PCIE > > > Device: 0 > > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > > DeviceProtocol: NVMe-MI1.0 > > > DeviceType: PCIeSSD > > > DriveFormFactor: 2.5 inch > > > FailurePredicted: NO > > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > > FreeSizeInBytes: Information Not Available > > > Function: 0 > > > HotSpareStatus: Information Not Available > > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > > Manufacturer: INTEL > > > MaximumCapableSpeed: 8 GT/s > > > MediaType: Solid State Drive > > > Model: Dell Express Flash NVMe P4510 1TB SFF > > > NegotiatedSpeed: 8 GT/s > > > PCIeCapableLinkWidth: x4 > > > PCIeNegotiatedLinkWidth: x4 > > > PrimaryStatus: Ok > > > ProductID: a54 > > > RaidStatus: Information Not Available > > > RAIDType: Unknown > > > RemainingRatedWriteEndurance: 100 % > > > Revision: VDV1DP23 > > > SerialNumber: PHLJxxWF1PN > > > SizeInBytes: 1000204886016 > > > Slot: 1 > > > State: Ready > > > SystemEraseCapability: CryptographicErasePD > > > > > > PCIe SSD in Slot 1 in Bay 1 - PCI Device > > > BusNumber: 130 > > > DataBusWidth: 4x or x4 > > > Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) > > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > > DeviceNumber: 0 > > > DeviceType: PCIDevice > > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > > FunctionNumber: 0 > > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > > LastSystemInventoryTime: 2020-04-17T03:21:39 > > > LastUpdateTime: 2020-03-31T13:55:06 > > > Manufacturer: Intel Corporation > > > PCIDeviceID: 0A54 > > > PCISubDeviceID: 2003 > > > PCISubVendorID: 1028 > > > PCIVendorID: 8086 > > > SlotLength: 2.5 Inch Drive Form Factor > > > SlotType: PCI Express Gen 3 SFF-8639 > > > > > > > > > Can anybody shed some light what the real problem is? > > > > > > Is the hardware not properly detected or is the driver completely > missing? > > > > > > NVME PCIe architecture is out of my knowledge. > > > > > > I really appreciate any help. > > > > > > Kind regards > > > Miroslav Lachman > > > ___ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to " > freebsd-current-unsubscr...@freebsd.org" > > > > _
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 17:54: Would that be the Intel VMD/VROC stuff? If so, there’s a driver for FreeBSD, but it’s not well tested yet. Will have to dig in further. This is AMD EPYC machine. Isn't VROC Intel only thing? On Apr 17, 2020, at 9:50 AM, Warner Losh wrote: On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT? Also, can you send me the output of ‘dmesg’? There was another thread that said there was a raid card in the way... It would be cool to find a way to get it out of the way... :) I didn't find any evidence of RAID card but maybe I cannot look for it. Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Would that be the Intel VMD/VROC stuff? If so, there’s a driver for FreeBSD, but it’s not well tested yet. Will have to dig in further. Scott > On Apr 17, 2020, at 9:50 AM, Warner Losh wrote: > > > > On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: > Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or > 13-CURRENT? Also, can you send me the output of ‘dmesg’? > > There was another thread that said there was a raid card in the way... It > would be cool to find a way to get it out of the way... :) > > Warner > > Thanks, > Scott > > > > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <000.f...@quip.cz> wrote: > > > > I already asked on stable@ but as I tried it on 13-CURRENT with the same > > result I am trying to ask for help here. > > > > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only. > > There are 2 NVME drives which I wanted to use as ZFS root pool. > > > > They are shown in iDRAC > > > > Device Description: PCIe SSD in Slot 1 in Bay 1 > > Device Protocol: NVMe-MI1.0 > > Model:Dell Express Flash NVMe P4510 1TB SFF > > Bus: 130 > > Manufacturer: INTEL > > Product ID: a54 > > Revision: VDV1DP23 > > Enclosure:PCIe SSD Backplane 1 > > > > > > pciconf -l show many things, some of them are named "noneN@pci..." but none > > "nvme" > > > > The is printscreen (12.1 but 13-CURRENT is the same) > > > > https://ibb.co/tPnymL7 > > > > But I booted Linux SystemRescueCd and nvme devices are there visible in > > /dev/ > > https://ibb.co/sj22Nwg > > > > There is verbose output of Linux lspci https://ibb.co/dPZTwV1 > > > > Linux dmesg contains: > > nvme nvme0: pci function :81:00.0 > > nvme nvme1: pci function :82:00.0 > > nvme nvme0: 32/0/0 default/read/poll queues > > nvme nvme1: 32/0/0 default/read/poll queues > > > > > > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P > > > > > > More details extracted from iDRAC web interface > > > > I found this informations > > > > PCIe SSD in Slot 1 in Bay 1 > > Bus: 82 > > BusProtocol: PCIE > > Device: 0 > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > DeviceProtocol: NVMe-MI1.0 > > DeviceType: PCIeSSD > > DriveFormFactor: 2.5 inch > > FailurePredicted: NO > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > FreeSizeInBytes: Information Not Available > > Function: 0 > > HotSpareStatus: Information Not Available > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > Manufacturer: INTEL > > MaximumCapableSpeed: 8 GT/s > > MediaType: Solid State Drive > > Model: Dell Express Flash NVMe P4510 1TB SFF > > NegotiatedSpeed: 8 GT/s > > PCIeCapableLinkWidth: x4 > > PCIeNegotiatedLinkWidth: x4 > > PrimaryStatus: Ok > > ProductID: a54 > > RaidStatus: Information Not Available > > RAIDType: Unknown > > RemainingRatedWriteEndurance: 100 % > > Revision: VDV1DP23 > > SerialNumber: PHLJxxWF1PN > > SizeInBytes: 1000204886016 > > Slot: 1 > > State: Ready > > SystemEraseCapability: CryptographicErasePD > > > > PCIe SSD in Slot 1 in Bay 1 - PCI Device > > BusNumber: 130 > > DataBusWidth: 4x or x4 > > Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > DeviceNumber: 0 > > DeviceType: PCIDevice > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > FunctionNumber: 0 > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > LastSystemInventoryTime: 2020-04-17T03:21:39 > > LastUpdateTime: 2020-03-31T13:55:06 > > Manufacturer: Intel Corporation > > PCIDeviceID: 0A54 > > PCISubDeviceID: 2003 > > PCISubVendorID: 1028 > > PCIVendorID: 8086 > > SlotLength: 2.5 Inch Drive Form Factor > > SlotType: PCI Express Gen 3 SFF-8639 > > > > > > Can anybody shed some light what the real problem is? > > > > Is the hardware not properly detected or is the driver completely missing? > > > > NVME PCIe architecture is out of my knowledge. > > > > I really appreciate any help. > > > > Kind regards > > Miroslav Lachman > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
On Fri, Apr 17, 2020 at 9:39 AM Scott Long wrote: > Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or > 13-CURRENT? Also, can you send me the output of ‘dmesg’? > There was another thread that said there was a raid card in the way... It would be cool to find a way to get it out of the way... :) Warner > Thanks, > Scott > > > > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <000.f...@quip.cz> wrote: > > > > I already asked on stable@ but as I tried it on 13-CURRENT with the > same result I am trying to ask for help here. > > > > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access > only. > > There are 2 NVME drives which I wanted to use as ZFS root pool. > > > > They are shown in iDRAC > > > > Device Description: PCIe SSD in Slot 1 in Bay 1 > > Device Protocol: NVMe-MI1.0 > > Model:Dell Express Flash NVMe P4510 1TB SFF > > Bus: 130 > > Manufacturer: INTEL > > Product ID: a54 > > Revision: VDV1DP23 > > Enclosure:PCIe SSD Backplane 1 > > > > > > pciconf -l show many things, some of them are named "noneN@pci..." but > none "nvme" > > > > The is printscreen (12.1 but 13-CURRENT is the same) > > > > https://ibb.co/tPnymL7 > > > > But I booted Linux SystemRescueCd and nvme devices are there visible in > /dev/ > > https://ibb.co/sj22Nwg > > > > There is verbose output of Linux lspci https://ibb.co/dPZTwV1 > > > > Linux dmesg contains: > > nvme nvme0: pci function :81:00.0 > > nvme nvme1: pci function :82:00.0 > > nvme nvme0: 32/0/0 default/read/poll queues > > nvme nvme1: 32/0/0 default/read/poll queues > > > > > > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P > > > > > > More details extracted from iDRAC web interface > > > > I found this informations > > > > PCIe SSD in Slot 1 in Bay 1 > > Bus: 82 > > BusProtocol: PCIE > > Device: 0 > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > DeviceProtocol: NVMe-MI1.0 > > DeviceType: PCIeSSD > > DriveFormFactor: 2.5 inch > > FailurePredicted: NO > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > FreeSizeInBytes: Information Not Available > > Function: 0 > > HotSpareStatus: Information Not Available > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > Manufacturer: INTEL > > MaximumCapableSpeed: 8 GT/s > > MediaType: Solid State Drive > > Model: Dell Express Flash NVMe P4510 1TB SFF > > NegotiatedSpeed: 8 GT/s > > PCIeCapableLinkWidth: x4 > > PCIeNegotiatedLinkWidth: x4 > > PrimaryStatus: Ok > > ProductID: a54 > > RaidStatus: Information Not Available > > RAIDType: Unknown > > RemainingRatedWriteEndurance: 100 % > > Revision: VDV1DP23 > > SerialNumber: PHLJxxWF1PN > > SizeInBytes: 1000204886016 > > Slot: 1 > > State: Ready > > SystemEraseCapability: CryptographicErasePD > > > > PCIe SSD in Slot 1 in Bay 1 - PCI Device > > BusNumber: 130 > > DataBusWidth: 4x or x4 > > Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) > > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > > DeviceNumber: 0 > > DeviceType: PCIDevice > > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > > FunctionNumber: 0 > > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > > LastSystemInventoryTime: 2020-04-17T03:21:39 > > LastUpdateTime: 2020-03-31T13:55:06 > > Manufacturer: Intel Corporation > > PCIDeviceID: 0A54 > > PCISubDeviceID: 2003 > > PCISubVendorID: 1028 > > PCIVendorID: 8086 > > SlotLength: 2.5 Inch Drive Form Factor > > SlotType: PCI Express Gen 3 SFF-8639 > > > > > > Can anybody shed some light what the real problem is? > > > > Is the hardware not properly detected or is the driver completely > missing? > > > > NVME PCIe architecture is out of my knowledge. > > > > I really appreciate any help. > > > > Kind regards > > Miroslav Lachman > > ___ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscr...@freebsd.org" > > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" > ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Scott Long wrote on 04/17/2020 17:38: Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT? Also, can you send me the output of ‘dmesg’? I have only iDRAC access to this machine, I cannot copy or export text. Can I send you printscreen images? Thank you! Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: PCIe NVME drives not detected on Dell R6515
Can you send me the output of ‘pciconf -llv’, either in 12-STABLE or 13-CURRENT? Also, can you send me the output of ‘dmesg’? Thanks, Scott > On Apr 17, 2020, at 5:23 AM, Miroslav Lachman <000.f...@quip.cz> wrote: > > I already asked on stable@ but as I tried it on 13-CURRENT with the same > result I am trying to ask for help here. > > I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only. > There are 2 NVME drives which I wanted to use as ZFS root pool. > > They are shown in iDRAC > > Device Description: PCIe SSD in Slot 1 in Bay 1 > Device Protocol: NVMe-MI1.0 > Model:Dell Express Flash NVMe P4510 1TB SFF > Bus: 130 > Manufacturer: INTEL > Product ID: a54 > Revision: VDV1DP23 > Enclosure:PCIe SSD Backplane 1 > > > pciconf -l show many things, some of them are named "noneN@pci..." but none > "nvme" > > The is printscreen (12.1 but 13-CURRENT is the same) > > https://ibb.co/tPnymL7 > > But I booted Linux SystemRescueCd and nvme devices are there visible in /dev/ > https://ibb.co/sj22Nwg > > There is verbose output of Linux lspci https://ibb.co/dPZTwV1 > > Linux dmesg contains: > nvme nvme0: pci function :81:00.0 > nvme nvme1: pci function :82:00.0 > nvme nvme0: 32/0/0 default/read/poll queues > nvme nvme1: 32/0/0 default/read/poll queues > > > The machine is Dell PowerEdge R6515 with AMD EPYC 7302P > > > More details extracted from iDRAC web interface > > I found this informations > > PCIe SSD in Slot 1 in Bay 1 > Bus: 82 > BusProtocol: PCIE > Device: 0 > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > DeviceProtocol: NVMe-MI1.0 > DeviceType: PCIeSSD > DriveFormFactor: 2.5 inch > FailurePredicted: NO > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > FreeSizeInBytes: Information Not Available > Function: 0 > HotSpareStatus: Information Not Available > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > Manufacturer: INTEL > MaximumCapableSpeed: 8 GT/s > MediaType: Solid State Drive > Model: Dell Express Flash NVMe P4510 1TB SFF > NegotiatedSpeed: 8 GT/s > PCIeCapableLinkWidth: x4 > PCIeNegotiatedLinkWidth: x4 > PrimaryStatus: Ok > ProductID: a54 > RaidStatus: Information Not Available > RAIDType: Unknown > RemainingRatedWriteEndurance: 100 % > Revision: VDV1DP23 > SerialNumber: PHLJxxWF1PN > SizeInBytes: 1000204886016 > Slot: 1 > State: Ready > SystemEraseCapability: CryptographicErasePD > > PCIe SSD in Slot 1 in Bay 1 - PCI Device > BusNumber: 130 > DataBusWidth: 4x or x4 > Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) > DeviceDescription: PCIe SSD in Slot 1 in Bay 1 > DeviceNumber: 0 > DeviceType: PCIDevice > FQDD: Disk.Bay.1:Enclosure.Internal.0-1 > FunctionNumber: 0 > InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 > LastSystemInventoryTime: 2020-04-17T03:21:39 > LastUpdateTime: 2020-03-31T13:55:06 > Manufacturer: Intel Corporation > PCIDeviceID: 0A54 > PCISubDeviceID: 2003 > PCISubVendorID: 1028 > PCIVendorID: 8086 > SlotLength: 2.5 Inch Drive Form Factor > SlotType: PCI Express Gen 3 SFF-8639 > > > Can anybody shed some light what the real problem is? > > Is the hardware not properly detected or is the driver completely missing? > > NVME PCIe architecture is out of my knowledge. > > I really appreciate any help. > > Kind regards > Miroslav Lachman > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
PCIe NVME drives not detected on Dell R6515
I already asked on stable@ but as I tried it on 13-CURRENT with the same result I am trying to ask for help here. I have rented dedicated server Dell PowerEdge R6515 with iDRAC access only. There are 2 NVME drives which I wanted to use as ZFS root pool. They are shown in iDRAC Device Description: PCIe SSD in Slot 1 in Bay 1 Device Protocol: NVMe-MI1.0 Model:Dell Express Flash NVMe P4510 1TB SFF Bus: 130 Manufacturer: INTEL Product ID: a54 Revision: VDV1DP23 Enclosure:PCIe SSD Backplane 1 pciconf -l show many things, some of them are named "noneN@pci..." but none "nvme" The is printscreen (12.1 but 13-CURRENT is the same) https://ibb.co/tPnymL7 But I booted Linux SystemRescueCd and nvme devices are there visible in /dev/ https://ibb.co/sj22Nwg There is verbose output of Linux lspci https://ibb.co/dPZTwV1 Linux dmesg contains: nvme nvme0: pci function :81:00.0 nvme nvme1: pci function :82:00.0 nvme nvme0: 32/0/0 default/read/poll queues nvme nvme1: 32/0/0 default/read/poll queues The machine is Dell PowerEdge R6515 with AMD EPYC 7302P More details extracted from iDRAC web interface I found this informations PCIe SSD in Slot 1 in Bay 1 Bus: 82 BusProtocol: PCIE Device: 0 DeviceDescription: PCIe SSD in Slot 1 in Bay 1 DeviceProtocol: NVMe-MI1.0 DeviceType: PCIeSSD DriveFormFactor: 2.5 inch FailurePredicted: NO FQDD: Disk.Bay.1:Enclosure.Internal.0-1 FreeSizeInBytes: Information Not Available Function: 0 HotSpareStatus: Information Not Available InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 Manufacturer: INTEL MaximumCapableSpeed: 8 GT/s MediaType: Solid State Drive Model: Dell Express Flash NVMe P4510 1TB SFF NegotiatedSpeed: 8 GT/s PCIeCapableLinkWidth: x4 PCIeNegotiatedLinkWidth: x4 PrimaryStatus: Ok ProductID: a54 RaidStatus: Information Not Available RAIDType: Unknown RemainingRatedWriteEndurance: 100 % Revision: VDV1DP23 SerialNumber: PHLJxxWF1PN SizeInBytes: 1000204886016 Slot: 1 State: Ready SystemEraseCapability: CryptographicErasePD PCIe SSD in Slot 1 in Bay 1 - PCI Device BusNumber: 130 DataBusWidth: 4x or x4 Description: Express Flash NVMe 1.0 TB 2.5" U.2 (P4510) DeviceDescription: PCIe SSD in Slot 1 in Bay 1 DeviceNumber: 0 DeviceType: PCIDevice FQDD: Disk.Bay.1:Enclosure.Internal.0-1 FunctionNumber: 0 InstanceID: Disk.Bay.1:Enclosure.Internal.0-1 LastSystemInventoryTime: 2020-04-17T03:21:39 LastUpdateTime: 2020-03-31T13:55:06 Manufacturer: Intel Corporation PCIDeviceID: 0A54 PCISubDeviceID: 2003 PCISubVendorID: 1028 PCIVendorID: 8086 SlotLength: 2.5 Inch Drive Form Factor SlotType: PCI Express Gen 3 SFF-8639 Can anybody shed some light what the real problem is? Is the hardware not properly detected or is the driver completely missing? NVME PCIe architecture is out of my knowledge. I really appreciate any help. Kind regards Miroslav Lachman ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
