Re: support of PCIe NVME drives

[Chris]

On Thu, 16 Apr 2020 19:57:21 -0700 Mel Pilgrim list_free...@bluerosetech.com 
said


On 2020-04-16 12:30, Miroslav Lachman wrote:
> Pete Wright wrote on 04/16/2020 20:23:
>>
>>
>> On 4/16/20 11:12 AM, Miroslav Lachman wrote:
>>> Kurt Jaeger wrote on 04/16/2020 20:07:
> 
>> I would try booting via UEFI if you can.  I just installed a laptop 
>> yesterday which has a nvme root device, it was detected by the 
>> 12-STABLE snapshot I used to boot from.  no other modifications were 
>> necessary on my end.
> 
> I changed BIOS settings to use UEFI boot method, booted 12.1 installer 
> ISO but without luck. Still no NVME disks :(
> 
> You can see it on printscreen from iDRAC https://ibb.co/tPnymL7
> 
> Anything more I can test?


Looking at server specs, the R6515's NVME support is only through the 
PERC S150 RAID controller.  If that's the case, I'm pretty sure you're 
out of luck.  The PERC S-series controllers are software-based RAID that 
require Dell's Windows or Linux drivers.  You'd need a PERC H-series 
card to get support in FreeBSD.


Someone please correct me if I'm wrong?

As I mentioned. I was suspicious of this. He should be able to flash the card,
making it a pass. I do a lot of them. If someone doesn't beat me to it. I'll
dig through what I have, and see if I can't find the right image(s), and 
program(s).


--Chris


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Mel Pilgrim]

On 2020-04-16 12:30, Miroslav Lachman wrote:

Pete Wright wrote on 04/16/2020 20:23:



On 4/16/20 11:12 AM, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:


I would try booting via UEFI if you can.  I just installed a laptop 
yesterday which has a nvme root device, it was detected by the 
12-STABLE snapshot I used to boot from.  no other modifications were 
necessary on my end.


I changed BIOS settings to use UEFI boot method, booted 12.1 installer 
ISO but without luck. Still no NVME disks :(


You can see it on printscreen from iDRAC https://ibb.co/tPnymL7

Anything more I can test?


Looking at server specs, the R6515's NVME support is only through the 
PERC S150 RAID controller.  If that's the case, I'm pretty sure you're 
out of luck.  The PERC S-series controllers are software-based RAID that 
require Dell's Windows or Linux drivers.  You'd need a PERC H-series 
card to get support in FreeBSD.


Someone please correct me if I'm wrong?
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Issue with gpart "Device Busy"

[ian]

Thank you for the advices, however it turns out it was user error.. I swapped 
the drive in from an apple machine, the new drive took ada0 and my existing 
drive moved to ada1.

I’m just playing with ZFS at the moment and what I didn’t realise is that the 
system swap was still assigned to ada0p2 but of course instead of being on my 
new ZFS drive was instead pointing to an apple-hfs data partition on the 
freshly added drive.

For some reason I assumed it was something to do with HFSFUSE and gpart when in 
actual fact it was just me not paying attention to the basics!


> On 16 Apr 2020, at 18:44, Harry Schmalzbauer  > wrote:
> 
> Am 15.04.2020 um 20:35 schrieb i...@dijix.com :
>> I have an issue with gpart, it will not let me delete partition ada0p2 
>> responding with “Device Busy”
>> The man page gpart(8) says this may be shown if a partition exists but I 
>> cannot seem to delete partition 2 in my case via gpart delete or gpart 
>> destroy
>> 
>> This is a used disk but new to the machine, I can modify the partition type 
>> and create partitions before and after partition 2 but I cannot delete it.
>> 
>> Here’s what I have tried so far:
>> 
>> 
>> root@beastie:~ # gpart show
>> =>34  1250263661  ada0  GPT  (596G)
>>34  409606- free -  (200M)
>>409640  1249591904 2  freebsd-ufs  (596G)
>> 1250001544  262151- free -  (128M)
>> 
>> =>   40  976773088  ada1  GPT  (466G)
>>   40   1024 1  freebsd-boot  (512K)
>> 1064984- free -  (492K)
>> 20484194304 2  freebsd-swap  (2.0G)
>>  4196352  972576768 3  freebsd-zfs  (464G)
>> 976773120  8- free -  (4.0K)
>> 
>> root@beastie:~ # gpart delete -i2 ada0
>> gpart: Device busy
> :
> :
>> :
>> root@beastie:~ # gpart destroy -F ada0
>> gpart: Device busy
> 
> There might still be situations where 'sysctl kern.geom.debugflags=16' helps, 
> but I never needed it in the last years (since 7.x I guess).
> Are you sure p2 (-i2) of ada0, most likely home for a ufs filesystem, isn't 
> mounted anymore? Was it a mountpoint inside a jail?  Stopping the jail might 
> leave network related active sockets blocking the filesystem (reboot without 
> starting the jail before deleteing the partition should work in that case).
> 
> -harry

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Miroslav Lachman]

Pete Wright wrote on 04/16/2020 23:06:



iDRAC does not allow me to do anything with the drives.
But I booted Linux SystemRescueCd and nvme devices are there visible 
in /dev/

printscreen https://ibb.co/sj22Nwg

So I think the HW is OK, but FreeBSD does not recognize the controller?

might be interesting to see what the dmesg buffer and lspci output for 
this device is under linux.  that should give pointers to how the device 
is being presented to the kernel.


There is verbose output of lspci https://ibb.co/dPZTwV1

dmesg contains:
nvme nvme0: pci function :81:00.0
nvme nvme1: pci function :82:00.0
nvme nvme0: 32/0/0 default/read/poll queues
nvme nvme1: 32/0/0 default/read/poll queues

Kind regards
Miroslav Lachman
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Chris]

On Thu, 16 Apr 2020 22:57:10 +0200 Miroslav Lachman 000.f...@quip.cz said


Chuck Tuffli wrote on 04/16/2020 22:29:
> On Thu, Apr 16, 2020 at 12:30 PM Miroslav Lachman <000.f...@quip.cz> wrote:
>>
>> Pete Wright wrote on 04/16/2020 20:23:
>>>
>>>
>>> On 4/16/20 11:12 AM, Miroslav Lachman wrote:
 Kurt Jaeger wrote on 04/16/2020 20:07:
>>
>>> I would try booting via UEFI if you can.  I just installed a laptop
>>> yesterday which has a nvme root device, it was detected by the 12-STABLE
>>> snapshot I used to boot from.  no other modifications were necessary on
>>> my end.
>>
>> I changed BIOS settings to use UEFI boot method, booted 12.1 installer
>> ISO but without luck. Still no NVME disks :(
>>
>> You can see it on printscreen from iDRAC https://ibb.co/tPnymL7
>>
>> Anything more I can test?
> 
> Fair warning, I don't deal much with Dell, but it's curious that iDrac

> lists the device protocol as MI
>> Device Protocol:  NVMe-MI1.0
> 
> MI or the Management Interface is defined by NVMe, but it isn't the

> same as a block interface. MI typically uses SMBUS and not PCIe to
> communicate with the device. So it's possible that the device is
> visible to iDrac via SMBUS but may not be on the PCIe bus for
> $reasons. Does iDrac have any tools to interact with the device via
> MI?

iDRAC does not allow me to do anything with the drives.
But I booted Linux SystemRescueCd and nvme devices are there visible in 
/dev/

printscreen https://ibb.co/sj22Nwg

So I think the HW is OK, but FreeBSD does not recognize the controller?


Does mps(4) or any of the other (often Dell) rebranded LSI drivers expose 
anything?



Kind regards
Miroslav Lachman


--Chris


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Pete Wright]

On 4/16/20 1:57 PM, Miroslav Lachman wrote:

Chuck Tuffli wrote on 04/16/2020 22:29:
On Thu, Apr 16, 2020 at 12:30 PM Miroslav Lachman <000.f...@quip.cz> 
wrote:


Pete Wright wrote on 04/16/2020 20:23:



On 4/16/20 11:12 AM, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:



I would try booting via UEFI if you can.  I just installed a laptop
yesterday which has a nvme root device, it was detected by the 
12-STABLE
snapshot I used to boot from.  no other modifications were 
necessary on

my end.


I changed BIOS settings to use UEFI boot method, booted 12.1 installer
ISO but without luck. Still no NVME disks :(

You can see it on printscreen from iDRAC https://ibb.co/tPnymL7

Anything more I can test?


Fair warning, I don't deal much with Dell, but it's curious that iDrac
lists the device protocol as MI

Device Protocol:  NVMe-MI1.0


MI or the Management Interface is defined by NVMe, but it isn't the
same as a block interface. MI typically uses SMBUS and not PCIe to
communicate with the device. So it's possible that the device is
visible to iDrac via SMBUS but may not be on the PCIe bus for
$reasons. Does iDrac have any tools to interact with the device via
MI?


iDRAC does not allow me to do anything with the drives.
But I booted Linux SystemRescueCd and nvme devices are there visible 
in /dev/

printscreen https://ibb.co/sj22Nwg

So I think the HW is OK, but FreeBSD does not recognize the controller?

might be interesting to see what the dmesg buffer and lspci output for 
this device is under linux.  that should give pointers to how the device 
is being presented to the kernel.


-p

--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Miroslav Lachman]

Chuck Tuffli wrote on 04/16/2020 22:29:

On Thu, Apr 16, 2020 at 12:30 PM Miroslav Lachman <000.f...@quip.cz> wrote:


Pete Wright wrote on 04/16/2020 20:23:



On 4/16/20 11:12 AM, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:



I would try booting via UEFI if you can.  I just installed a laptop
yesterday which has a nvme root device, it was detected by the 12-STABLE
snapshot I used to boot from.  no other modifications were necessary on
my end.


I changed BIOS settings to use UEFI boot method, booted 12.1 installer
ISO but without luck. Still no NVME disks :(

You can see it on printscreen from iDRAC https://ibb.co/tPnymL7

Anything more I can test?


Fair warning, I don't deal much with Dell, but it's curious that iDrac
lists the device protocol as MI

Device Protocol:  NVMe-MI1.0


MI or the Management Interface is defined by NVMe, but it isn't the
same as a block interface. MI typically uses SMBUS and not PCIe to
communicate with the device. So it's possible that the device is
visible to iDrac via SMBUS but may not be on the PCIe bus for
$reasons. Does iDrac have any tools to interact with the device via
MI?


iDRAC does not allow me to do anything with the drives.
But I booted Linux SystemRescueCd and nvme devices are there visible in 
/dev/

printscreen https://ibb.co/sj22Nwg

So I think the HW is OK, but FreeBSD does not recognize the controller?

Kind regards
Miroslav Lachman
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Chuck Tuffli]

On Thu, Apr 16, 2020 at 12:30 PM Miroslav Lachman <000.f...@quip.cz> wrote:
>
> Pete Wright wrote on 04/16/2020 20:23:
> >
> >
> > On 4/16/20 11:12 AM, Miroslav Lachman wrote:
> >> Kurt Jaeger wrote on 04/16/2020 20:07:
>
> > I would try booting via UEFI if you can.  I just installed a laptop
> > yesterday which has a nvme root device, it was detected by the 12-STABLE
> > snapshot I used to boot from.  no other modifications were necessary on
> > my end.
>
> I changed BIOS settings to use UEFI boot method, booted 12.1 installer
> ISO but without luck. Still no NVME disks :(
>
> You can see it on printscreen from iDRAC https://ibb.co/tPnymL7
>
> Anything more I can test?

Fair warning, I don't deal much with Dell, but it's curious that iDrac
lists the device protocol as MI
> Device Protocol:  NVMe-MI1.0

MI or the Management Interface is defined by NVMe, but it isn't the
same as a block interface. MI typically uses SMBUS and not PCIe to
communicate with the device. So it's possible that the device is
visible to iDrac via SMBUS but may not be on the PCIe bus for
$reasons. Does iDrac have any tools to interact with the device via
MI?

--chuck
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Miroslav Lachman]

Navdeep Parhar wrote on 04/16/2020 21:51:

On 4/16/20 12:30 PM, Miroslav Lachman wrote:

Pete Wright wrote on 04/16/2020 20:23:



On 4/16/20 11:12 AM, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:



I would try booting via UEFI if you can.  I just installed a laptop
yesterday which has a nvme root device, it was detected by the
12-STABLE snapshot I used to boot from.  no other modifications were
necessary on my end.


I changed BIOS settings to use UEFI boot method, booted 12.1 installer
ISO but without luck. Still no NVME disks :(

You can see it on printscreen from iDRAC https://ibb.co/tPnymL7

Anything more I can test?


Does the nvme controller show up in pciconf -l?

# pciconf -l | grep nvme


Empty result.

pciconf -l show many things, tome of them are named "noneN@pci..."

The machine is Dell PowerEdge R6515 with AMD EPYC 7302P

Is it possible that the controller is not recognized?

Miroslav Lachman
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Issue with gpart "Device Busy"

[Paul Mather]

On Apr 15, 2020, at 2:35 PM, i...@dijix.com wrote:

I have an issue with gpart, it will not let me delete partition ada0p2  
responding with “Device Busy”
The man page gpart(8) says this may be shown if a partition exists but I  
cannot seem to delete partition 2 in my case via gpart delete or gpart  
destroy


This is a used disk but new to the machine, I can modify the partition  
type and create partitions before and after partition 2 but I cannot  
delete it.


Here’s what I have tried so far:


root@beastie:~ # gpart show
=>34  1250263661  ada0  GPT  (596G)
   34  409606- free -  (200M)
   409640  1249591904 2  freebsd-ufs  (596G)
1250001544  262151- free -  (128M)

=>   40  976773088  ada1  GPT  (466G)
  40   1024 1  freebsd-boot  (512K)
1064984- free -  (492K)
20484194304 2  freebsd-swap  (2.0G)
 4196352  972576768 3  freebsd-zfs  (464G)
976773120  8- free -  (4.0K)

root@beastie:~ # gpart delete -i2 ada0
gpart: Device busy

root@beastie:~ # gpart add -t freebsd-boot ada0
ada0p1 added

root@beastie:~ # gpart show
=>34  1250263661  ada0  GPT  (596G)
   34  409606 1  freebsd-boot  (200M)
   409640  1249591904 2  freebsd-ufs  (596G)
1250001544  262151- free -  (128M)

=>   40  976773088  ada1  GPT  (466G)
  40   1024 1  freebsd-boot  (512K)
1064984- free -  (492K)
20484194304 2  freebsd-swap  (2.0G)
 4196352  972576768 3  freebsd-zfs  (464G)
976773120  8- free -  (4.0K)

root@beastie:~ # gpart delete -i2 ada0
gpart: Device busy

root@beastie:~ # gpart delete -i1 ada0
ada0p1 deleted

root@beastie:~ # gpart show
=>34  1250263661  ada0  GPT  (596G)
   34  409606- free -  (200M)
   409640  1249591904 2  freebsd-ufs  (596G)
1250001544  262151- free -  (128M)

=>   40  976773088  ada1  GPT  (466G)
  40   1024 1  freebsd-boot  (512K)
1064984- free -  (492K)
20484194304 2  freebsd-swap  (2.0G)
 4196352  972576768 3  freebsd-zfs  (464G)
976773120  8- free -  (4.0K)

root@beastie:~ # gpart destroy -F ada0
gpart: Device busy

root@beastie:~ # gpart modify -i2 -t freebsd-boot ada0
ada0p2 modified
root@beastie:~ # gpart show
=>34  1250263661  ada0  GPT  (596G)
   34  409606- free -  (200M)
   409640  1249591904 2  freebsd-boot  (596G)
1250001544  262151- free -  (128M)

=>   40  976773088  ada1  GPT  (466G)
  40   1024 1  freebsd-boot  (512K)
1064984- free -  (492K)
20484194304 2  freebsd-swap  (2.0G)
 4196352  972576768 3  freebsd-zfs  (464G)
976773120  8- free -  (4.0K)


I’m not sure where to go from here, I’m tempted to pull the drive and  
reformat elsewhere


I have all tried dd’ing the disk as root but dd /dev/ada0 errors with  
unauthorised.


Am I missing something obvious?



I don't know whether it can be considered obvious, but, in my experience,  
this sort of inability to delete a partition can happen because another  
GEOM layer has that partition open under some other guise.  Typical  
culprits are the various "label" classes and other GEOM classes that  
"autodetect" things belonging to them.


A case in point that happened to me recently: I was trying to install on  
what I thought were two empty drives.  Unfortunately, when it came to  
actually partitioning ada0 and ada1 it failed.  It turned out that the  
drives were previously used in a "fake RAID" setup and GEOM_RAID was  
detecting an old RAID volume on the drives because the RAID metadata/label  
was being detected.  This meant I couldn't directly access the underlying  
devices, ada0 and ada1.


My solution was to use graid to destroy the errant RAID volume, after which  
I was able to write/partition ada0 and ada1 directly.


Sometimes it is sufficient to disable autodetection of various label  
classes in /boot/loader.conf to be able to access the lower devices.


Cheers,

Paul.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Navdeep Parhar]

On 4/16/20 12:30 PM, Miroslav Lachman wrote:
> Pete Wright wrote on 04/16/2020 20:23:
>>
>>
>> On 4/16/20 11:12 AM, Miroslav Lachman wrote:
>>> Kurt Jaeger wrote on 04/16/2020 20:07:
> 
>> I would try booting via UEFI if you can.  I just installed a laptop
>> yesterday which has a nvme root device, it was detected by the
>> 12-STABLE snapshot I used to boot from.  no other modifications were
>> necessary on my end.
> 
> I changed BIOS settings to use UEFI boot method, booted 12.1 installer
> ISO but without luck. Still no NVME disks :(
> 
> You can see it on printscreen from iDRAC https://ibb.co/tPnymL7
> 
> Anything more I can test?

Does the nvme controller show up in pciconf -l?

# pciconf -l | grep nvme

Regards,
Navdeep

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Miroslav Lachman]

Pete Wright wrote on 04/16/2020 20:23:



On 4/16/20 11:12 AM, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:


I would try booting via UEFI if you can.  I just installed a laptop 
yesterday which has a nvme root device, it was detected by the 12-STABLE 
snapshot I used to boot from.  no other modifications were necessary on 
my end.


I changed BIOS settings to use UEFI boot method, booted 12.1 installer 
ISO but without luck. Still no NVME disks :(


You can see it on printscreen from iDRAC https://ibb.co/tPnymL7

Anything more I can test?

Kind regards
Miroslav Lachman
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Rebecca Cran]

On 4/16/20 12:23 PM, Pete Wright wrote:

I would try booting via UEFI if you can.  I just installed a laptop 
yesterday which has a nvme root device, it was detected by the 
12-STABLE snapshot I used to boot from.  no other modifications were 
necessary on my end.


I can confirm, NVMe drives tend to not have any support under the 
traditional/legacy BIOS, and require UEFI.



--
Rebecca Cran


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Mel Pilgrim]

On 2020-04-16 11:12, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:

Hi!

I was requested to install FreeBSD 11.3 on a new Dell machine with 
only 2
NVME drives in ZFS mirror. The problem is that installer does not see 
the

drives. Are there any special procedure to use NVME drives for
installation a later for booting?


I use 2 NVMe drives as zfs mirror to boot from on my testbox,
but it runs CURRENT, since approx. November 2018.

So maybe try it with 12.1 ? I know, that does not help if you are asked
to install 11.3, but at least it gives you an idea...



I tried 12.1 few minutes ago but the result is the same - no NVME drives 
listed.

Should I try something with kernel modules, some sysctl tweaks?
Should I try UEFI boot? (I never did)


Make sure the UEFI BIOS can see them and enable dual-mode or UEFI-only 
boot. IIRC I couldn't legacy boot from the NVME drives.


My home and work servers have UEFI-boot gptzfsboot/root-on-zfs NVME 
drives that initially started with 11.2 and now run 12.1.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Karl Denninger]

On 4/16/2020 13:23, Pete Wright wrote:
>
>
> On 4/16/20 11:12 AM, Miroslav Lachman wrote:
>> Kurt Jaeger wrote on 04/16/2020 20:07:
>>> Hi!
>>>
 I was requested to install FreeBSD 11.3 on a new Dell machine with
 only 2
 NVME drives in ZFS mirror. The problem is that installer does not
 see the
 drives. Are there any special procedure to use NVME drives for
 installation a later for booting?
>>>
>>> I use 2 NVMe drives as zfs mirror to boot from on my testbox,
>>> but it runs CURRENT, since approx. November 2018.
>>>
>>> So maybe try it with 12.1 ? I know, that does not help if you are asked
>>> to install 11.3, but at least it gives you an idea...
>>>
>>
>> I tried 12.1 few minutes ago but the result is the same - no NVME
>> drives listed.
>> Should I try something with kernel modules, some sysctl tweaks?
>> Should I try UEFI boot? (I never did)
>>
>
> I would try booting via UEFI if you can.  I just installed a laptop
> yesterday which has a nvme root device, it was detected by the
> 12-STABLE snapshot I used to boot from.  no other modifications were
> necessary on my end.
>
> -pete
>
Yeah my Lenovo Carbon X1 has an nVME drive in it, and nothing else -
12-Stable found it immediately and works fine.

-- 
Karl Denninger
k...@denninger.net 
/The Market Ticker/
/[S/MIME encrypted email preferred]/


smime.p7s
Description: S/MIME Cryptographic Signature

Re: support of PCIe NVME drives

[Pete Wright]

On 4/16/20 11:12 AM, Miroslav Lachman wrote:

Kurt Jaeger wrote on 04/16/2020 20:07:

Hi!

I was requested to install FreeBSD 11.3 on a new Dell machine with 
only 2
NVME drives in ZFS mirror. The problem is that installer does not 
see the

drives. Are there any special procedure to use NVME drives for
installation a later for booting?


I use 2 NVMe drives as zfs mirror to boot from on my testbox,
but it runs CURRENT, since approx. November 2018.

So maybe try it with 12.1 ? I know, that does not help if you are asked
to install 11.3, but at least it gives you an idea...



I tried 12.1 few minutes ago but the result is the same - no NVME 
drives listed.

Should I try something with kernel modules, some sysctl tweaks?
Should I try UEFI boot? (I never did)



I would try booting via UEFI if you can.  I just installed a laptop 
yesterday which has a nvme root device, it was detected by the 12-STABLE 
snapshot I used to boot from.  no other modifications were necessary on 
my end.


-pete

--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: support of PCIe NVME drives

[Miroslav Lachman]

Kurt Jaeger wrote on 04/16/2020 20:07:

Hi!


I was requested to install FreeBSD 11.3 on a new Dell machine with only 2
NVME drives in ZFS mirror. The problem is that installer does not see the
drives. Are there any special procedure to use NVME drives for
installation a later for booting?


I use 2 NVMe drives as zfs mirror to boot from on my testbox,
but it runs CURRENT, since approx. November 2018.

So maybe try it with 12.1 ? I know, that does not help if you are asked
to install 11.3, but at least it gives you an idea...



I tried 12.1 few minutes ago but the result is the same - no NVME drives 
listed.

Should I try something with kernel modules, some sysctl tweaks?
Should I try UEFI boot? (I never did)

Kind regards
Miroslav Lachman

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

support of PCIe NVME drives

[Miroslav Lachman]

I was requested to install FreeBSD 11.3 on a new Dell machine with only 
2 NVME drives in ZFS mirror. The problem is that installer does not see 
the drives. Are there any special procedure to use NVME drives for 
installation a later for booting?


Drives are described in iDRAC:

Device Description:   PCIe SSD in Slot 1 in Bay 1
Device Protocol:  NVMe-MI1.0
Model:Dell Express Flash NVMe P4510 1TB SFF
Bus:  130
Manufacturer: INTEL
Product ID:   a54
Revision: VDV1DP23
Enclosure:PCIe SSD Backplane 1

There is no special HW RAID card.

I switched to a shell in an installer and NVME drives are not listen in 
dmesg.boot.


I tried to manually load nvme and nvd modules but the error messages 
says they are already in the installer kernel.


How can I make it work?


Kind regards
Miroslav Lachman
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Issue with gpart "Device Busy"

[Harry Schmalzbauer]

Am 15.04.2020 um 20:35 schrieb i...@dijix.com:

I have an issue with gpart, it will not let me delete partition ada0p2 
responding with “Device Busy”
The man page gpart(8) says this may be shown if a partition exists but I cannot 
seem to delete partition 2 in my case via gpart delete or gpart destroy

This is a used disk but new to the machine, I can modify the partition type and 
create partitions before and after partition 2 but I cannot delete it.

Here’s what I have tried so far:


root@beastie:~ # gpart show
=>34  1250263661  ada0  GPT  (596G)
34  409606- free -  (200M)
409640  1249591904 2  freebsd-ufs  (596G)
1250001544  262151- free -  (128M)

=>   40  976773088  ada1  GPT  (466G)
   40   1024 1  freebsd-boot  (512K)
 1064984- free -  (492K)
 20484194304 2  freebsd-swap  (2.0G)
  4196352  972576768 3  freebsd-zfs  (464G)
976773120  8- free -  (4.0K)

root@beastie:~ # gpart delete -i2 ada0
gpart: Device busy

:
:

:
root@beastie:~ # gpart destroy -F ada0
gpart: Device busy


There might still be situations where 'sysctl kern.geom.debugflags=16' 
helps, but I never needed it in the last years (since 7.x I guess).
Are you sure p2 (-i2) of ada0, most likely home for a ufs filesystem, 
isn't mounted anymore? Was it a mountpoint inside a jail?  Stopping the 
jail might leave network related active sockets blocking the filesystem 
(reboot without starting the jail before deleteing the partition should 
work in that case).


-harry
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: What was the intention about "jail -e" in the first place ?

[David Marec]

Le 16/04/2020 à 16:22, Eugene Grosbein a écrit :

>> But,how was "jail -e" intending to be used, actually ?
>
> "jail -e" mode is used by periodic/weekly/340.noid script to 
differentiate parts of mounted file trees
> belonging to the host and to the configured full-blown jails, no 
matter started or not.

>
> This is documentation ambiguity as "jail -e" was not intended to take 
jail name as additional argument.


Oh, I had a deeper look at the "-e" section of the man page where this 
statement is clear enough.Thanks.


> Do you have any real use case this addition?

No, not a case in the real world. I was just playing around with jails 
on a server and tried to get the ip4 field of a specific one to make 
sure it was set to 'inherit'.




P.-S.:

However, I had time to write a short patch (attached to the email) to 
make it work with a jail list as arguments.

( and that do not change the header file anymore )

This one also add a dedicated line for the "-e" command and fix few 
typos in the usage() output.


// To improve the lookup in the nested loop, I first called "TAILQ 
REMOVE" on jailnames when found but this would produce errors if the 
user add more than once the same name in the list .//




Regards,

--
David Marec
https://diablotins.lapinbilly.eu/doku.php?id=jails:zfs
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: What was the intention about "jail -e" in the first place ?

[Eugene Grosbein]

16.04.2020 16:31, David Marec wrote:

> The manpage and usage output doesn't match.
> 
> The manpage tells us that this option should be used alone on the command 
> line, as any other command will be discarded.
> 
> The usage ouput does not mention the "-r" flag but "cmr" (with a typo).
> 
> Both suggest that the user can request information about one single jail, or 
> all the jails using a wildcard or no other argument.
> 
> Well, looking at the code, it sounds that the only way to make it work is to 
> use it alone on the command line, and their is no way get information about a 
> single jail.
> 
> Attached is a short patch to get information about one single jail or all 
> jails (wildcards or empty).
> 
> 
> But,how was "jail -e" intending to be used, actually ?

"jail -e" mode is used by periodic/weekly/340.noid script to differentiate 
parts of mounted file trees
belonging to the host and to the configured full-blown jails, no matter started 
or not.

This is documentation ambiguity as "jail -e" was not intended to take jail name 
as additional argument.
Do you have any real use case this addition?

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

What was the intention about "jail -e" in the first place ?

[David Marec]

The manpage and usage output doesn't match.

The manpage tells us that this option should be used alone on the 
command line, as any other command will be discarded.


The usage ouput does not mention the "-r" flag but "cmr" (with a typo).

Both suggest that the user can request information about one single 
jail, or all the jails using a wildcard or no other argument.


Well, looking at the code, it sounds that the only way to make it work 
is to use it alone on the command line, and their is no way get 
information about a single jail.


Attached is a short patch to get information about one single jail or 
all jails (wildcards or empty).



But,how was "jail -e" intending to be used, actually ?


--
David Marec
https://lapinbilly.eu/
Index: jail.c
===
--- jail.c	(revision 359745)
+++ jail.c	(working copy)
@@ -282,16 +282,12 @@
 	? NULL : "false");
 			}
 		}
-	} else if (op == JF_STOP || op == JF_SHOW) {
-		/* Just print list of all configured non-wildcard jails */
-		if (op == JF_SHOW) {
-			load_config();
-			show_jails();
-			exit(0);
-		}
+	} else if (op == JF_STOP ) {
+
 		/* Jail remove, perhaps using the config file */
 		if (!docf || argc == 0)
 			usage();
+
 		if (!Rflag)
 			for (i = 0; i < argc; i++)
 if (strchr(argv[i], '='))
@@ -300,6 +296,17 @@
 		 (!strcmp(cfname, "-") || stat(cfname, ) == 0)))
 			load_config();
 		note_remove = docf || argc > 1 || wild_jail_name(argv[0]);
+	} else if(op == JF_SHOW) {
+		/* Just print list of all configured non-wildcard jails */
+		if (op == JF_SHOW && (argc==0|| wild_jail_name(argv[0]))) {
+			load_config();
+			show_jails();
+			exit(0);
+		}
+
+		if(argc>1)
+			usage();
+		load_config();
 	} else if (argc > 1 || (argc == 1 && strchr(argv[0], '='))) {
 		/* Single jail specified on the command line */
 		if (Rflag)
@@ -474,6 +481,9 @@
 			dep_done(j, 0);
 			break;
 
+		case JF_SHOW:
+			print_jail(stdout, j, 0, 0);
+			break;
 		case JF_STOP:
 		case JF_RESTART:
 			if (j->comparam == NULL) {
@@ -1040,8 +1050,8 @@
 	(void)fprintf(stderr,
 	"usage: jail [-dhilqv] [-J jid_file] [-u username] [-U username]\n"
 	"-[cmr] param=value ... [command=command ...]\n"
-	"   jail [-dqv] [-f file] [-e separator] -[cmr] [jail]\n"
-	"   jail [-qv] [-f file] -[rR] ['*' | jail ...]\n"
+	"   jail [-dqv] [-f file] [-e separator] [-cmr] [jail]\n"
+	"   jail [-qv] [-f file] [-rR] ['*' | jail ...]\n"
 	"   jail [-dhilqv] [-J jid_file] [-u username] [-U username]\n"
 	"[-n jailname] [-s securelevel]\n"
 	"path hostname [ip[,...]] command ...\n");
Index: jailp.h
===
--- jailp.h	(revision 359745)
+++ jailp.h	(working copy)
@@ -69,7 +69,7 @@
 #define JF_FROM_RUNQ	0x0800	/* Has already been on the run queue */
 #define JF_SHOW		0x1000	/* -e Exhibit list of configured jails */
 
-#define JF_OP_MASK		(JF_START | JF_SET | JF_STOP)
+#define JF_OP_MASK		(JF_START | JF_SET | JF_STOP | JF_SHOW)
 #define JF_RESTART		(JF_START | JF_STOP)
 #define JF_START_SET		(JF_START | JF_SET)
 #define JF_SET_RESTART		(JF_SET | JF_STOP)
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: CFT: if_bridge performance improvements

[Kristof Provost]

On 16 Apr 2020, at 10:36, Peter Blok wrote:
Another issue I found with pf was with "set skip on bridge”. It 
doesn’t work on the interface group, unless a bridge exists prior to 
enabling pf. Makes sense, but I didn’t think of it. Other rules work 
fine with interface groups.



I am aware of this problem and have unfinished work to fix it.

No promises about a timeline though.

Best regards,
Kristof
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: CFT: if_bridge performance improvements

[peter . blok]

Hi Mark/Kristof,

I have been using ng_bridge for more than a year. It was very stable and it 
allowed to have members with different MTU. My jails were using jng to setup 
the bridge and I changed iohyve to use ng_bridge.

But I recently switched to if_bridge. I needed to have pf work on a member 
interface, which wasn’t easy with ng_bridge. It was not easy to make it work 
due to two members (VLAN) coming frome the same trunk.The behavior was erratic.

I have a trusted VLAN bridged to an untrusted physical and Wifi network. All 
members are on the same IP segment, but with pf I can make sure that the 
untrusted IOT devices are only able to go outside towards the internet. The 
untrusted devices can’t create connections to the trusted devices, but the 
trusted devices can create connections to the untrusted devices.

Another issue I found with pf was with "set skip on bridge”. It doesn’t work on 
the interface group, unless a bridge exists prior to enabling pf. Makes sense, 
but I didn’t think of it. Other rules work fine with interface groups.

My jails and bhyve now runs fine with if_bridge, which is easier to setup and I 
don’t need any changes in iohyve.

Peter 

> On 16 Apr 2020, at 09:44, Kristof Provost  wrote:
> 
> Hi Mark,
> 
> I wouldn’t expect these changes to make a difference in the performance of 
> this setup.
> My work mostly affects setups with multi-core systems that see a lot of 
> traffic. Even before these changes I’d expect the if_bridge code to saturate 
> a wifi link easily.
> 
> I also wouldn’t expect ng_bridge vs. if_bridge to make a significant 
> difference in wifi features.
> 
> Best regards,
> Kristof
> 
> On 16 Apr 2020, at 3:56, Mark Saad wrote:
> 
>> Kristof
>> Up until a month ago I ran a set of FreeBSD based ap in my house and even 
>> long ago at work . They were Pc engines apu ‘s or Alix’s with one em/igb nic 
>> and one ath nic in a bridge .  They worked well for a long time however the 
>> need for more robust wifi setup caused me to swap them  out with cots aps 
>> from tp-link .  The major issues were the lack of WiFi features and 
>> standards that work oob on Linux based aps .
>> 
>> So I always wanted to experiment with ng_bridge vs if_bridge for the same 
>> task . But I never got around to it . Do you have any insight into using one 
>> vs the other . Imho if_bridge is easier to setup and get working .
>> 
>> 
>> ---
>> Mark Saad | nones...@longcount.org
>> 
>>> On Apr 15, 2020, at 1:37 PM, Kristof Provost  wrote:
>>> 
>>> On 15 Apr 2020, at 19:16, Mark Saad wrote:
 All
 Should this improve wifi to wired bridges in some way ? Has this been 
 tested ?
 
>>> What sort of setup do you have to bridge wired and wireless? Is the FreeBSD 
>>> box also a wifi AP?
>>> 
>>> I’ve not done any tests involving wifi.
>>> 
>>> Best regards,
>>> Kristof
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: CFT: if_bridge performance improvements

[Kristof Provost]

Hi Mark,

I wouldn’t expect these changes to make a difference in the 
performance of this setup.
My work mostly affects setups with multi-core systems that see a lot of 
traffic. Even before these changes I’d expect the if_bridge code to 
saturate a wifi link easily.


I also wouldn’t expect ng_bridge vs. if_bridge to make a significant 
difference in wifi features.


Best regards,
Kristof

On 16 Apr 2020, at 3:56, Mark Saad wrote:


Kristof
  Up until a month ago I ran a set of FreeBSD based ap in my house and 
even long ago at work . They were Pc engines apu ‘s or Alix’s with 
one em/igb nic and one ath nic in a bridge .  They worked well for a 
long time however the need for more robust wifi setup caused me to 
swap them  out with cots aps from tp-link .  The major issues were the 
lack of WiFi features and standards that work oob on Linux based aps .


So I always wanted to experiment with ng_bridge vs if_bridge for the 
same task . But I never got around to it . Do you have any insight 
into using one vs the other . Imho if_bridge is easier to setup and 
get working .



---
Mark Saad | nones...@longcount.org


On Apr 15, 2020, at 1:37 PM, Kristof Provost  wrote:

On 15 Apr 2020, at 19:16, Mark Saad wrote:

All
  Should this improve wifi to wired bridges in some way ? Has this 
been tested ?


What sort of setup do you have to bridge wired and wireless? Is the 
FreeBSD box also a wifi AP?


I’ve not done any tests involving wifi.

Best regards,
Kristof

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: CFT: if_bridge performance improvements

[Kristof Provost]

On 16 Apr 2020, at 8:34, Pavel Timofeev wrote:

Hi!
Thank you for your work!
Do you know if epair suffers from the same issue as tap?

I’ve not tested it, but I believe that epair scales significantly 
better than tap.
It has a per-cpu mutex (or more accurately, a mutex in each of its 
per-cpu structures), so I’d expect much better throughput from epair 
than you’d see from tap.


Best regards,
Kristof
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: CFT: if_bridge performance improvements

[Pavel Timofeev]

вт, 14 апр. 2020 г., 12:51 Kristof Provost :

> Hi,
>
> Thanks to support from The FreeBSD Foundation I’ve been able to work
> on improving the throughput of if_bridge.
> It changes the (data path) locking to use the NET_EPOCH infrastructure.
> Benchmarking shows substantial improvements (x5 in test setups).
>
> This work is ready for wider testing now.
>
> It’s under review here: https://reviews.freebsd.org/D24250
>
> Patch for CURRENT: https://reviews.freebsd.org/D24250?download=true
> Patches for stable/12:
> https://people.freebsd.org/~kp/if_bridge/stable_12/
>
> I’m not currently aware of any panics or issues resulting from these
> patches.
>
> Do note that if you run a Bhyve + tap on bridges setup the tap code
> suffers from a similar bottleneck and you will likely not see major
> improvements in single VM to host throughput. I would expect, but have
> not tested, improvements in overall throughput (i.e. when multiple VMs
> send traffic at the same time).
>
> Best regards,
> Kristof
>

Hi!
Thank you for your work!
Do you know if epair suffers from the same issue as tap?

>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"