> On May 9, 2025, at 1:57 PM, Warner Losh <[email protected]> wrote:
>
> On Tue, May 6, 2025 at 11:31 AM Charles Sprickman <[email protected]> wrote:
>>
>>
>>> On May 6, 2025, at 1:04 AM, Colin Percival <[email protected]> wrote:
>>>
>>> On 5/5/25 21:54, Charles Sprickman wrote:
>>>> Anyhow, we're trying purchase a few servers (likely this Supermicro:
>>>> https://www.supermicro.com/en/products/system/UP/1U/SYS-111E-WR) and since
>>>> manufacturers don't seem to validate for FreeBSD these days, I was hoping
>>>> for some community input.
>>>> Last time I looked into this NVMe (specifically the "U.2" format for
>>>> servers that includes hot swap) was supported in FreeBSD but hot swap was
>>>> kind of iffy. I don't think too many FreeBSD users/developers at that time
>>>> had the hardware, most people seemed to just be using an on-board NVMe
>>>> drive on desktop or laptop systems, and there's not any real call for hot
>>>> swap in that segment.
>>>> I have a thread on the forums linked below, but really what I'm looking to
>>>> find is some larger orgs that are running NVMe drives and can confirm hot
>>>> swap works as expected, if there's any gotchas to be on the lookout for,
>>>> etc. Trying not to overcomplicate it, but we don't want to be stuck with a
>>>> bunch of servers that might be less stable than our current SAS/SATA
>>>> servers. We exclusively use ZFS, if that's useful info.
>>>
>>> I can't speak to *physical* hotplug, but I've put a lot of energy over the
>>> past few months into making sure that nvme "hotplug" is 100% functional in
>>> Amazon EC2. If you run into problems it's probably going to be due to
>>> broken firmware,
>> [...]
>>
>> Can you explain a bit here if you've got a few minutes?
>>
>> I'm used to the old world concept of IDE/SCSI/SAS controllers (esp. RAID
>> controllers) where you've got a lot of processing going on in the card to go
>> from the drive interface to whatever kind of bus the controller sits on in
>> the PC. But in the case of NVMe, what even is a "controller"? Is it not just
>> a card that's combining/arbitrating PCI-e lanes from a bunch of drives to
>> the PCI-e bus? Where would be the firmware and what would be the
>> compatibility issues? I admit I don't use any NVMe stuff anywhere, so I'm
>> not very familiar with it, and especially not with this setup where we have
>> a "controller" involved...
>
> I've had access to a couple of hotplug chassis / motherboards. For
> x86, they've just worked for me. While the controller is bundled onto
> the nvme card, the PCIe bus has protocols to cope with a card being
> removed. FreeBSD has support for the hotplug standards around this.
Were any of these Supermicro or are you strictly working with tier-1 vendors
(ie: Dell, HPE, etc.)?
Followups on this thread regarding Supermicro are not promising so far (3
people with issues requiring a reboot to re-plug, all Supermicro), but that's
what we've been buying for decades...
Thanks,
Charles
> I've had some dodgy firmware on arm64 systems fail, though. I've not
> had the time to puzzle ou why....
>
> Warner
>
>
>> Thanks,
>>
>> Charles
>>
>> [...]
>>> so unless anyone has experience with that specific server
>>> I think the best answer you can get is "it should work but you need to test
>>> it and find out".
>>>
>>> --
>>> Colin Percival
>>> FreeBSD Release Engineering Lead & EC2 platform maintainer
>>> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
>>>
>>>
>>
>>
