Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi, Peter Merchant wrote: > > I saw this the other day, and was wondering about it. > > https://www.tomshardware.com/raspberry-pi/add-on-board-lets-you-use-four-nvme-ssds-at-once-with-raspberry-pi-5 Thanks Peter, that's interesting. I wanted two NVMes for a local RAID 1 mirror. Four seems a bit overkill given the Pi's PCI 2.0 bottleneck. Apparently, a simple configuration change lets it run at PCI 3.0 speeds, which are roughly double in theory, but that's officially unsupported and I can't see the data-corruption risk is worth it. Plus this board uses a ‘ASM1184e PCIe Gen 2’ chip so the limit would still apply even if the Pi's limit was raised. https://en.wikipedia.org/wiki/PCI_Express#Comparison_table Rhys wrote: > While on the topic of drives on Pi's; I did read this post by Jeff > Geerling the other day. Might be something worth looking into if you > won't benefit from the additional bandwidth and features of NVMe. > https://www.jeffgeerling.com/blog/2024/radxas-sata-hat-makes-compact-pi-5-nas True, I won't benefit from the NVMes. I was going for those instead of my previous SATA SSD purchases with the idea they'd be more useful in other devices in the future. But you've made me consider if I should just go for capacity instead. > I wonder if I could get SATA and NVMe running as > some kind of cache, with the SATA drives being used for mass storage? There's https://docs.kernel.org/admin-guide/bcache.html SSD in front of RAID array: https://www.jeffgeerling.com/blog/2021/htgwa-use-bcache-ssd-caching-on-raspberry-pi > I am also looking into some of the non-pi SBCs like the Pine64 and > Radxa's for a project Yes, the non-Pi boards interest me, but given I'd like this machine to be stable and not lot of trouble, I appreciate the large effort invested in the Pi 5 and Raspbian. -- Cheers, Ralph. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
While on the topic of drives on Pi's; I did read this post by Jeff Geerling the other day. Might be something worth looking into if you won't benefit from the additional bandwidth and features of NVMe. https://www.jeffgeerling.com/blog/2024/radxas-sata-hat-makes-compact-pi-5-nas Seems somewhat interesting if you were, say, to build a small NAS. I'm not too sure how well ZFS or whatever filesystem you decide to use would fair performance-wise when used in RAID but seems to be worth a deeper dive into. I wonder if I could get SATA and NVMe running as some kind of cache, with the SATA drives being used for mass storage? I have seen people make some fairly interesting boards for the Pi `Compute Modules`, which might be worth looking into too at some point, but I am also looking into some of the non-pi SBCs like the Pine64 and Radxa's for a project, so who knows where that will go. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
I saw this the other day, and was wondering about it. Peter https://www.tomshardware.com/raspberry-pi/add-on-board-lets-you-use-four-nvme-ssds-at-once-with-raspberry-pi-5 On 22/05/2024 11:12, Ralph Corderoy wrote: Hi Bob, You might want to look at the PIMORONI site, single and dual NVMe adapters up to 2280 in length. Thanks, I found which does allow two up to the 2280's 80 mm. https://shop.pimoroni.com/products/nvme-base-duo-for-raspberry-pi-5 And thanks for the pointer to Argon cases. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi Rhys, > As for NVMe HAT's, personally, I use the Geekwork x1001 on mine > https://geekworm.com/products/x1001 Thanks, I did not know of the Shenzhen based Geekworm and see they have a dual 2280 board too. This one mentions a firmware update. ‘The X1004 is designed specifically for data storage purposes and currently support NVMe boot with the latest firmware EEPROM 2024/05/17 version.’ — https://geekworm.com/products/x1004 And they have a case which can accomodate it. https://geekworm.com/products/p579 -- Cheers, Ralph. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi Ralph, Finally managed to find the mailing list join details! As for NVMe HAT's, personally, I use the Geekwork x1001[1] on mine with a Samsung 990 EVO (Overkill, I know, but it was free from work) When it comes to NVMe over TCP, RedHat does have some fairly decent documentation[2] on how it might work, you'd just need to find similar tooling for Debian if you don't want to use an RPM distro. I'm not entirely sure if the Pi has any kind of RDMA support, so I might have to tinker with that a little this week, but theoretically it should? And if it does, that would hopefully improve performance dramatically. I do wonder if I could somehow get InfiniBand running on a Pi, albeit I expect performance to be poor, but that's a story for another day. <https://geekworm.com/products/x1001> [1] https://geekworm.com/products/x1001 [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_storage_devices/configuring-nvme-over-fabrics-using-nvme-tcp_managing-storage-devices From: dorset on behalf of Ralph Corderoy Sent: 22 May 2024 11:12:18 To: Dorset Linux User Group Subject: Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs. Hi Bob, > You might want to look at the PIMORONI site, single and dual NVMe > adapters up to 2280 in length. Thanks, I found which does allow two up to the 2280's 80 mm. https://shop.pimoroni.com/products/nvme-base-duo-for-raspberry-pi-5 And thanks for the pointer to Argon cases. -- Cheers, Ralph. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi Bob, > You might want to look at the PIMORONI site, single and dual NVMe > adapters up to 2280 in length. Thanks, I found which does allow two up to the 2280's 80 mm. https://shop.pimoroni.com/products/nvme-base-duo-for-raspberry-pi-5 And thanks for the pointer to Argon cases. -- Cheers, Ralph. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi Patrick, > That was an interesting read. Good. > > The Linux kernel has NVMe support since 5.0; package nvme-cli drives > > it. > > Linux certainly had at least some support for NVMe before version 5.0. > I think you must be referring to a subset of the NVMe functionality > being added in version 5.0. Yes, sorry, I should have written ‘NVMe over TCP/IP’. ‘What is needed on the Host side (hardware, firmware, software, drivers, etc.) to support NVMe over TCP? ‘NVMe/TCP software does not require any special hardware/firmware to operate, although different classes of CPU and network adapters can benefit from better performance. NVMe/TCP host and NVM subsystem software need to be installed in order to run NVMe/TCP. The software is available with Linux Kernel (v5.0) and SPDK (v.19.01), as well as commercial NVMe/TCP target devices.’ — https://nvmexpress.org/answering-your-questions-nvme-tcp-what-you-need-to-know-about-the-specification-webcast-qa/ -- Cheers, Ralph. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi, You might want to look at the PIMORONI site [1], single and dual NVMe adapters up to 2280 in length. You can boot from the single slot adapters but not the dual as the existing bootstrap code doesn't recognise the packet switcher/multiplexer although I understand this is being worked on. The PiHut [2] resell some of the PIMORONI adaptors as well as Argon cases with adapters for NVMe with Pi5 and Pi4. The latter via a USB bridge which is ugly. [1] https://shop.pimoroni.com/search?q=NVMe [2] https://thepihut.com/ -- Bob Dunlop -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
Re: [Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi Ralph, That was an interesting read. I wasn't up-to-date enough to know about NVMe over fabrics, nor that the Pi 5 can be attached to an NVMe SSD. On Tue, 21 May 2024 17:31:07 +0100, Ralph Corderoy wrote: > The Linux kernel has NVMe support since 5.0; package nvme-cli drives > it. Linux certainly had at least some support for NVMe before version 5.0. I think you must be referring to a subset of the NVMe functionality being added in version 5.0. The earliest Linux distribution I installed on NVMe storage was Kubuntu 16.04.1, using Ubuntu's 4.4.0-36-generic kernel and later versions. It had to be installed from a modified ISO image, since the standard installation media wouldn't detect the NVMe SSD. Wikipedia says NVMe support was first added to mainline Linux in version 3.3. Best wishes, Patrick -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
[Dorset] Pi 5's PCIe connector for NVMe SSDs.
Hi, I wrote a private email to a friend. The content may be of interest to some on the list, so here it is. Do you know much about NVMe over PCIe for SSDs? I don't, only having bought SATA SSDs in the past. The Pi 5 has a ×1 PCIe connector, meaning one channel of two twisted pairs to allow a full-duplex connection. A ×4 PCIe has four of them allowing data to be sent in parallel, 4 b at a time. Raspberry Pi have recently released their PCIe HAT+ for plugging in an M.2 NVMe SSD, though only in one of two short form factors. https://www.raspberrypi.com/news/m-2-hat-on-sale-now-for-12/ That's handy. For $12 one can plug in large reliable storage and boot from it. Crucial, really Micron, don't make M.2 SSDs that short, but Samsung do so I'll probably plump for them as a second preference. But it's only one drive. What would be really nice is two in a RAID 1 mirror. The two peers sat at either end of the PCIe can negotiate the ×푛 channels downwards. This is how a ×4 can plug into a ×16. Some controllers can also support ‘bifurcation’, splitting a ×16, say, into a ×8 and two ×4 so a passive adapter card can plug into the ×16 and provide three connectors. This also needs BIOS or OS support. (Bifurcation seems the wrong word but perhaps it's split by a multiple of two each time.) (Sorry if you know all this. By putting it in writing, I'm having to think it through for myself.) Anyway, the Pi 5's ×1 can't be split. So a passive adapter, one which is effectively just wiring, wouldn't work. My slight understanding is an active adapter, meaning it has a PCIe controller, is required to bring forth a ×16 or whatever by intelligently multiplex. Obviously, it's all squeezed down a ×1 in the end, but if performance was key then the Pi 5 wouldn't be used. It's just a compact way to get considerable RAID 1 storage to hand. Ah, I see a reference to a ‘PCI switch’ rather than controller to multiplex. That's led me to https://pineboards.io/products/hatdrive-dual-2xnvme-2230-2242-gen-2-for-raspberry-pi-5 which uses an ASMedia device. So such things do exist. €40 seems a little high, but they're low volume. NVMe has other standardised transports, e.g. TCP/IP with TLS. This suggests it's just a ‘simple’ matter of software to have a RAID 1 with one M.2 drive dangling off the local ×1 PCIe and the other over Ethernet. Ethernet can stretch from LAN to WAN: a second local Pi 5 on the back of the TV running LibreElec, or an off-site non-Pi. The Linux kernel has NVMe support since 5.0; package nvme-cli drives it. There's also the BSD-licensed SPDK which is a user-mode C implementation of NMVe along with a network stack; to avoid the kernel overhead. https://spdk.io/doc/about.html. Which makes me wonder what the minimal hardware is to deploy either and drive local NVMe storage for network access. Not corporate-level data-centre SAN but home, hobbyist, small-office level. Easiest would be some SoC, ARM or RISC-V, running Linux. SPDK on a non-Linux SoC with PCIe would seem possible. https://spdk.io/doc/porting.html -- Cheers, Ralph. -- Next meeting: Online, Jitsi, Tuesday, 2024-06-04 20:00 Check to whom you are replying Meetings, mailing list, IRC, ... http://dorset.lug.org.uk New thread, don't hijack: mailto:dorset@mailman.lug.org.uk
