On Tuesday, 28 September 2021 12:38:42 BST Rich Freeman wrote: > You keep mentioning USB3, but I think the main factor here is that the > external drive is probably a spinning hard drive (I don't think you > explicitly mentioned this but it seems likely esp with the volume of > data). That math works out to 78MB/s. Hard drive transfer speeds > depend on the drive itself and especially whether there is more than > one IO task to be performed, so I can't be entirely sure, but I'm > guessing that the USB3 interface itself is having almost no adverse > impact on the transfer rate.
I'm sure you're right Rich, and yes, this is 2.5" drive. I'm seeing about 110MB/s reading from USB feeding into zstd. > The main thing to avoid is doing other sustained read/writes from the > drive at the same time. Quite so. > It looks like you ended up doing the bulk of the compression on an > SSD, and obviously those don't care nearly as much about IOPS. Yes, input from USB and output to SSD. > I've been playing around with lizardfs for bulk storage and found that > USB3 hard drives actually work very well, as long as you're mindful > about what physical ports are on what USB hosts and so on. A USB3 > host can basically handle two hard drives with no loss of performance. > I'm not dealing with a ton of IO though so I can probably stack more > drives with pretty minimal impact unless there is a rebuild (in which > case the gigabit ethernet is probably still the larger bottleneck). > Even a Raspberry Pi 4 has two USB3 hosts, which means you could stack > 4 hard drives on one and get basically the same performance as SATA. > When you couple that with the tendency of manufacturers to charge less > for USB3 drives than SATA drives of the same performance it just > becomes a much simpler solution than messing with HBAs and so on and > limiting yourself to hardware that can actually work with an HBA. -- Regards, Peter.
