On Mon, 19 Jun 2017, Mayuresh wrote:
Just curious. How does iscsi compare with NFS? Guess even NFS has a
notion of block size, that would help optimize io.
Sorry for butting in, but I'd point out that NFS is file-based and layers
on top of an existing filesystem. So, the block size of the underlying
file system is going to determine the block size. There is also the
consideration of the network parameters such as send and receive buffer
sizes and several others that matter quite a bit (depending on the
layer-4 protocol in use and the version of NFS).
iSCSI only provides block devices, it can't do file-based I/O natively
without a filesystem on top of it. My experience with iSCSI has overall
been quite poor. I once did a long whitepaper on iSCSI vs AoE. Being a big
fan of SCSI (and not a huge fan of ATA) I was hoping & expecting iSCSI was
going to be better than it turned out. However, the experience turned out
completely opposite. Not only did AoE stomp it in every performance test I
tried, it also scaled better, recovered from failures better, and so
forth. iSCSI also has a million dials and settings for mostly useless crap
few are going to fiddle with. It feels like some kind of top-heavy
machination designed by some committee somewhere that never has to use
network block storage in-real-life.
I've also seen large scale iSCSI deployments be fraught with pain and
peril simply because network engineers can't be trusted to leave the VLANs
it runs on alone and can't be bothered to put it on discrete switches.
Of course AoE runs on top of layer-2 and iSCSI is a layer-5 protocol. The
extra layers underneath iSCSI make it routable, but destroy performance.
With AoE you don't have to tune TCP/IP (but it's non-routable).
I also remember hearing about HyperSCSI which is supposed to be hybrid
strategy that uses SCSI CDB's over Ethernet frames like AoE does. My guess
is, based on AoE's good-showing, that approach would rock if they got it
off the ground.
I guess I should also point out that iSCSI is widely supported across a
larger number of operating systems than AoE and has much more vendor
acceptance since AoE is seen as the domain of the CORAID (or whatever they
are called now) folks.
Anyhow, based on my bad experience, I wouldn't recommend iSCSI for anyone
unless they simply had no other choice. I have seen it be workable,
especially with dedicated hardware (Equallogic gear seems to work okay,
and it's got NetBSD bits in there too!), but overall, I'd run screaming
away.
iSCSI does give a block device to use with CGD, though. I bet it would
work fine with CGD, despite being kind of a poor idea in general (iSCSI
not CGD).
-Swift
Just my opinions here. If you use iSCSI and love it, YMMV, and more power
to you.
