Okay. So: 1) Don't use EXT4 with LVM/RAID, it performs terribly with QCOW2. Use XFS. 2) I didn't do anything to my NFS mount options and they came out fine:
10.100.255.3:/storage/primary3 on /mnt/0ab13de9-2310-334c-b438-94dfb0b8ec84 type nfs4 (rw,relatime,sync,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,noac,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.100.255.2,local_lock=none,addr=10.100.255.3) 3) ZFS works relatively well, but if you are using it with SSD's you *must* set the alignment to match the SSD alignment, and *must* set the ZFS block size to match the SSD internal block size, else performance is terrible. For performance, use LVM/XFS/RAID. *EXCEPT* it is really hard to make that perform on SSD's too, if you have a block size and alignment mismatch performance will be *terrible* with LVM/XFS/RAID. 4) Hardware RAID with a battery backup tends to result in much faster writes *if writing to rotating storage* and only if using Linux LVM/XFS. ZFS does its own redundancy better, so don't use BBU with ZFS. If writing to SSD's, you will get better performance with the Linux MD RAID or ZFS, but note you must be *very* careful about block sizes and alignment. Don't use a BBU hardware raid with SSD's, performance will be terrible compared to Linux MD RAID or ZFS for a number of reasons. 5) Needless to say the ZFS NFS shares work fine, *if* you've done your homework and set them up well with proper alignment and block size for your hardware. However, for rotational storage the LVM/XFS/RAID will be faster, especially with the hardware raid and BBU. My own CloudStack implementation has two storage servers that use LVM/XFS/RAID for storage and one storage server that uses ZFS for storage. The ZFS server has two 12-disk RAID groups, one made up of SSD's for a database, the other made up of large rotational storage drives. It also has a NVMe card that is used for log and cache for the rotational storage. I spent a *lot* of time trying to get the SSD's to perform under RAID/LVM/XFS and just couldn't get everything to agree on alignment. That was when I said foo on that and put ZFS on there, and since I was using ZFS for one RAID group, it made sense to use it on the other too. I'm using RAID10 on the SSD RAID group, and RAIDZ2 on the rotational storage (which is there for bulk storage where performance isn't a priority). Storage is not a limitation on my cloud, especially since I have four other storage servers that I can throw at it if necessary. RAM and CPU are, so that's my next task -- get more compute servers into the Cloudstack cluster. > On Oct 19, 2017, at 22:16, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com> wrote: > > Hello, friends. > > I'm planning to deploy new cluster with KVM and shared NFS storage soon. > Right now I already have such deploys that operate fine. Currently, I'm > trying to compare different storage options for new cluster between > - LVM+EXT4+HWRAID+BBU, > - LVM+EXT4+MDADM and > - ZFS. > > HW setup is: > - 5 x Micron 5100PRO 1.9TB > - LSI9300-8i HBA or LSI9266-8i+BBU > - Cisco UCS server w/ 24x2.5" / 2xE5-2650 / 64GB RAM > > I have got competitive performance results between them locally already, > but now I need to test over NFS. I'm pretty sure that first two options > will operate nice with ACS default NFS mount args (because I already have > such cases in prod), but ZFS is quite smart thing, so I started to > investigate how to change NFS client mount options and unfortunately > haven't succeed defining the proper place where cloudstack agent determines > how to mount share and what args to use. I read a lot of ZFS-related > articles and people write rsize/wsize affect quite much, so I wonder how to > instruct cloudstack agent to use specific rsize/wsize args to mount primary > storage. > > Also, I haven't found in ACS archives mentions about ZFS NFS share, so > might be it's a bad case for ACS because of Qcow image format?, but I think > it could be a good one so want to test personally. > > Any suggestions are welcome. > > -- > With best regards, Ivan Kudryavtsev > Bitworks Software, Ltd. > Cell: +7-923-414-1515 > WWW: http://bitworks.software/ <http://bw-sw.com/>
