Re: [zfs-discuss] NTFS on NFS and iSCSI always generates small IO's
On Thu, March 10, 2011 13:04, wessels wrote: And make sure you align your NTFS partition regardless off the underlying storage. Windows 2003 and before DONT do this by default, 7 and 2008 choose a default offset off 1Mb. But better check it in advance with diskpart. Lastly format your NTFS filesystem with an appropriate cluster size. Which needs to be determined on a case by case basis. Large video file are a different workload than an exchange server. Or have separate VMDKs for your OS and data (either as a separate drive letter or a volume mount point). ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] NTFS on NFS and iSCSI always generates small IO's
Hi All, I've run into a problem with my OpenSolaris system and NTFS, I can't seem to make sense of it. The server running the virtual machines is Ubuntu Server 10.04 running KVM. Storage is presented via NFS over Infiniband. ZFS is not running compression or dedup. Zil is also currently disabled because it was causing terrible NFS performance. ESXi also displayed the same behaviour when running Windows VM's. OpenSolaris box is running - OS b147 Xeon 5520 24GB DDR3 24x 1.5TB 7200rpm sata drives 24 bay expander connected to an LSI raid card via 4 lane SAS cable 2x 64GB Intel X25-M L2ARC 2x 32GB Intel X25-M SLOG Host - 4KB reads writes are full speed. Tested using DD. Also tested using Samba writing to the NFS share (Bare metal PC - samba - NFS) and it was writing as fast as the server could dish out the data (50MB/s). Test VM 1 - OS is Ubuntu Server 10.10 on ext4. Reads writes are full speed at 4kb, tested using DD. Sequential read/write seems fine. Tested using Samba also, same results as above (50MB/s). Test VM 2- OS is Windows Server 2008 on NTFS 4kb cluster size. 4KB reads writes are 30MB/s, tested using the ATTO disk benchmark. Sequential reads and writes max out at 30MB/s also and generate a lot of IO on the storage box. IOStat on the storage box shows 3k+ IOPS and only 30MB/s throughput. Also tested a bare metal system running iSCSI over gigabit. NTFS (default 4kb cluster size) directly on the ZFS block volume. Results are the same as Test VM 2, lots of small IO (3k+ IOPS) and small throughput 30MB/s I also did the same test using SRP. 4kb reads and writes have the same 30MB/s. Any IO above or equal to 128kb seems to pick up speed drastically, easily 700MB/s. I have a feeling it's to do with ZFS's recordsize property but haven't been able to find any solid testing done with NTFS. I'm going to do some testing using smaller record sizes tonight to see if that helps the issue. At the moment I'm surviving on cache and am quickly running out of capacity. Can anyone suggest any further tests or have any idea about what's going on? Thanks -Matt ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] NTFS on NFS and iSCSI always generates small IO's
On Thu, Mar 10, 2011 at 12:15 AM, Matthew Anderson matth...@ihostsolutions.com.au wrote: I have a feeling it's to do with ZFS's recordsize property but haven't been able to find any solid testing done with NTFS. I'm going to do some testing using smaller record sizes tonight to see if that helps the issue. At the moment I'm surviving on cache and am quickly running out of capacity. Can anyone suggest any further tests or have any idea about what's going on? The default blocksize for a zfs volume is 8k, so 4k writes will probably require a read as well. You can try creating a new volume with volblocksize set to 4k and see if that helps. The value can't be changed once set, so you'll have to make a new dataset. Make sure the wcd property is set to false for the volume in stmfadm in order to enable the write cache. It shouldn't make a huge difference with the zil disabled, but it certainly won't hurt. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] NTFS on NFS and iSCSI always generates small IO's
On Mar 10, 2011, at 12:15 AM, Matthew Anderson wrote: Hi All, I've run into a problem with my OpenSolaris system and NTFS, I can't seem to make sense of it. The server running the virtual machines is Ubuntu Server 10.04 running KVM. Storage is presented via NFS over Infiniband. ZFS is not running compression or dedup. Zil is also currently disabled because it was causing terrible NFS performance. Default recordsize for NFS is 128K. For the VM case, you will want to match the block size of the clients. However, once the file (on the NFS server) is created with 128K records, it will remain at 128K forever. So you will need to create a new VM store after the recordsize is tuned. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] NTFS on NFS and iSCSI always generates small IO's
On Thu, Mar 10, 2011 at 9:45 AM, Richard Elling richard.ell...@gmail.com wrote: Default recordsize for NFS is 128K. For the VM case, you will want to match the block size of the clients. However, once the file (on the NFS server) is created with 128K records, it will remain at 128K forever. So you will need to create a new VM store after the recordsize is tuned. You can change the recordsize and copy the vmdk files on the nfs server, which will re-write them with a smaller recordsize. -B -- Brandon High : bh...@freaks.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] NTFS on NFS and iSCSI always generates small IO's
And make sure you align your NTFS partition regardless off the underlying storage. Windows 2003 and before DONT do this by default, 7 and 2008 choose a default offset off 1Mb. But better check it in advance with diskpart. Lastly format your NTFS filesystem with an appropriate cluster size. Which needs to be determined on a case by case basis. Large video file are a different workload than an exchange server. I've got a set of thin provisioned vmdk's of varying size and cluster size. Which I can clone for new machines. This saves me the diskpart step, which you cannot run when installing windows. On Thu, Mar 10, 2011 at 6:45 PM, Richard Elling richard.ell...@gmail.com wrote: On Mar 10, 2011, at 12:15 AM, Matthew Anderson wrote: Hi All, I've run into a problem with my OpenSolaris system and NTFS, I can't seem to make sense of it. The server running the virtual machines is Ubuntu Server 10.04 running KVM. Storage is presented via NFS over Infiniband. ZFS is not running compression or dedup. Zil is also currently disabled because it was causing terrible NFS performance. Default recordsize for NFS is 128K. For the VM case, you will want to match the block size of the clients. However, once the file (on the NFS server) is created with 128K records, it will remain at 128K forever. So you will need to create a new VM store after the recordsize is tuned. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
