Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Greg Mason writes: We're running into a performance problem with ZFS over NFS. When working with many small files (i.e. unpacking a tar file with source code), a Thor (over NFS) is about 4 times slower than our aging existing storage solution, which isn't exactly speedy to begin with (17 minutes versus 3 minutes). We took a rough stab in the dark, and started to examine whether or not it was the ZIL. Performing IO tests locally on the Thor shows no real IO problems, but running IO tests over NFS, specifically, with many smaller files we see a significant performance hit. Just to rule in or out the ZIL as a factor, we disabled it, and ran the test again. It completed in just under a minute, around 3 times faster than our existing storage. This was more like it! Are there any tunables for the ZIL to try to speed things up? Or would it be best to look into using a high-speed SSD for the log device? And, yes, I already know that turning off the ZIL is a Really Bad Idea. We do, however, need to provide our users with a certain level of performance, and what we've got with the ZIL on the pool is completely unacceptable. Thanks for any pointers you may have... I think you found out for the replies, this NFS issue is not related to ZFS nor a ZIL malfunction in any way. http://blogs.sun.com/roch/entry/nfs_and_zfs_a_fine NFS (particularly lightly threaded load) is much speeded up with any form of SSD|NVRAM storage and that's independant on the backing filesystem used (provided the Filesystem is safe). For ZFS the best way to acheive NFS performance for lightly threaded loads is to have a separate intent log in a low latency device such as in the 7000 line. -r -- Greg Mason Systems Administrator Michigan State University High Performance Computing Center ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Nicholas Lee writes: Another option to look at is: set zfs:zfs_nocacheflush=1 http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide Best option is to get a a fast ZIL log device. Depends on your pool as well. NFS+ZFS means zfs will wait for write completes before responding to a sync NFS write ops. If you have a RAIDZ array, writes will be slower than a RAID10 style pool. Nicholas, Raid-Z requires a more complexity in software however the total amount of I/O to disk is less than raid-10. So the net performance effect is often in favor of Raid-10 must not necessarely so. -r ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Eric D. Mudama writes: On Mon, Jan 19 at 23:14, Greg Mason wrote: So, what we're looking for is a way to improve performance, without disabling the ZIL, as it's my understanding that disabling the ZIL isn't exactly a safe thing to do. We're looking for the best way to improve performance, without sacrificing too much of the safety of the data. The current solution we are considering is disabling the cache flushing (as per a previous response in this thread), and adding one or two SSD log devices, as this is similar to the Sun storage appliances based on the Thor. Thoughts? In general principles, the evil tuning guide states that the ZIL should be able to handle 10 seconds of expected synchronous write workload. To me, this implies that it's improving burst behavior, but potentially at the expense of sustained throughput, like would be measured in benchmarking type runs. If you have a big JBOD array with say 8+ mirror vdevs on multiple controllers, in theory, each VDEV can commit from 60-80MB/s to disk. Unless you are attaching a separate ZIL device that can match the aggregate throughput of that pool, wouldn't it just be better to have the default behavior of the ZIL contents being inside the pool itself? The best practices guide states that the max ZIL device size should be roughly 50% of main system memory, because that's approximately the most data that can be in-flight at any given instant. For a target throughput of X MB/sec and given that ZFS pushes transaction groups every 5 seconds (and have 2 outstanding), we also expect the ZIL to not grow beyond X MB/sec * 10 sec. So to service 100MB/sec of synchronous writes, 1 GBytes of log device should be sufficient. But, no comments are made on the performance requirements of the ZIL device(s) relative to the main pool devices. Clicking around finds this entry: http://blogs.sun.com/perrin/entry/slog_blog_or_blogging_on ...which appears to indicate cases where a significant number of ZILs were required to match the bandwidth of just throwing them in the pool itself. Big topic. Some write requests are synchronous and some not, some start as non synchronous and end up being synced. For non-synchronous loads, ZFS does not commit data to the slog. The presence of the slog is transparent and won't hinder performance. For synchronous loads, the performance is normally governed by fewer threads commiting more modest amount of data; performance here is dominated by latency effect, not disk throughput and this is where a slog greatly helps (10X). Now you're right to point out that some workloads might end up as synchronous while still manageing large quantity of data. The Storage 7000 line was tweaked to handle some of those cases. So when commiting more say 10MB in a single operation, the first MB will go to the SSD but the rest will actually be send to the main storage pool. All these I/Os being issued concurrently. The latency response of a 1 MB to our SSD is expected to be similar to the response of regular disks. -r --eric -- Eric D. Mudama edmud...@mail.bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Eric D. Mudama writes: On Tue, Jan 20 at 21:35, Eric D. Mudama wrote: On Tue, Jan 20 at 9:04, Richard Elling wrote: Yes. And I think there are many more use cases which are not yet characterized. What we do know is that using an SSD for the separate ZIL log works very well for a large number of cases. It is not clear to me that the efforts to characterize a large number of cases is worthwhile, when we can simply throw an SSD at the problem and solve it. -- richard I think the issue is, like a previous poster discovered, there's not a lot of available data on exact performance changes of adding ZIL/L2ARC devices in a variety of workloads, so people wind up spending money and doing lots of trial and error, without clear expectations of whether their modifications are working or not. Sorry for that terrible last sentence, my brain is fried right now. I was trying to say that most people don't know what they're going to get out of an SSD or other ZIL/L2ARC device ahead of time, since it varies so much by workload, configuration, etc. and it's an expensive problem to solve through trial an error since these performance-improving devices are many times more expensive than the raw SAS/SATA devices in the main pool. I agree with you on the L2ARC front but not on the SSD for ZIL. We clearly expect 10X gain for lightly threaded workloads and that's a big satifyer because not everything happen with large amount of concurrency and some high value tasks do not. On the L2ARC the benefit are less direct because of the L1 ARC presence. The gains, if present will be of the similar nature with 8-10X gain to workloads that are lightly threaded and served from L2ARC vs disk. Note that it's possible to configurewhich (higher businessvalue) filesystems are allowed to install in the L2ARC. One dirty way to evaluate if the L2ARC will be effective in your environment is to consider if the last X GB of added memory had a positive impact on your performance metrics (does nailing down memory reduces performance ?). If so then on the graph of performance vs caching you are still on a positive slope and L2ARC is likely to help. When request you care most about are served from caches, or when something else saturates (e.g. total CPU) then it's time to stop. -r -- Eric D. Mudama edmud...@mail.bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Good observations, Eric, more below... Eric D. Mudama wrote: On Mon, Jan 19 at 23:14, Greg Mason wrote: So, what we're looking for is a way to improve performance, without disabling the ZIL, as it's my understanding that disabling the ZIL isn't exactly a safe thing to do. We're looking for the best way to improve performance, without sacrificing too much of the safety of the data. The current solution we are considering is disabling the cache flushing (as per a previous response in this thread), and adding one or two SSD log devices, as this is similar to the Sun storage appliances based on the Thor. Thoughts? In general principles, the evil tuning guide states that the ZIL should be able to handle 10 seconds of expected synchronous write workload. To me, this implies that it's improving burst behavior, but potentially at the expense of sustained throughput, like would be measured in benchmarking type runs. Yes. Workloads that tend to be latency sensitive also tend to be bursty. Or, perhaps that is just how it feels to a user. Similar observations are made in the GUI design business where user interactions are bursty, but latency sensitive. If you have a big JBOD array with say 8+ mirror vdevs on multiple controllers, in theory, each VDEV can commit from 60-80MB/s to disk. Unless you are attaching a separate ZIL device that can match the aggregate throughput of that pool, wouldn't it just be better to have the default behavior of the ZIL contents being inside the pool itself? The problem is that the ZIL writes must be committed to disk and magnetic disks rotate. So the time to commit to media is, on average, disregarding seeks, 1/2 the rotational period. This ranges from 2 ms (15k rpm) to 5.5 ms (5,400 rpm). If the workload is something like a tar -x of small files (source code) then a 4.17 ms (7,200 rpm) disk would limit my extraction to a maximum of 240 files/s. If these are 4kByte files, the bandwidth would peak at about 1 MByte/s. Upgrading to a 15k rpm disk would move the peak to about 500 files/s or 2.25 MBytes/s. Using a decent SSD would change this to 5000 files/s or 22.5 MBytes/s. The best practices guide states that the max ZIL device size should be roughly 50% of main system memory, because that's approximately the most data that can be in-flight at any given instant. There is a little bit of discussion about this point, because it really speaks to the ARC in general. Look for it to be clarified soon. Also note that this is much more of a problem for small memory machines. For a target throughput of X MB/sec and given that ZFS pushes transaction groups every 5 seconds (and have 2 outstanding), we also expect the ZIL to not grow beyond X MB/sec * 10 sec. So to service 100MB/sec of synchronous writes, 1 GBytes of log device should be sufficient. It is a little bit more complicated than that because if the size of the ZIL write is 32 kBYtes, then it will be written directly to the main pool, not the ZIL log. This is because if you have lots of large synchronous writes, then the system can become bandwidth limited rather than latency limited and the way to solve bandwidth problems is to reduce bandwidth demand. But, no comments are made on the performance requirements of the ZIL device(s) relative to the main pool devices. Clicking around finds this entry: http://blogs.sun.com/perrin/entry/slog_blog_or_blogging_on ...which appears to indicate cases where a significant number of ZILs were required to match the bandwidth of just throwing them in the pool itself. Yes. And I think there are many more use cases which are not yet characterized. What we do know is that using an SSD for the separate ZIL log works very well for a large number of cases. It is not clear to me that the efforts to characterize a large number of cases is worthwhile, when we can simply throw an SSD at the problem and solve it. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Any recommendations for an SSD to work with an X4500 server? Will the SSDs used in the 7000 series servers work with X4500s or X4540s? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
d...@yahoo.com said: Any recommendations for an SSD to work with an X4500 server? Will the SSDs used in the 7000 series servers work with X4500s or X4540s? The Sun System Handbook (sunsolve.sun.com) for the 7210 appliance (an X4540-based system) lists the logzilla device with this fine print: PN#371-4192 Solid State disk drives can only be installed in slots 3 and 11. Makes me wonder if they would work in our X4500 NFS server. Our ZFS pool is already deployed (Solaris-10), but we have four hot spares -- two of which could be given up in favor of a mirrored ZIL. An OS upgrade to S10U6 would give the separate-log functionality, if the drivers, etc. supported the actual SSD device. I doubt we'll go out and buy them before finding out if they'll actually work -- it would be a real shame if they didn't, though. Regards, Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
On Tue, Jan 20 at 9:04, Richard Elling wrote: Yes. And I think there are many more use cases which are not yet characterized. What we do know is that using an SSD for the separate ZIL log works very well for a large number of cases. It is not clear to me that the efforts to characterize a large number of cases is worthwhile, when we can simply throw an SSD at the problem and solve it. -- richard I think the issue is, like a previous poster discovered, there's not a lot of available data on exact performance changes of adding ZIL/L2ARC devices in a variety of workloads, so people wind up spending money and doing lots of trial and error, without clear expectations of whether their modifications are working or not. -- Eric D. Mudama edmud...@mail.bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
On Tue, Jan 20 at 21:35, Eric D. Mudama wrote: On Tue, Jan 20 at 9:04, Richard Elling wrote: Yes. And I think there are many more use cases which are not yet characterized. What we do know is that using an SSD for the separate ZIL log works very well for a large number of cases. It is not clear to me that the efforts to characterize a large number of cases is worthwhile, when we can simply throw an SSD at the problem and solve it. -- richard I think the issue is, like a previous poster discovered, there's not a lot of available data on exact performance changes of adding ZIL/L2ARC devices in a variety of workloads, so people wind up spending money and doing lots of trial and error, without clear expectations of whether their modifications are working or not. Sorry for that terrible last sentence, my brain is fried right now. I was trying to say that most people don't know what they're going to get out of an SSD or other ZIL/L2ARC device ahead of time, since it varies so much by workload, configuration, etc. and it's an expensive problem to solve through trial an error since these performance-improving devices are many times more expensive than the raw SAS/SATA devices in the main pool. -- Eric D. Mudama edmud...@mail.bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS over NFS, poor performance with many small files
We're running into a performance problem with ZFS over NFS. When working with many small files (i.e. unpacking a tar file with source code), a Thor (over NFS) is about 4 times slower than our aging existing storage solution, which isn't exactly speedy to begin with (17 minutes versus 3 minutes). We took a rough stab in the dark, and started to examine whether or not it was the ZIL. Performing IO tests locally on the Thor shows no real IO problems, but running IO tests over NFS, specifically, with many smaller files we see a significant performance hit. Just to rule in or out the ZIL as a factor, we disabled it, and ran the test again. It completed in just under a minute, around 3 times faster than our existing storage. This was more like it! Are there any tunables for the ZIL to try to speed things up? Or would it be best to look into using a high-speed SSD for the log device? And, yes, I already know that turning off the ZIL is a Really Bad Idea. We do, however, need to provide our users with a certain level of performance, and what we've got with the ZIL on the pool is completely unacceptable. Thanks for any pointers you may have... -- Greg Mason Systems Administrator Michigan State University High Performance Computing Center ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Another option to look at is: set zfs:zfs_nocacheflush=1 http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide Best option is to get a a fast ZIL log device. Depends on your pool as well. NFS+ZFS means zfs will wait for write completes before responding to a sync NFS write ops. If you have a RAIDZ array, writes will be slower than a RAID10 style pool. On Tue, Jan 20, 2009 at 11:08 AM, Greg Mason gma...@msu.edu wrote: We're running into a performance problem with ZFS over NFS. When working with many small files (i.e. unpacking a tar file with source code), a Thor (over NFS) is about 4 times slower than our aging existing storage solution, which isn't exactly speedy to begin with (17 minutes versus 3 minutes). We took a rough stab in the dark, and started to examine whether or not it was the ZIL. Performing IO tests locally on the Thor shows no real IO problems, but running IO tests over NFS, specifically, with many smaller files we see a significant performance hit. Just to rule in or out the ZIL as a factor, we disabled it, and ran the test again. It completed in just under a minute, around 3 times faster than our existing storage. This was more like it! Are there any tunables for the ZIL to try to speed things up? Or would it be best to look into using a high-speed SSD for the log device? And, yes, I already know that turning off the ZIL is a Really Bad Idea. We do, however, need to provide our users with a certain level of performance, and what we've got with the ZIL on the pool is completely unacceptable. Thanks for any pointers you may have... ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Greg Mason wrote: We're running into a performance problem with ZFS over NFS. When working with many small files (i.e. unpacking a tar file with source code), a Thor (over NFS) is about 4 times slower than our aging existing storage solution, which isn't exactly speedy to begin with (17 minutes versus 3 minutes). We took a rough stab in the dark, and started to examine whether or not it was the ZIL. It is. I've recently added some clarification to this section in the Evil Tuning Guide which might help you to arrive at a better solution. http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 Feedback is welcome. -- richard Performing IO tests locally on the Thor shows no real IO problems, but running IO tests over NFS, specifically, with many smaller files we see a significant performance hit. Just to rule in or out the ZIL as a factor, we disabled it, and ran the test again. It completed in just under a minute, around 3 times faster than our existing storage. This was more like it! Are there any tunables for the ZIL to try to speed things up? Or would it be best to look into using a high-speed SSD for the log device? And, yes, I already know that turning off the ZIL is a Really Bad Idea. We do, however, need to provide our users with a certain level of performance, and what we've got with the ZIL on the pool is completely unacceptable. Thanks for any pointers you may have... -- Greg Mason Systems Administrator Michigan State University High Performance Computing Center ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
So, what we're looking for is a way to improve performance, without disabling the ZIL, as it's my understanding that disabling the ZIL isn't exactly a safe thing to do. We're looking for the best way to improve performance, without sacrificing too much of the safety of the data. The current solution we are considering is disabling the cache flushing (as per a previous response in this thread), and adding one or two SSD log devices, as this is similar to the Sun storage appliances based on the Thor. Thoughts? -Greg On Jan 19, 2009, at 6:24 PM, Richard Elling wrote: We took a rough stab in the dark, and started to examine whether or not it was the ZIL. It is. I've recently added some clarification to this section in the Evil Tuning Guide which might help you to arrive at a better solution. http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 Feedback is welcome. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Greg Mason wrote: So, what we're looking for is a way to improve performance, without disabling the ZIL, as it's my understanding that disabling the ZIL isn't exactly a safe thing to do. We're looking for the best way to improve performance, without sacrificing too much of the safety of the data. The current solution we are considering is disabling the cache flushing (as per a previous response in this thread), and adding one or two SSD log devices, as this is similar to the Sun storage appliances based on the Thor. Thoughts? Good idea. Thor has a CF slot, too, if you can find a high speed CF card. -- richard -Greg On Jan 19, 2009, at 6:24 PM, Richard Elling wrote: We took a rough stab in the dark, and started to examine whether or not it was the ZIL. It is. I've recently added some clarification to this section in the Evil Tuning Guide which might help you to arrive at a better solution. http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29 Feedback is welcome. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
On Mon, 19 Jan 2009, Greg Mason wrote: The current solution we are considering is disabling the cache flushing (as per a previous response in this thread), and adding one or two SSD log devices, as this is similar to the Sun storage appliances based on the Thor. Thoughts? You need to add some sort of fast non-volatile cache. The Sun storage appliances are actually using battery backed DRAM for their write caches. This sort of hardware is quite rare. Fast SSD log devices are apparently pretty expensive. Some of the ones for sale are actually pretty slow. Bob == Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
Good idea. Thor has a CF slot, too, if you can find a high speed CF card. -- richard We're already using the CF slot for the OS. We haven't really found any CF cards that would be fast enough anyways :) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS over NFS, poor performance with many small files
On Mon, Jan 19 at 23:14, Greg Mason wrote: So, what we're looking for is a way to improve performance, without disabling the ZIL, as it's my understanding that disabling the ZIL isn't exactly a safe thing to do. We're looking for the best way to improve performance, without sacrificing too much of the safety of the data. The current solution we are considering is disabling the cache flushing (as per a previous response in this thread), and adding one or two SSD log devices, as this is similar to the Sun storage appliances based on the Thor. Thoughts? In general principles, the evil tuning guide states that the ZIL should be able to handle 10 seconds of expected synchronous write workload. To me, this implies that it's improving burst behavior, but potentially at the expense of sustained throughput, like would be measured in benchmarking type runs. If you have a big JBOD array with say 8+ mirror vdevs on multiple controllers, in theory, each VDEV can commit from 60-80MB/s to disk. Unless you are attaching a separate ZIL device that can match the aggregate throughput of that pool, wouldn't it just be better to have the default behavior of the ZIL contents being inside the pool itself? The best practices guide states that the max ZIL device size should be roughly 50% of main system memory, because that's approximately the most data that can be in-flight at any given instant. For a target throughput of X MB/sec and given that ZFS pushes transaction groups every 5 seconds (and have 2 outstanding), we also expect the ZIL to not grow beyond X MB/sec * 10 sec. So to service 100MB/sec of synchronous writes, 1 GBytes of log device should be sufficient. But, no comments are made on the performance requirements of the ZIL device(s) relative to the main pool devices. Clicking around finds this entry: http://blogs.sun.com/perrin/entry/slog_blog_or_blogging_on ...which appears to indicate cases where a significant number of ZILs were required to match the bandwidth of just throwing them in the pool itself. --eric -- Eric D. Mudama edmud...@mail.bounceswoosh.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
