Re: [zfs-discuss] Is another drive worth anything? [Summary]
On Jun 2, 2011, at 20:50, Edward Ned Harvey wrote: > Also, if you have an SSD for cache device, you accelerate reads, and there > is absolutely no data risk. In the event of cache device failure, > performance degrades back to the "normal" level and everything continues > just fine. Dropping back down to the "normal" level of performance though may be an issue if you've now become dependent on the higher performance an SSD has presumably been giving you. "Normal" may in fact be "degraded". ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything? [Summary]
On Thu, Jun 02, 2011 at 09:59:39PM -0400, Edward Ned Harvey wrote: > > From: Daniel Carosone [mailto:d...@geek.com.au] > > Sent: Thursday, June 02, 2011 9:03 PM > > > > Separately, with only 4G of RAM, i think an L2ARC is likely about a > > wash, since L2ARC entries also consume RAM. > > True the L2ARC requires some ARC consumption to support it, but for typical > user data, it's a huge multiplier... The ARC consumption is static per entry > (say, 176 bytes, depending on your platform) but a typical payload for user > data would be whatever your average blocksize is ... 40K, 127K, or something > similar probably. Yes, but that's not the whole story. In order for the L2ARC to be an effective performance boost, it itself needs to be large enough to save enough hits on the disks. Further, the penalty of these hits is more in IOPS than size. Both these tend to reduce or nullify the (space) scaling factor, other than getting the very largest blocks out of primary cache. Addiing read iops with a third submirror, at no cost, is the way to go (or at least the way to start) in this case. -- Dan. pgpQobf345OXI.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything? [Summary]
> From: Daniel Carosone [mailto:d...@geek.com.au] > Sent: Thursday, June 02, 2011 9:03 PM > > Separately, with only 4G of RAM, i think an L2ARC is likely about a > wash, since L2ARC entries also consume RAM. True the L2ARC requires some ARC consumption to support it, but for typical user data, it's a huge multiplier... The ARC consumption is static per entry (say, 176 bytes, depending on your platform) but a typical payload for user data would be whatever your average blocksize is ... 40K, 127K, or something similar probably. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything? [Summary]
Thanks, I like this summary format and the effort it took to produce seems well-spent. On Thu, Jun 02, 2011 at 08:50:58PM -0400, Edward Ned Harvey wrote: > > but I figured spending 500G on ZIL > > would be unwise. > > You couldn't possibly ever use 500G of ZIL, because the ZIL is required to > be flushed to disk at least once every 5sec to 30sec (depending on which > build you're running.) Even if you have a 4G dedicated log device, that's > more than plenty for most purposes. It is also limited to at most half of physical memory, as I recall. Remember that SZIL is nonvolatile backing store for in-memory write structures that have to remain until txg close anyway. Separately, with only 4G of RAM, i think an L2ARC is likely about a wash, since L2ARC entries also consume RAM. The extra details provided just confirm that the 3-way-mirror is the best tweak for this existing system with no cost. -- Dan. pgpt5VI8Bht4K.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything? [Summary]
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Dave U.Random > > I am planning to revisit SSD again > when the consumer drives are reliable enough and don't have wear issues. > Right now overall integrity and long service life are more important > than absolute performance on this box, although since I have the integrity > with the ZFS mirror I could add an SSD but I really don't want to deal with > another failure as long as I don't have to. I do want additional performance > if I can afford it, but not at the expense of possible data loss. Right now, (some) consumer drives are reliable enough and don't have wear issues. Not all drives are created equal, but I'm sure you can find an affordable one. Also, if you have an SSD for cache device, you accelerate reads, and there is absolutely no data risk. In the event of cache device failure, performance degrades back to the "normal" level and everything continues just fine. If you have an unmirrored SSD for log device, you accelerate certain types of writes, and while it's not technically true to say there's no data risk, you can generally say there is essentially no data risk. The real truth is, only in the condition of the SSD going bad, and an ungraceful system crash for some unrelated reason coinciding within 5 sec or 30 sec of each other, then up to 5 sec or 30 sec of sync writes could possibly be lost, but the filesystem will never lose integrity. So really truly, there is essentially no data risk. Acceptable for home use, not acceptable for a bank processing transactions. > I don't know what L2ARC is, but I'll take a look on the net. L2ARC is synonymous with "cache" device. System uses the SSD to cache data from the primary storage pool, for the sake of accelerating reads. > I did hear > about ZIL but don't understand it fully, People often say ZIL when they're really talking about a dedicated ZIL log device. It's not correct to use the terms like that, so please stick with "log device" when talking about a dedicated log device. ZIL is the temporary place where sync writes are stored, until they can be written to their permanent locations on primary storage pool. If you don't have a dedicated log device, the ZIL resides on disk with the primary storage pool. So you're able to accelerate sync writes, if you have a dedicated log device which is faster than the primary storage pool. > but I figured spending 500G on ZIL > would be unwise. You couldn't possibly ever use 500G of ZIL, because the ZIL is required to be flushed to disk at least once every 5sec to 30sec (depending on which build you're running.) Even if you have a 4G dedicated log device, that's more than plenty for most purposes. Normally you add a SSD or something like a DDRDrive for dedicated log. Any device which is really fast, even if it is really small, no problem. As long as it's nonvolatile. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything? [Summary]
Many thanks to all who responded. I learned a lot from this thread! For now I have decided to make a 3 way mirror because of the read performance. I don't want to take a risk on an unmirrored drive. Instead of replying to everyone separately I am following the Sun Managers system since I read that newsgroup occasionalliy also. Here's a summary of the responses. Jim Klimov wrote: > Well, you can use this drive as a separate "scratch area", as a separate > single-disk pool, without redundancy. You'd have a separate spindle for > some dedicated tasks with data you're okay with losing. I thought about that and I really don't like losing data. I also don't generate much temporary data so I love ZFS because it makes mirroring easy. On my other systems where I don't have ZFS I run hourly backups from drive to drive. Consumer drives are pretty good these days but you never know when one will fail. I had a failure recently on a Linux box and although I didn't lose data because I back up hourly it's still annoying to deal with. If I hadn't had another good drive with that data on it I would have lost critical data. > You can also make the rpool a three-way mirror which may increase read > speeds if you have enough concurrentcy. And when one drive breaks, your > rpool is still mirrored. I think that's the best suggestion. I didn't realize a 3 way mirror would help performance but you and several others said it does, so that's what I will do. Thanks for the suggestions, Jim. Roy pointed out a theoretical 50% read increase when adding the third drive. Thanks Roy! Edward Ned Harvey wrote: > In my benchmarking, I found 2-way mirror reads 1.97x the speed of a single > disk, and a 3-way mirror reads 2.91x a single disk. Always great having hard data to base a decision on! That helped me make my decision! Thanks Edward! Jim Klimov answered a question that came up based on comments that read performance was improved in a three way mirror: > Writes in a mirror are deemed to be not faster than the slowest disk - all > two or three drives must commit a block before it is considered written > (in sync write mode), likewise for TXG sync but with some optimization by > caching and write-coalescing. Thanks Jim! Good to know. Edward Ned Harvey pointed out "If you make it a 3-way mirror, your write performance will be unaffected, but your read performance will increase 50% over a 2-way mirror. All 3 drives can read different data simultaneously for the net effect of 3x a single disk read performance." Bob clarified the theoretical benefit of adding a third drive to a mirror by saying "I think that a read performance increase of (at most) 33.3% is more correct. You might obtain (at most) 50% over one disk by mirroring it. Zfs makes a random selection of which disk to read from in a mirror set so the improvement is not truely linear." Thanks guys, that makes sense. Daniel Carosone suggested keeping the extra drive around in case of a failure and in the meantime using an SSD in the 3rd SATA slot. He pointed out a few other options that could help with performance besides creating a 3 way mirror when he wrote: > Namely, leave the third drive on the shelf as a cold spare, and use the > third sata connector for an ssd, as L2ARC, ZIL or even possibly both > (which will affect selection of which device to use). That's not an option for me right now but I am planning to revisit SSD again when the consumer drives are reliable enough and don't have wear issues. Right now overall integrity and long service life are more important than absolute performance on this box, although since I have the integrity with the ZFS mirror I could add an SSD but I really don't want to deal with another failure as long as I don't have to. I do want additional performance if I can afford it, but not at the expense of possible data loss. Daniel also wrote: > L2ARC is likely to improve read latency (on average) even more than a > third submirror. ZIL will be unmirrored, but may improve writes at an > acceptable risk for development system. If this risk is acceptable, you > may wish to consider whether setting sync=disabled is also acceptable at > least for certain datasets. I don't know what L2ARC is, but I'll take a look on the net. I did hear about ZIL but don't understand it fully, but I figured spending 500G on ZIL would be unwise. By that I mean I understand ZIL doesn't require much storage but if I don't have an identical drive I can't add a drive or slice with less storage than the other drives in a mirror to that mirror, so I would be forced to waste a lot of storage to implement ZIL. > Finally, if you're considering spending money, can you increase the RAM > instead? If so, do that first. This mobo is maxed out at 4G, it's a socket 775 I bought a couple of years ago. I have always seen the benefits to more RAM and I agree with you it helps more than people generally believe. Next time I buy a new box I am hoping to g
Re: [zfs-discuss] Is another drive worth anything?
On Wed, Jun 1, 2011 at 7:06 AM, Bill Sommerfeld wrote: > On 05/31/11 09:01, Anonymous wrote: >> Hi. I have a development system on Intel commodity hardware with a 500G ZFS >> root mirror. I have another 500G drive same as the other two. Is there any >> way to use this disk to good advantage in this box? I don't think I need any >> more redundancy, I would like to increase performance if possible. I have >> only one SATA port left so I can only use 3 drives total unless I buy a PCI >> card. Would you please advise me. Many thanks. > > I'd use the extra SATA port for an ssd, and use that ssd for some > combination of boot/root, ZIL, and L2ARC. > > I have a couple systems in this configuration now and have been quite > happy with the config. While slicing an ssd and using one slice for > root, one slice for zil, and one slice for l2arc isn't optimal from a > performance standpoint and won't scale up to a larger configuration, it > is a noticeable improvement from a 2-disk mirror. > > I used an 80G intel X25-M, with 1G for zil, with the rest split roughly > 50:50 between root pool and l2arc for the data pool. Does anyone have a benchmark or history data on how reliable an SSD is nowadays? Cheap-ish sandforce-based MLC SSDs usually say they support 1 million write cycles, and that they have some kind of wear-leveling. How does this translates when it's used as L2ARC? Can we expect something like one year or three years lifetime when the pool is relatively busy? -- Fajar ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On Wed, Jun 01, 2011 at 05:45:14AM +0400, Jim Klimov wrote: > Also, in a mirroring scenario is there any good reason to keep a warm spare > instead of making a three-way mirror right away (beside energy saving)? > Rebuild times and non-redundant windows can be decreased considerably ;) Perhaps where the spare may be used for any of several pools, whichever has a failure first. Not relevant to this case.. In this case, if the drive is warm, it might as well be live. My point was that, even as a cold spare it is worth something, and that the sata port may be worth more, since the OP is more interested in performance than extra redundancy. -- Dan. pgp8cB9ApGE1h.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
> If it is powered on, then it is a warm spare :-) > Warm spares are a good idea. For some platforms, you can > spin down the > disk so it doesn't waste energy. But I should note that we've had issues with a hot spare disk added to rpool in particular, preventing boots on Solaris 10u8. It turned out to be a known bug which may have since been fixed... Also, in a mirroring scenario is there any good reason to keep a warm spare instead of making a three-way mirror right away (beside energy saving)? Rebuild times and non-redundant windows can be decreased considerably ;) //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On May 31, 2011, at 5:16 PM, Daniel Carosone wrote: > Namely, leave the third drive on the shelf as a cold spare, and use > the third sata connector for an ssd, as L2ARC, ZIL or even possibly > both (which will affect selection of which device to use). If it is powered on, then it is a warm spare :-) Warm spares are a good idea. For some platforms, you can spin down the disk so it doesn't waste energy. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
> What about writes? Writes in a mirror are deemed to be not faster than the slowest disk - all two or three drives must commit a block before it is considered written (in sync write mode), likewise for TXG sync but with some optimization by caching and write-coalescing. //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On Wed, Jun 01, 2011 at 10:16:28AM +1000, Daniel Carosone wrote: > On Tue, May 31, 2011 at 06:57:53PM -0400, Edward Ned Harvey wrote: > > If you make it a 3-way mirror, your write performance will be unaffected, > > but your read performance will increase 50% over a 2-way mirror. All 3 > > drives can read different data simultaneously for the net effect of 3x a > > single disk read performance. > > This would be my recommendation too, but for the sake of completeness, > there are other options that may provide better performance > improvement (at a cost) depending on your needs. In fact, I should state even more clearly: do this, since there is very little reason not to. Measure the benefit. Move on to the other things if the benefit is not enough. When doing so, consider what kind of benefit you're looking for. > Namely, leave the third drive on the shelf as a cold spare, and use > the third sata connector for an ssd, as L2ARC, ZIL or even possibly > both (which will affect selection of which device to use). > > L2ARC is likely to improve read latency (on average) even more than a > third submirror. ZIL will be unmirrored, but may improve writes at an > acceptable risk for development system. If this risk is acceptable, > you may wish to consider whether setting sync=disabled is also > acceptable at least for certain datasets. > > Finally, if you're considering spending money, can you increase the > RAM instead? If so, do that first. > > -- > Dan. > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss pgpHRSk23bsVr.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On Tue, May 31, 2011 at 06:57:53PM -0400, Edward Ned Harvey wrote: > If you make it a 3-way mirror, your write performance will be unaffected, > but your read performance will increase 50% over a 2-way mirror. All 3 > drives can read different data simultaneously for the net effect of 3x a > single disk read performance. This would be my recommendation too, but for the sake of completeness, there are other options that may provide better performance improvement (at a cost) depending on your needs. Namely, leave the third drive on the shelf as a cold spare, and use the third sata connector for an ssd, as L2ARC, ZIL or even possibly both (which will affect selection of which device to use). L2ARC is likely to improve read latency (on average) even more than a third submirror. ZIL will be unmirrored, but may improve writes at an acceptable risk for development system. If this risk is acceptable, you may wish to consider whether setting sync=disabled is also acceptable at least for certain datasets. Finally, if you're considering spending money, can you increase the RAM instead? If so, do that first. -- Dan. pgpt1w2jn0CGs.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On 05/31/11 09:01, Anonymous wrote: > Hi. I have a development system on Intel commodity hardware with a 500G ZFS > root mirror. I have another 500G drive same as the other two. Is there any > way to use this disk to good advantage in this box? I don't think I need any > more redundancy, I would like to increase performance if possible. I have > only one SATA port left so I can only use 3 drives total unless I buy a PCI > card. Would you please advise me. Many thanks. I'd use the extra SATA port for an ssd, and use that ssd for some combination of boot/root, ZIL, and L2ARC. I have a couple systems in this configuration now and have been quite happy with the config. While slicing an ssd and using one slice for root, one slice for zil, and one slice for l2arc isn't optimal from a performance standpoint and won't scale up to a larger configuration, it is a noticeable improvement from a 2-disk mirror. I used an 80G intel X25-M, with 1G for zil, with the rest split roughly 50:50 between root pool and l2arc for the data pool. - Bill ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On May 31, 2011, at 19:00, Edward Ned Harvey wrote: >> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- >> boun...@opensolaris.org] On Behalf Of Roy Sigurd Karlsbakk >> >> Theoretically, you'll get a 50% read increase, but I doubt it'll be that >> high in >> practice. What about writes? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
On Tue, 31 May 2011, Edward Ned Harvey wrote: If you make it a 3-way mirror, your write performance will be unaffected, but your read performance will increase 50% over a 2-way mirror. All 3 drives can read different data simultaneously for the net effect of 3x a single disk read performance. I think that a read performance increase of (at most) 33.3% is more correct. You might obtain (at most) 50% over one disk by mirroring it. Zfs makes a random selection of which disk to read from in a mirror set so the improvement is not truely linear. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Roy Sigurd Karlsbakk > > Theoretically, you'll get a 50% read increase, but I doubt it'll be that high > in > practice. In my benchmarking, I found 2-way mirror reads 1.97x the speed of a single disk, and a 3-way mirror reads 2.91x a single disk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Anonymous > > Hi. I have a development system on Intel commodity hardware with a 500G > ZFS > root mirror. I have another 500G drive same as the other two. Is there any > way to use this disk to good advantage in this box? I don't think I need any > more redundancy, I would like to increase performance if possible. I have > only one SATA port left so I can only use 3 drives total unless I buy a PCI > card. Would you please advise me. Many thanks. If you make it a 3-way mirror, your write performance will be unaffected, but your read performance will increase 50% over a 2-way mirror. All 3 drives can read different data simultaneously for the net effect of 3x a single disk read performance. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
> Hi. I have a development system on Intel commodity hardware with a > 500G ZFS > root mirror. I have another 500G drive same as the other two. Is there > any > way to use this disk to good advantage in this box? I don't think I > need any > more redundancy, I would like to increase performance if possible. I > have > only one SATA port left so I can only use 3 drives total unless I buy > a PCI > card. Would you please advise me. Many thanks. A third drive in the mirror (aka three-way mirror) will increase read performance from the pool, as ZFS reads from all drives in a mirror. Theoretically, you'll get a 50% read increase, but I doubt it'll be that high in practice. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Is another drive worth anything?
> Hi. I have a development system on Intel commodity hardware with > a 500G ZFS > root mirror. I have another 500G drive same as the other two. Is > there any > way to use this disk to good advantage in this box? I don't > think I need any > more redundancy, I would like to increase performance if > possible. I have > only one SATA port left so I can only use 3 drives total unless > I buy a PCI > card. Would you please advise me. Many thanks. Well, you can use this drive as a separate "scratch area", as a separate single-disk pool, without redundancy. You'd have a separate spindle for some dedicated tasks with data you're okay with losing. You can also make the rpool a three-way mirror which may increase read speeds if you have enough concurrentcy. And when one drive breaks, your rpool is still mirrored. HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
