Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Is there an issue with having additional resource that at support each other? If information is well documented, then it will be easy to tell if it is out of date. Regardless does the current HCL answer the questions I posed? On Fri, Feb 6, 2009 at 2:26 PM, Richard Elling wrote: > Nicholas Lee wrote: > > >> On Fri, Feb 6, 2009 at 11:29 AM, Richard Elling >> > richard.ell...@gmail.com>> wrote: >> >> >>Seriously, is it so complicated that a best practice page is needed? >> >> While you might be right about that, I think there is a need for a good >> shared experiences site, howtos, etc. >> >> For example, I want to put a new 2U 12 disk storage system together. I'll >> use a Supermicro chasis, but I wonder what is best for: >> >>* AMD or Intel? >>* Is the AOC-SAT2-MV8 still the best SATA controller or should I >> get a LSI card. >>* The AOC-USASLP-S8iR has battery back cache option. Will this >> work with Solaris? Which release? Can it do JBOD? Will the >> battery back cache actually be useful? >>* Then of course there is all the SSD options. Which SSD units are >> better for write/ZIL, which is better for read/L2ARC? >> >> >> All these questions are pretty general, and being asked often. I think it >> would be good to find some way to consolidate this stuff. >> > > Yes, but this becomes yet another HCL -- not best practices. The > info will be outdated in weeks or even days. Does the current HCL suit? If > not, how would you propose a better solution? > -- richard > > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Nicholas Lee wrote: > > On Fri, Feb 6, 2009 at 11:29 AM, Richard > Elling mailto:richard.ell...@gmail.com>> wrote: > > > Seriously, is it so complicated that a best practice page is needed? > > > While you might be right about that, I think there is a need for a > good shared experiences site, howtos, etc. > > For example, I want to put a new 2U 12 disk storage system together. > I'll use a Supermicro chasis, but I wonder what is best for: > > * AMD or Intel? > * Is the AOC-SAT2-MV8 still the best SATA controller or should I > get a LSI card. > * The AOC-USASLP-S8iR has battery back cache option. Will this > work with Solaris? Which release? Can it do JBOD? Will the > battery back cache actually be useful? > * Then of course there is all the SSD options. Which SSD units are > better for write/ZIL, which is better for read/L2ARC? > > > All these questions are pretty general, and being asked often. I > think it would be good to find some way to consolidate this stuff. Yes, but this becomes yet another HCL -- not best practices. The info will be outdated in weeks or even days. Does the current HCL suit? If not, how would you propose a better solution? -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
On Fri, Feb 6, 2009 at 11:29 AM, Richard Elling wrote: > > Seriously, is it so complicated that a best practice page is needed? While you might be right about that, I think there is a need for a good shared experiences site, howtos, etc. For example, I want to put a new 2U 12 disk storage system together. I'll use a Supermicro chasis, but I wonder what is best for: - AMD or Intel? - Is the AOC-SAT2-MV8 still the best SATA controller or should I get a LSI card. - The AOC-USASLP-S8iR has battery back cache option. Will this work with Solaris? Which release? Can it do JBOD? Will the battery back cache actually be useful? - Then of course there is all the SSD options. Which SSD units are better for write/ZIL, which is better for read/L2ARC? All these questions are pretty general, and being asked often. I think it would be good to find some way to consolidate this stuff. Nicholas ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Orvar Korvar wrote: > I'll second that. A wiki page on the ZFS wiki, with best practices and > recommendations about adding SSD would be great. There is not much > information on this subject, I feel. Case scenarios, blogs, etc showing some > numbers. > Seriously, is it so complicated that a best practice page is needed? NB Best practices tend to be neither time nor version-specific, because versions change over time. For version-specific things, release notes are more appropriate. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
I'll second that. A wiki page on the ZFS wiki, with best practices and recommendations about adding SSD would be great. There is not much information on this subject, I feel. Case scenarios, blogs, etc showing some numbers. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Not sure is best to put something like this. There is wikis like http://www.solarisinternals.com/wiki/index.php/Solaris_Internals_and_Performance_FAQ http://wiki.genunix.org/wiki/index.php/WhiteBox_ZFSStorageServer But I haven't seen anything which has an active community like http://www.thinkwiki.org/wiki/ThinkWiki Is it worthwhile to setup a something like this? Adding together information like: http://jkontherun.com/2009/02/04/new-192gb-ssd-from-transcend-runs-at-high-speed/ I think would be very useful. On Thu, Feb 5, 2009 at 6:30 AM, Miles Nordin wrote: > In case you want to start one, here are the (incomplete) notes I've > kept from the list: > > http://www.vmetro.com/category4304.html -- ZFS log device, prestoserv of > the future. also Gigabyte iRAM. > there are some companies, Crucial and STEC come to mind, > sell SSDs which fit in disk form factors. IIRC, Mac Book Air and EMC > use STEC's SSDs. -- Richard Elling > the ones that Mtron sells were testing as the fastest ones on the market. > -- Brian Hechinger 2008-06-27 > http://www.xbitlabs.com/articles/storage/display/ssd-iram_5.html > http://www.stec-inc.com/product/zeusiops.php > http://www.fusionio.com/ > > http://forums.storagereview.net/index.php?s=&showtopic=27190&view=findpost&p=253758 > http://www.hyperossystems.co.uk/ > - Ross > http://www.opensolaris.org/jive/thread.jspa?threadID=65074&tstart=30 > - relling > > http://www.marketwatch.com/news/story/STEC-Support-Suns-Unified-Storage/story.aspx?guid=%7B07043E00-7628-411D-B24A-2FFEC8B8F706%7D >The ZEUS product line makes a fine slog while the MACH8 product line > works nicely for L2ARC. [from the Sun 7000] > - Greg Mason > Intel X25E. It's around $600 It's got about half the performance of STEC > Zeus drives > - David Dyer-Bennet > There's a problem in the MLC SSD drives with particular JMicron > controller chips where some patterns of small writes trigger ~~10sec. > hangs, which is most often called "stuttering" in online discussions of it > that I've seen. My particular example is an OCZ Core 2 64GB > unit. > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Is it possible for someone to put up a wiki page somewhere with the various SSD, ZIL, L2ARC options with Pros, Cons and Benchmarks. Especially with notes like the below. Given this is a key area of interest for zfs at the moment, seems like it would be a useful resource. On Wed, Feb 4, 2009 at 11:01 AM, David Dyer-Bennet wrote: > > On Tue, February 3, 2009 15:26, Richard Elling wrote: > > Orvar Korvar wrote: > > >> Which is the best SSD disk to add? > > > > The one that somebody else pays for :-) > > Does the access pattern for say the ZIL tickle the "stuttering" bug in the > JMicron-controller multi-level-cell SSDs? That is, in all the cheap ones? > Somehow early reviews managed to miss it, so I'm stuck with one, and will > eventually replace it with something faster -- and then will be looking > for a way to use it productively. Or has all the testing centered around > enterprise-grade stuff and nobody knows about the cheaper drives' > performance? > > (There's a problem in the MLC SSD drives with particular JMicron > controller chips where some patterns of small writes trigger ~~10sec. > hangs, which is most often called "stuttering" in online discussions of it > that I've seen. Using it as a Windows system disk unfortunately hits these > patterns far too frequently. My particular example is an OCZ Core 2 64GB > unit. Oh; I should say I've been through the list of registry tweaks, and > I can't see they've helped more than a little.) > > -- > David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/ > Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ > Photos: http://dd-b.net/photography/gallery/ > Dragaera: http://dragaera.info > > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
On Tue, February 3, 2009 15:26, Richard Elling wrote: > Orvar Korvar wrote: >> Which is the best SSD disk to add? > > The one that somebody else pays for :-) Does the access pattern for say the ZIL tickle the "stuttering" bug in the JMicron-controller multi-level-cell SSDs? That is, in all the cheap ones? Somehow early reviews managed to miss it, so I'm stuck with one, and will eventually replace it with something faster -- and then will be looking for a way to use it productively. Or has all the testing centered around enterprise-grade stuff and nobody knows about the cheaper drives' performance? (There's a problem in the MLC SSD drives with particular JMicron controller chips where some patterns of small writes trigger ~~10sec. hangs, which is most often called "stuttering" in online discussions of it that I've seen. Using it as a Windows system disk unfortunately hits these patterns far too frequently. My particular example is an OCZ Core 2 64GB unit. Oh; I should say I've been through the list of registry tweaks, and I can't see they've helped more than a little.) -- David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Orvar Korvar wrote: > So are there no guide lines how to add a SSD disk as a home user? I'm not sure any guidelines are needed beyond that which is already documented in the zpool man page or the ZFS Administration Guide. What do you believe is missing from those docs? > Which is the best SSD disk to add? The one that somebody else pays for :-) > What percentage improvements are typical? Or, will a home user not benefit > from adding a SSD drive? YMMV. I do not believe there will be a way to provide a generic answer to this question. > It is only enterprise SSD drives that works, together with some esoteric > software from Fishworks? The Sun Storage 7000 systems use ZFS to manage the storage pools. > It requires Enterprise hardware to get a boost from SSD? Not possible? Or? > > No one has done this yet? Many people have experimented with the separate log features using a wide variety of devices. -- richard > What does the Fishworks team say? > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Orvar, With my testing, i've seen a 5x improvement with small file creation when working specifically with NFS. This is after I added an SSD for the ZIL. I recommend Richard Elling's zilstat (he posted links earlier). It'll let you see if a dedicated device for the ZIL will help your specific workload. My understanding is that you'll get "more bang for the buck" using an SSD for the ZIL rather than the L2ARC. Performing some of your own benchmarks is really the only way see what will help improve performance for your specific workload. I recommend reading up on the ZFS ARC and L2ARC, to help try to determine if testing a dedicated L2ARC device is even worthwhile for your uses. I know it wasn't really helpful for me, as our read performance is already great. As for a specific SSD, I've tested the Intel X25E. It's around $600 or so. It's got about half the performance of the snazzy, pricey STEC Zeus drives. With the specific workload I was trying to accelerate, I wasn't hitting any of the limits of the Intel SSDs (but I was definitely WAY past the performance limits of a standard hard disk). Again, all of this was for accelerating the ZIL, not for use on the L2ARC, so YMMV. Fishworks does this. They use an SSD both for the read cache as well as the ZIL. -Greg Orvar Korvar wrote: > So are there no guide lines how to add a SSD disk as a home user? Which is > the best SSD disk to add? What percentage improvements are typical? Or, will > a home user not benefit from adding a SSD drive? It is only enterprise SSD > drives that works, together with some esoteric software from Fishworks? It > requires Enterprise hardware to get a boost from SSD? Not possible? Or? > > No one has done this yet? What does the Fishworks team say? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
So are there no guide lines how to add a SSD disk as a home user? Which is the best SSD disk to add? What percentage improvements are typical? Or, will a home user not benefit from adding a SSD drive? It is only enterprise SSD drives that works, together with some esoteric software from Fishworks? It requires Enterprise hardware to get a boost from SSD? Not possible? Or? No one has done this yet? What does the Fishworks team say? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
How were you running this test? were you running it locally on the machine, or were you running it over something like NFS? What is the rest of your storage like? just direct-attached (SAS or SATA, for example) disks, or are you using a higher-end RAID controller? -Greg kristof wrote: > Kebabber, > > You can't expose zfs filesystems over iSCSI. > > You only can expose ZFS volumes (raw volumes) over iscsi. > > PS: 2 weeks ago I did a few tests, using filebench. > > I saw little to no improvement using a 32GB Intel X25E SSD. > > Maybe this is because filebench is flushing the cache in between tests. > > I also compared iscsi boot time (using gpxe as boot loader) , > > We are using raidz storagepool (4disks). here again, adding the X25E as cache > device did not speedup the boot proccess. So I did not see real improvement. > > PS: We have 2 master volumes (xp and vista) which we clone to provision > additional guests. > > I'm now waiting for new SSD disks (STEC Zeus 18GB en STEC Mach 100GB.), since > those are used in SUN 7000 product. I hope they perform better. > > Kristof ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Kebabber, You can't expose zfs filesystems over iSCSI. You only can expose ZFS volumes (raw volumes) over iscsi. PS: 2 weeks ago I did a few tests, using filebench. I saw little to no improvement using a 32GB Intel X25E SSD. Maybe this is because filebench is flushing the cache in between tests. I also compared iscsi boot time (using gpxe as boot loader) , We are using raidz storagepool (4disks). here again, adding the X25E as cache device did not speedup the boot proccess. So I did not see real improvement. PS: We have 2 master volumes (xp and vista) which we clone to provision additional guests. I'm now waiting for new SSD disks (STEC Zeus 18GB en STEC Mach 100GB.), since those are used in SUN 7000 product. I hope they perform better. Kristof -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Imagine 10 SATA discs in raidz2 and one or two SSD drives as a cache. Each Vista client reaches ~90MB/sec to the server, using Solaris CIFS and iSCSI. So you want to use iSCSI with this. (iSCSI allows ZFS to export a file system as a native SCSI disc to a desktop PC. The desktop PC can mount this iSCSI disk as a native SCSI disk and format it with NTFS - on top of ZFS with snapshots, etc. This is done from desktop PC bios). Now, you install WinXP on the iSCSI ZFS volume, and clone it with a snapshot. Then you can boot from the clone, with the iSCSI volume on a desktop PC. Thus, your desktop PC doesnt need any hard drive at all. It uses the iSCSI volume on the ZFS server as a native SCSI disk, which has WinXP installed. This way, you can deploy lots of desktop PC in an instant. Using the cloned WinXP snapshot. And if there are problems e.g virus, just destroy the clone and create a new one in one second. Has anyone does this? Does the SSD provide extra speed? Any stories to share? (Ive read this iSCSI suggestion on a blogg with black background color, it's not my idea). If I add a SSD disk as a cache, can I remove it? No? Will there be problems if I remove it? Can I exchange it with a bigger? (Trying to convert these Windows people) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Thanx for your answers guys. :o) I dont contemplating trying this for my ZFS raid, as the SSD drives are expensive right now. I just want to be able to answer questions when I convert Windows/Linux to Solaris. And therefore collect info. Has anyone tried this on a blogg? Would be cool to blog about. "Before" and "After". BTW, did I tell you that Solaris rocks? :o) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Orvar, In an existing RAIDZ configuration, you would add the cache device like this: # zpool add pool-name cache device-name Currently, cache devices are only supported in the OpenSolaris and SXCE releases. The important thing is determining whether the cache device would improve your workload's performance, following Richard's advice. Can you try and buy an SSD? :-) Cindy Mark J Musante wrote: > On Wed, 28 Jan 2009, Richard Elling wrote: > > >>Orvar Korvar wrote: >> >> >>>I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a >>>similar vein? Would it be easy to do? >> >>Yes. >> > > > To be specific, you use the 'cache' argument to zpool, as in: > > zpool create <...> cache > > > Regards, > markm > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
On Wed, 28 Jan 2009, Richard Elling wrote: > Orvar Korvar wrote: > >> I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a >> similar vein? Would it be easy to do? > > Yes. > To be specific, you use the 'cache' argument to zpool, as in: zpool create <...> cache Regards, markm ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
Orvar Korvar wrote: > I understand Fishworks has a L2ARC cache, which as I have understood it, is a > SSD drive as a cache? > Fishworks is an engineering team, I hear they have many L2ARCs in their lab :-) Yes, the Sun Storage 7000 series systems can have read-optimized SSDs for use as L2ARC devices. > I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a similar > vein? Yes. You should look for a read-optimized SSD, for best performance gains. > Would it be easy to do? Yes. > What would be the impact? It depends on the workload. In general, it helps workloads which do random reads, perhaps by 10x or more. > Has anyone tried this? > Yes. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Add SSD drive as L2ARC(?) cache to existing ZFS raid?
I understand Fishworks has a L2ARC cache, which as I have understood it, is a SSD drive as a cache? I have 5 terabyte discs in a raidz1. Could I add one SSD drive in a similar vein? Would it be easy to do? What would be the impact? Has anyone tried this? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
