So, assume your queue depth is 10, and you need a 1TB volume. You can present a 1TB LUN, and your whole SCSI queue for that volume is 10. If you present 2 500GB LUNs, your combined SCSI queue is 20 - one SCSI mailbox per LUN. If you present 10 100GB LUNs, you get a combined queue of 100. If you're doing small-block Oracle, with 95-98% cache read rates on the array (not unheard of), you can process more I/Os in-flight at any given point in time. You do make the valid point that making changes such as this can move the bottleneck from being storage-bound to being cpu-bound, and that's a not-uncommon result of this tuning on under-provisioned hardware. When I do this exercise with my customers, we probably run into that problem ~half the time.
As to stripe vs. concat specifically -- using a more artificial example, imagine you're putting together a 100GB volume. If you concatenate it as 2 50GB LUNs, you'll use one of the SCSI queues only, until you've filled up the first 50GB, then you'll start using both queues (mostly - this is an example). If you made the volume from, say, 10 10GB LUNs, you have a greater mathematical chance of any in-flight I/Os sorting across multiple queues, thus raising your overall I/O the application can request at any given time. Again, I want to stress that this does not mean this is *always* the right way to go, YMMV, please consult your local tuning professional or BOFH, etc. This is just an argument for why you'd ever _want_ to do it, using artificial examples. Most applications at most entities do not need this level of storage discipline. Those that do need it often stop falling over every night, by a simple storage change (or learn they need more box to meet SLAs, empirically). ZFS masks the gory details, but "ZFS" == "VxFS + VxVM", there are just different features in each product suite. Underneath the filesystem, it is still a RAID volume created on the host, that's managed by software - even if it's just one LUN. ZFS has features SF doesn't, and vice-versa, and we also have a CLI front-end that removes most of the complexity of the product for "simple" setups. My personal opinion is they fill different niches, ultimately, and neither is "better", just "better at some tasks". As to the Oracle issue, that's what the ODM API from Oracle solves, amongst other solutions . This article (http://blogs.sun.com/bobs/entry/one_i_o_two_i ) explains the benefits of concurrent, unbuffered I/O for Oracle better than I am able to. Cheers, jf As always, anything sent to this list are my own thoughts, and I speak only for myself. From: veritas-vx-boun...@mailman.eng.auburn.edu [mailto:veritas-vx-boun...@mailman.eng.auburn.edu] On Behalf Of Hudes, Dana Sent: Wednesday, January 06, 2010 11:10 AM To: William Havey Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] Relayout volume from 2 to 4 columns So indeed this feature turns your $$$$$$$ Hitachi 9990V into a JBOD. Wow. I guess there are products that need this sort of thing. And enterprises where the people running the SAN array can't manage it properly so the server administrators have need of bypassing them. The other question of SCSI queues one per column as a benefit of striping is interesting. Doesn't this just generate extra disk i/o? The array is already doing RAID with 2 parity stripes. Now what? Yet this is effectively what ZFS does so there must be a performance gain. Hmm. Multiple SCSI queues actually might possibly make sense if you have a large number of CPUs (like the Sun 4v architecture esp. 5240 with 128 cores or 5440 with 256, or a 4+ board domain on the SunFire 25K which gives you 32 cores) all of which are running threads that do disk i/o. This benefit seems more practicaly in the ZFS approach where you have one volume-equivalent (the zpool is both disk group and VM volume in that it has storage layout) and many filesystems so you would likely have multiple processes doing independent disk i/o. In the VxVM one volume one filesstem model your e.g. Oracle table space is in one filesystem as one huge file (possibly other databases are files in other filesystems). Even if you have multiple listeners doing their thing ultimately there's one file they're working on...of course Oracle has row locking and other paralleization ....hmm. ________________________________ From: William Havey [mailto:bbha...@gmail.com] Sent: Wednesday, January 06, 2010 12:30 PM To: Hudes, Dana Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] Relayout volume from 2 to 4 columns Yes, it certainly does. And that is why Symantec put the feature in the VM product; to use host-based software to construct and control storage from host to physical disk. This would help eliminate multi-vendor chaos in the storage aspects of the data center. On Wed, Jan 6, 2010 at 12:19 PM, Hudes, Dana <hud...@hra.nyc.gov> wrote: >the ISP feature of VM would allow you to drill down to individual spindles and place subdisks on each spindle. Individual spindles of the RAID group? Doesn't that defeat the purpose of the RAID group? Striping across LUNs gets ...interesting; we usually just use them concat. Of course that's with a real SAN array such as Hitachi 99x0 or Sun 61x0. I'm not sure I see the point of striping LUNs. If you are having performance problems from the array, fix the layout of the RAID group on the array: that's why you pay the big bucks to Hitachi for their hardware. I not sure I want to know about the load that could flatline a RAID-6 array of 6 15K RPM Fiber channel disks backed by a multigigabyte RAM cache. I have certainly seen bad storage layout on the host cause hot spots. That's when people make ridiculous numbers of small (gigabyte or so) volumes scattered all over the place -- another argument against the old way of doing things with databases and raw volumes (if you're going to use raw volumes at least use decent size ones not 2GB each ). While old (< 10) Solaris AIO did indeed suck dead bunnies thorugh a straw for performance, that's no longer a problem in Solaris 10 ZFS if you use it natively (using "branded" zones to run Solaris 8 and 9 puts the old AIO interface in front) nor would I expect it to be a problem with VxFS. ________________________________ From: veritas-vx-boun...@mailman.eng.auburn.edu [mailto:veritas-vx-boun...@mailman.eng.auburn.edu] On Behalf Of William Havey Sent: Wednesday, January 06, 2010 12:00 PM To: przemol...@poczta.fm Cc: veritas-vx@mailman.eng.auburn.edu Subject: Re: [Veritas-vx] Relayout volume from 2 to 4 columns VM views the two raid groups as single LUNs. It needn't be concerned with the layout of each raid group. To change from 2 columns to 4 columns use the relayout option to vxassist and also specify the two new LUNs on which to place the two new columns. That being said, the ISP feature of VM would allow you to drill down to individual spindles and place subdisks on each spindle. Bill On Wed, Jan 6, 2010 at 6:36 AM, <przemol...@poczta.fm> wrote: Hello, we are using VSF 5.0 MP3 on Solaris 10 attached to SAN-based hardware array. On this array we have created 2 raid groups and on each RG we have created a few LUNs: raid group: RG1 RG2 LUN1 LUN7 LUN2 LUN8 LUN3 LUN9 LUN4 LUN10 LUN5 LUN11 LUN6 LUN12 For performance reason some of our volumes are striped between the two raid groups (using two columns ncol=2) e.g.: pl <name> <vol> ENABLED ACTIVE 419256320 STRIPE 2/128 RW In this configuration IOs involves two raid groups. It seems that in the future in certain cases performance might be not as expected so we would like to add two additional LUNs (taken from two additional raid groups) and relayout the whole volume from 2-col to 4-cols e.g.: raid group: RG1 RG2 RG3 RG4 LUN1 LUN7 LUN13 LUN19 LUN2 LUN8 LUN14 LUN20 LUN3 LUN9 LUN15 LUN21 LUN4 LUN10 LUN16 LUN22 LUN5 LUN11 LUN17 LUN23 LUN6 LUN12 LUN18 LUN24 Is it possible to order relayout of existings volumes to spread it over all four RGs ? Can I point somehow that it should relayout using these particular LUNs ? Regards Przemyslaw Bak (przemol) -- http://przemol.blogspot.com/ ---------------------------------------------------------------------- Milosc, praca, pieniadze. Sprawdz swoj horoskop na dzis >> http://link.interia.pl/f2531 _______________________________________________ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx _______________________________________________ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
_______________________________________________ Veritas-vx maillist - Veritas-vx@mailman.eng.auburn.edu http://mailman.eng.auburn.edu/mailman/listinfo/veritas-vx
