OK, that was the magic incantation I was looking for:
- changing the noprefetch option opened the floodgates to the L2ARC
- changing the max queue depth relived the wait time on the drives, although
I may undo this again in the benchmarking since these drives all have NCQ
I went from all four disks of the array at 100%, doing about 170 read
IOPS/25MB/s
to all four disks of the array at 0%, once hitting nealyr 500 IOPS/65MB/s
off the cache drive (@ only 50% load).
This bodes well for adding a second mirrored cache drive to push for the
1KIOPS.
Now I am ready to insert the mirror for the ZIL and the CACHE, and we will
be ready
for some production benchmarking.
device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
sd0 170.0 0.4 7684.7 0.0 0.0 35.0 205.3 0 100 11 80 0 82
sd1 168.4 0.4 7680.2 0.0 0.0 34.6 205.1 0 100
sd2 172.0 0.4 7761.7 0.0 0.0 35.0 202.9 0 100
sd4 170.0 0.4 7727.1 0.0 0.0 35.0 205.3 0 100
sd5 1.6 2.6 182.4 104.8 0.0 0.5 117.8 0 31
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d1
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d2
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d3
285.2 0.8 36236.2 14.4 0.0 0.5 0.0 1.8 1 37 c0t0d4
And, keep in mind this was on less than $1000 of hardware.
Thanks,
Tracey
On Sat, Feb 13, 2010 at 9:22 AM, Richard Elling <richard.ell...@gmail.com>wrote:
> comment below...
>
> On Feb 12, 2010, at 2:25 PM, TMB wrote:
> > I have a similar question, I put together a cheapo RAID with four 1TB WD
> Black (7200) SATAs, in a 3TB RAIDZ1, and I added a 64GB OCZ Vertex SSD, with
> slice 0 (5GB) for ZIL and the rest of the SSD for cache:
> > # zpool status dpool
> > pool: dpool
> > state: ONLINE
> > scrub: none requested
> > config:
> >
> > NAME STATE READ WRITE CKSUM
> > dpool ONLINE 0 0 0
> > raidz1 ONLINE 0 0 0
> > c0t0d0 ONLINE 0 0 0
> > c0t0d1 ONLINE 0 0 0
> > c0t0d2 ONLINE 0 0 0
> > c0t0d3 ONLINE 0 0 0
> > [b] logs
> > c0t0d4s0 ONLINE 0 0 0[/b]
> > [b] cache
> > c0t0d4s1 ONLINE 0 0 0[/b]
> > spares
> > c0t0d6 AVAIL
> > c0t0d7 AVAIL
> >
> > capacity operations bandwidth
> > pool used avail read write read write
> > ---------- ----- ----- ----- ----- ----- -----
> > dpool 72.1G 3.55T 237 12 29.7M 597K
> > raidz1 72.1G 3.55T 237 9 29.7M 469K
> > c0t0d0 - - 166 3 7.39M 157K
> > c0t0d1 - - 166 3 7.44M 157K
> > c0t0d2 - - 166 3 7.39M 157K
> > c0t0d3 - - 167 3 7.45M 157K
> > c0t0d4s0 20K 4.97G 0 3 0 127K
> > cache - - - - - -
> > c0t0d4s1 17.6G 36.4G 3 1 249K 119K
> > ---------- ----- ----- ----- ----- ----- -----
> > I just don't seem to be getting any bang for the buck I should be. This
> was taken while rebuilding an Oracle index, all files stored in this pool.
> The WD disks are at 100%, and nothing is coming from the cache. The cache
> does have the entire DB cached (17.6G used), but hardly reads anything from
> it. I also am not seeing the spike of data flowing into the ZIL either,
> although iostat show there is just write traffic hitting the SSD:
> >
> > extended device statistics cpu
> > device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id
> > sd0 170.0 0.4 7684.7 0.0 0.0 35.0 205.3 0 100 11 8 0 82
> > sd1 168.4 0.4 7680.2 0.0 0.0 34.6 205.1 0 100
> > sd2 172.0 0.4 7761.7 0.0 0.0 35.0 202.9 0 100
> > sd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
> > sd4 170.0 0.4 7727.1 0.0 0.0 35.0 205.3 0 100
> > [b]sd5 1.6 2.6 182.4 104.8 0.0 0.5 117.8 0 31 [/b]
>
> iostat has a "n" option, which is very useful for looking at device names
> :-)
>
> The SSD here is perfoming well. The rest are clobbered. 205 millisecond
> response time will be agonizingly slow.
>
> By default, for this version of ZFS, up to 35 I/Os will be queued to the
> disk, which is why you see 35.0 in the "actv" column. The combination
> of actv=35 and svc_t>200 indicates that this is the place to start working.
> Begin by reducing zfs_vdev_max_pending from 35 to something like 1 to 4.
> This will reduce the concurrent load on the disks, thus reducing svc_t.
>
> http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29
>
> -- richard
>
> > Since this SSD is in a RAID array, and just presents as a regular disk
> LUN, is there a special incantation required to turn on the Turbo mode?
> >
> > Doesnt it seem that all this traffic should be maxing out the SSD? Reads
> from the cache, and writes to the ZIL? I have a seocnd identical SSD I
> wanted to add as a mirror, but it seems pointless if there's no zip to be
> had....
> >
> > help?
> >
> > Thanks,
> > Tracey
> > --
> > This message posted from opensolaris.org
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss@opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
