On Fri, Feb 12, 2010 at 02:25:51PM -0800, TMB wrote: > I have a similar question, I put together a cheapo RAID with four 1TB WD > Black (7200) SATAs, in a 3TB RAIDZ1, and I added a 64GB OCZ Vertex SSD, with > slice 0 (5GB) for ZIL and the rest of the SSD for cache: > # zpool status dpool > pool: dpool > state: ONLINE > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > dpool ONLINE 0 0 0 > raidz1 ONLINE 0 0 0 > c0t0d0 ONLINE 0 0 0 > c0t0d1 ONLINE 0 0 0 > c0t0d2 ONLINE 0 0 0 > c0t0d3 ONLINE 0 0 0 > [b] logs > c0t0d4s0 ONLINE 0 0 0[/b] > [b] cache > c0t0d4s1 ONLINE 0 0 0[/b] > spares > c0t0d6 AVAIL > c0t0d7 AVAIL > > capacity operations bandwidth > pool used avail read write read write > ---------- ----- ----- ----- ----- ----- ----- > dpool 72.1G 3.55T 237 12 29.7M 597K > raidz1 72.1G 3.55T 237 9 29.7M 469K > c0t0d0 - - 166 3 7.39M 157K > c0t0d1 - - 166 3 7.44M 157K > c0t0d2 - - 166 3 7.39M 157K > c0t0d3 - - 167 3 7.45M 157K > c0t0d4s0 20K 4.97G 0 3 0 127K > cache - - - - - - > c0t0d4s1 17.6G 36.4G 3 1 249K 119K > ---------- ----- ----- ----- ----- ----- ----- > I just don't seem to be getting any bang for the buck I should be. This was > taken while rebuilding an Oracle index, all files stored in this pool. The > WD disks are at 100%, and nothing is coming from the cache. The cache does > have the entire DB cached (17.6G used), but hardly reads anything from it. I > also am not seeing the spike of data flowing into the ZIL either, although > iostat show there is just write traffic hitting the SSD: > > extended device statistics cpu > device r/s w/s kr/s kw/s wait actv svc_t %w %b us sy wt id > sd0 170.0 0.4 7684.7 0.0 0.0 35.0 205.3 0 100 11 8 0 82 > sd1 168.4 0.4 7680.2 0.0 0.0 34.6 205.1 0 100 > sd2 172.0 0.4 7761.7 0.0 0.0 35.0 202.9 0 100 > sd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 > sd4 170.0 0.4 7727.1 0.0 0.0 35.0 205.3 0 100 > [b]sd5 1.6 2.6 182.4 104.8 0.0 0.5 117.8 0 31 [/b] > > Since this SSD is in a RAID array, and just presents as a regular disk LUN, > is there a special incantation required to turn on the Turbo mode? > > Doesnt it seem that all this traffic should be maxing out the SSD? Reads from > the cache, and writes to the ZIL? I have a seocnd identical SSD I wanted to > add as a mirror, but it seems pointless if there's no zip to be had....
The most likely reason is that this workload has been identified as streaming by ZFS, which is prefetching from disk instead of the L2ARC (l2arc_nopreftch=1). It also looks like you've used a 128 Kbyte ZFS record size. Is Oracle doing 128 Kbyte random I/O? We usually tune that down before creating the database; which will use the L2ARC device more efficiently. Brendan -- Brendan Gregg, Fishworks http://blogs.sun.com/brendan _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
