Running a zpool scrub on our production pool is showing a scrub rate
of about 400K/s. (When this pool was first set up we saw rates in the
MB/s range during a scrub).
Both zpool iostat and an iostat -Xn show lots of idle disk times, no
above average service times, no abnormally high busy percentag
> The scrub I/O has lower priority than other I/O.
>
> In later ZFS releases, scrub I/O is also throttled. When the throttle
> kicks in, the scrub can drop to 5-10 IOPS. This shouldn't be much of
> an issue, scrubs do not need to be, and are not intended to be, run
> very often -- perhaps once a qu
> Can you share your 'zpool status' output for both pools?
Faster, smaller server:
~# zpool status pool0
pool: pool0
state: ONLINE
scan: scrub repaired 0 in 2h18m with 0 errors on Sat May 14 13:28:58 2011
Much larger, more capable server:
~# zpool status pool0 | head
pool: pool0
state: ONLINE
> Can you send the entire 'zpool status' output? I wanted to see your
> pool configuration. Also run the mdb command in a loop (at least 5
> tiimes) so we can see if spa_last_io is changing. I'm surprised you're
> not finding the symbol for 'spa_scrub_inflight' too. Can you check
> that you didn't
> I copy and pasted to make sure that wasn't the issue :)
Which, ironically, turned out to be the problem- there was an extra
carriage return in there that mdb did not like:
Here is the output:
spa_name = [ "pool0" ]
spa_last_io = 0x82721a4
spa_scrub_inflight = 0x1
spa_name = [ "pool0" ]
spa_las
Here is another example of the performance problems I am seeing:
~# dd if=/dev/zero of=/pool0/ds.test bs=1024k count=2000 2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 56.2184 s, 37.3 MB/s
37MB/s seems like some sort of bad joke for all these disks. I can
write the same a
> You mentioned that the pool was somewhat full, can you send the output
> of 'zpool iostat -v pool0'?
~# zpool iostat -v pool0
capacity operationsbandwidth
poolalloc free read write read write
-- - - - - -
> You mentioned that the pool was somewhat full, can you send the output
> of 'zpool iostat -v pool0'? You can also try doing the following to
> reduce 'metaslab_min_alloc_size' to 4K:
>
> echo "metaslab_min_alloc_size/Z 1000" | mdb -kw
So just changing that setting moved my write rate from 40MB/s
As a followup:
I ran the same DD test as earlier- but this time I stopped the scrub:
pool0 14.1T 25.4T 88 4.81K 709K 262M
pool0 14.1T 25.4T104 3.99K 836K 248M
pool0 14.1T 25.4T360 5.01K 2.81M 230M
pool0 14.1T 25.4T305 5.69K 2.38M 231M
> metaslab_min_alloc_size is not the metaslab size. From the source
Sorry- that was simply a slip of the mind- it was a long day.
> By reducing this value, it is easier for the allocator to identify a
> metaslab for allocation as the file system becomes full.
Thank you for clarifying. Is there a
I posted this to the forums a little while ago but I believe the list
was split at the time:
Does anyone have any recommendations for changing the ZFS volblocksize
when creating zfs volumes to serve as VMFS backing stores?
I've seen several people recommend that the volblocksize be set to 64k
in
Wow- so a bit of an update:
With the default scrub delay:
echo "zfs_scrub_delay/K" | mdb -kw
zfs_scrub_delay:20004
pool0 14.1T 25.3T165499 1.28M 2.88M
pool0 14.1T 25.3T146 0 1.13M 0
pool0 14.1T 25.3T147 0 1.14M 0
pool0 14.1T
> Try setting the zfs_scrub_delay to 1 but increase the
> zfs_top_maxinflight to something like 64.
The array is running some regression tests right now but when it
quiets down I'll try that change.
-Don
___
zfs-discuss mailing list
zfs-discuss@opensolar
>> Try setting the zfs_scrub_delay to 1 but increase the
>> zfs_top_maxinflight to something like 64.
With the delay set to 1 or higher it doesn't matter what I set the
maxinflight value to- when I check with:
echo "::walk spa | ::print spa_t spa_name spa_last_io spa_scrub_inflight"
The value ret
> Two drives have been resilvered, but the old drives still stick. The drive
> that has died still hasn't been taken over by a spare, although the two
> spares show up as AVAIL.
For the one that hasn't been replaced try doing:
zpool replace dbpool c8t24d0 c4t43d0
For the two that have already be
> Yup; reset storms affected us as well (we were using the X-25 series
> for ZIL/L2ARC). Only the ZIL drives were impacted, but it was a large
> impact :)
What did you see with your reset storm? Were there log errors in
/var/adm/messages or did you need to check the controller loogs with
something
>> One day, the write performance of zfs degrade.
>> The write performance decrease from 60MB/s to about 6MB/s in sequence
>> write.
>>
>> Command:
>> date;dd if=/dev/zero of=block bs=1024*128 count=1;date
See this thread:
http://www.opensolaris.org/jive/thread.jspa?threadID=139317&tstart=45
> In Solaris 10u8:
> root@nas-hz-01:~# uname -a
> SunOS nas-hz-01 5.10 Generic_141445-09 i86pc i386 i86pc
> root@nas-hz-01:~# echo "metaslab_min_alloc_size/K" | mdb -kw
> mdb: failed to dereference symbol: unknown symbol name
Fair enough. I don't have anything older than b147 at this point so I
was
> Another (less satisfying) workaround is to increase the amount of free space
> in the pool, either by reducing usage or adding more storage. Observed
> behavior is that allocation is fast until usage crosses a threshhold, then
> performance hits a wall.
We actually tried this solution. We were at
> There is snapshot of metaslab layout, the last 51 metaslabs have 64G free
> space.
After we added all the disks to our system we had lots of free
metaslabs- but that didn't seem to matter. I don't know if perhaps the
system was attempting to balance the writes across more of our devices
but whate
20 matches
Mail list logo
