On Dec 31, 2009, at 2:49 AM, Robert Milkowski wrote:
judging by a *very* quick glance it looks like you have an issue
with c3t0d0 device which is responding very slowly.
Yes, there is an I/O stuck on the device which is not getting serviced.
See below...
--
Robert Milkowski
http://milek.blogspot.com
On 31/12/2009 09:10, Emily Grettel wrote:
Hi,
I'm using OpenSolaris 127 from my previous posts to address CIFS
problems. I have a few zpools but lately (with an uptime of 32
days) we've started to get CIFS issues and really bad IO
performance. I've been running scrubs on a nightly basis.
I'm not sure why its happenning either - I'm new to OpenSolaris.
I ran fsstat whilst trying to unrar a 8.4Gb file with an ISO inside
it:
fsstat zfs 1
new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
3.29K 466 367 633K 1.50K 1.66M 8.66K 964K 15.6G 314K 9.38G
zfs
0 0 0 4 0 8 0 135 5.13M 93 5.02M
zfs
0 0 0 7 0 18 0 205 5.63M 137 5.64M
zfs
0 0 0 4 0 8 0 90 3.92K 49 14.6K
zfs
0 0 0 4 0 8 0 115 16.4K 65 27.5K
zfs
0 0 0 8 0 13 0 153 8.36M 113 8.38M
zfs
0 0 0 4 0 8 0 94 3.96K 53 19.1K
zfs
0 0 0 7 0 18 0 80 800 42 1.13K
zfs
0 0 0 4 0 8 0 90 3.92K 48 7.62K
zfs
0 0 0 4 0 8 0 99 132K 53 7.14K
zfs
0 0 0 4 0 8 0 188 5.99K 96 5.62K
zfs
0 0 0 4 0 8 0 95 664K 52 420K
zfs
0 0 0 9 0 22 0 164 7.97K 92 12.2K
zfs
new name name attr attr lookup rddir read read write write
file remov chng get set ops ops ops bytes ops bytes
0 0 0 4 0 8 0 111 2.63M 70 2.63M
zfs
0 0 0 4 0 8 0 262 6.63M 153 6.63M
zfs
0 0 0 4 0 8 0 80 800 44 1.70K
zfs
0 0 0 4 0 8 0 337 18.1M 247 18.1M
zfs
0 0 0 7 0 18 0 127 5.75M 89 5.63M
zfs
0 0 0 4 0 8 0 80 800 50 25.6K
zfs
My iostat appears below this message (its quite long! to give you
an idea). I'm really not sure why the performance has really
dropped all of a sudden or how to diagnose it. CIFS shares
occasionally drop out too.
Its a bit of a downer to be experiencing on the 31st of December. I
hope everyone has a Safe & Happy New Years :-)
I'm unable to upgrade to the latest release because of an issue
with python:
pfexec pkg image-update
Creating Plan /pkg: Cannot remove 'pkg://opensolaris.org/sunwipkg-gui-l...@0.5.11
,5.11-0.127:20091111T075414Z' due to the following packages that
depend on it:
pkg://opensolaris.org/SUNWipkg-
g...@0.5.11,5.11-0.127:20091111T075333Z
So I'm stuck on 127 until I can rebuild this machine :(
Cheers,
Em
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 0.5 2.8 1.0 2815.9 1000.0 100 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 100 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 6.0 0.0 81.5 1.2 1.0 198.6 166.6 60 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 9.0 0.0 31.5 0.0 0.4 0.0 41.7 0 38 c7t3d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t5d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.1 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
56.0 122.0 5972.3 8592.2 0.0 0.5 0.0 2.9 0 25 c7t1d0
55.0 136.0 5998.3 8590.2 0.0 0.7 0.0 3.8 0 29 c7t2d0
0.0 111.0 0.0 4342.9 0.0 2.2 0.0 20.2 0 57 c7t3d0
103.0 153.0 5868.3 8590.7 0.0 0.4 0.0 1.7 0 21 c7t4d0
96.0 130.0 5946.8 8591.2 0.0 0.7 0.0 3.2 0 35 c7t5d0
56.0 120.0 5998.8 8592.7 0.0 0.5 0.0 2.7 0 24 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 4.0 0.0 257.0 0.0 1.8 0.0 437.8 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
Here you can see that actv=1, which means that there is an I/O
queued to the device. But there is no r/s or w/s, so the I/O is
not being serviced.
This could be a broken disk, or it could be some other
hardware/software/firmware issue. Check the errors on the
device with
iostat -En
You should also check the fma logs:
fmadm faulty
fmdump -eV
-- richard
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 22.0 0.0 1987.5 0.0 1.0 0.0 45.4 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
18.0 18.0 2050.5 2004.0 0.0 1.0 0.0 27.7 0 100 c7t3d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 10.6 0 1 c7t4d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t5d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
35.0 0.0 3948.0 0.0 0.0 1.0 0.0 28.5 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 10.0 0.0 6.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 3.0 0.0 1.5 0.0 2.2 0.0 749.8 0 100 c7t3d0
0.0 8.0 0.0 6.5 0.0 0.1 0.0 7.3 0 2 c7t4d0
0.0 7.0 0.0 7.0 0.0 0.0 0.0 1.6 0 0 c7t5d0
0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 26.0 0.0 24.0 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 26.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 9.0 0.0 8.5 0.0 4.0 0.0 443.9 0 100 c7t3d0
0.0 14.0 0.0 23.5 0.0 0.0 0.0 3.1 0 1 c7t4d0
0.0 25.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c7t5d0
0.0 27.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 21.0 0.0 19.5 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 21.0 0.0 17.5 0.0 0.0 0.0 0.4 0 0 c7t2d0
0.0 15.0 0.0 23.5 0.0 4.0 0.0 266.3 0 100 c7t3d0
0.0 17.0 0.0 16.5 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 16.0 0.0 19.0 0.0 0.1 0.0 4.8 0 5 c7t5d0
0.0 21.0 0.0 21.5 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 0.5 0.0 4.0 0.0 3953.4 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t1d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t2d0
0.0 15.0 0.0 20.0 0.0 2.2 0.0 146.8 0 73 c7t3d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t4d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c7t5d0
0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t3d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.4 0 0 c7t4d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0
0.0 5.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 144.9 0.0 7431.8 0.0 0.4 0.0 2.6 0 14 c7t1d0
0.0 162.9 0.0 6404.2 0.0 0.4 0.0 2.3 0 14 c7t2d0
0.0 210.9 0.0 5983.8 0.0 0.5 0.0 2.5 0 15 c7t3d0
0.0 295.9 0.0 8585.9 0.0 0.2 0.0 0.5 0 6 c7t4d0
0.0 161.9 0.0 5710.9 0.0 0.3 0.0 1.7 0 15 c7t5d0
0.0 176.9 0.0 8588.4 0.0 0.3 0.0 1.9 0 14 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
56.0 11.0 5974.7 1157.9 0.0 0.2 0.0 3.7 0 13 c7t1d0
208.1 19.0 5481.0 2184.3 0.0 0.1 0.0 0.4 0 6 c7t2d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
211.1 0.0 5506.5 0.0 0.0 0.1 0.0 0.3 0 4 c7t4d0
117.0 25.0 5818.6 2877.5 0.0 0.2 0.0 1.3 0 10 c7t5d0
100.0 0.0 6000.7 0.0 0.0 0.1 0.0 1.4 0 4 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 4.0 0.0 231.0 0.0 3.0 0.0 762.5 0 100 c7t3d0
0.0 89.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t0d0
0.0 88.0 0.0 3859.3 0.0 0.1 0.2 0.8 1 7 c3t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 21.0 0.0 1961.5 0.0 1.0 0.0 47.5 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t1d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c7t2d0
31.0 4.0 3589.0 413.0 0.0 1.0 0.0 28.6 0 100 c7t3d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.4 0 0 c7t4d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 13.1 0 1 c7t5d0
0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.1 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
21.0 0.0 2410.0 0.0 0.0 1.0 0.0 47.6 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 11.0 0.0 8.0 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 9.0 0.0 6.5 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 3.0 0.0 1.5 0.0 1.1 0.0 354.8 0 100 c7t3d0
0.0 8.0 0.0 6.5 0.0 0.0 0.0 1.8 0 0 c7t4d0
0.0 7.0 0.0 6.0 0.0 0.1 0.0 9.2 0 2 c7t5d0
0.0 11.0 0.0 8.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 41.0 0.0 25.5 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 34.0 0.0 21.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 9.0 0.0 9.5 0.0 4.0 0.0 443.9 0 100 c7t3d0
0.0 37.0 0.0 22.0 0.0 0.0 0.0 0.2 0 0 c7t4d0
0.0 31.0 0.0 20.0 0.0 0.0 0.0 0.2 0 0 c7t5d0
0.0 38.0 0.0 23.5 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 29.0 0.0 21.0 0.0 0.0 0.0 0.4 0 0 c7t1d0
0.0 26.0 0.0 16.5 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 22.0 0.0 27.0 0.0 4.0 0.0 181.5 0 100 c7t3d0
0.0 19.0 0.0 18.5 0.0 0.0 0.0 2.3 0 1 c7t4d0
0.0 24.0 0.0 16.0 0.0 0.0 0.0 0.3 0 0 c7t5d0
0.0 29.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 24.0 0.0 21.0 0.0 0.0 0.0 0.3 0 0 c7t1d0
0.0 21.0 0.0 17.0 0.0 0.0 0.0 0.2 0 0 c7t2d0
0.0 16.0 0.0 18.0 0.0 4.0 0.0 249.6 0 100 c7t3d0
0.0 22.0 0.0 18.0 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 18.0 0.0 17.0 0.0 0.1 0.0 3.7 0 6 c7t5d0
0.0 22.0 0.0 20.5 0.0 0.0 0.0 0.3 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c7t1d0
0.0 6.0 0.0 4.0 0.0 0.3 0.0 50.8 0 8 c7t2d0
0.0 24.0 0.0 25.5 0.0 0.4 0.0 16.6 0 10 c7t3d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t4d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.3 0 0 c7t5d0
0.0 6.0 0.0 4.0 0.0 0.0 0.0 0.2 0 0 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
14.0 102.0 1256.0 6792.5 0.0 0.3 0.0 2.7 0 16 c7t1d0
15.0 146.0 1281.5 6791.5 0.0 0.3 0.0 1.9 0 10 c7t2d0
0.0 57.0 0.0 2028.0 0.0 3.1 0.0 54.1 0 77 c7t3d0
36.0 67.0 1204.0 6792.0 0.0 0.3 0.0 2.5 0 12 c7t4d0
37.0 128.0 1203.5 6792.0 0.0 0.4 0.0 2.6 0 17 c7t5d0
30.0 89.0 1281.5 6793.0 0.0 0.3 0.0 2.4 0 13 c0t1d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 c7t3d0
Australia's #1 job site If It Exists, You'll Find it on SEEK
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
