Hi all,
We are running pretty large vdevs since the initial testing showed that
our setup was not too much off the optimum. However, under real world
load we do see quite some weird behaviour:
The system itself is a X4500 with 500 GB drives and right now the system
seems to be under heavy load, e.g. ls takes minutes to return on only a
few hundred entries, top shows 10% kernel, rest idle.
zpool ioststat -v atlashome 60 shows (not the first output):
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
atlashome 2.11T 18.8T 2.29K 36 71.7M 138K
raidz2 466G 6.36T 493 11 14.9M 34.1K
c0t0d0 - - 48 5 1.81M 3.52K
c1t0d0 - - 48 5 1.81M 3.46K
c4t0d0 - - 48 5 1.81M 3.27K
c6t0d0 - - 48 5 1.81M 3.40K
c7t0d0 - - 47 5 1.81M 3.40K
c0t1d0 - - 47 5 1.81M 3.20K
c1t1d0 - - 47 6 1.81M 3.59K
c4t1d0 - - 47 6 1.81M 3.53K
c5t1d0 - - 47 5 1.81M 3.33K
c6t1d0 - - 48 6 1.81M 3.67K
c7t1d0 - - 48 6 1.81M 3.66K
c0t2d0 - - 48 5 1.82M 3.42K
c1t2d0 - - 48 6 1.81M 3.56K
c4t2d0 - - 48 6 1.81M 3.54K
c5t2d0 - - 48 5 1.81M 3.41K
raidz2 732G 6.10T 800 12 24.6M 52.3K
c6t2d0 - - 139 5 7.52M 4.54K
c7t2d0 - - 139 5 7.52M 4.81K
c0t3d0 - - 140 5 7.52M 4.98K
c1t3d0 - - 139 5 7.51M 4.47K
c4t3d0 - - 139 5 7.51M 4.82K
c5t3d0 - - 139 5 7.51M 4.99K
c6t3d0 - - 139 5 7.52M 4.44K
c7t3d0 - - 139 5 7.52M 4.78K
c0t4d0 - - 139 5 7.52M 4.97K
c1t4d0 - - 139 5 7.51M 4.60K
c4t4d0 - - 139 5 7.51M 4.86K
c6t4d0 - - 139 5 7.51M 4.99K
c7t4d0 - - 139 5 7.51M 4.52K
c0t5d0 - - 139 5 7.51M 4.78K
c1t5d0 - - 138 5 7.51M 4.94K
raidz2 960G 6.31T 1.02K 12 32.2M 52.0K
c4t5d0 - - 178 5 9.29M 4.79K
c5t5d0 - - 178 5 9.28M 4.64K
c6t5d0 - - 179 5 9.29M 4.44K
c7t5d0 - - 178 4 9.26M 4.26K
c0t6d0 - - 178 5 9.28M 4.78K
c1t6d0 - - 178 5 9.20M 4.58K
c4t6d0 - - 178 5 9.26M 4.25K
c5t6d0 - - 177 4 9.21M 4.18K
c6t6d0 - - 178 5 9.29M 4.69K
c7t6d0 - - 177 5 9.26M 4.61K
c0t7d0 - - 177 5 9.29M 4.34K
c1t7d0 - - 177 5 9.24M 4.28K
c4t7d0 - - 177 5 9.29M 4.78K
c5t7d0 - - 177 5 9.27M 4.75K
c6t7d0 - - 177 5 9.29M 4.34K
c7t7d0 - - 177 5 9.27M 4.28K
---------- ----- ----- ----- ----- ----- -----
Questions:
(a) Why the first vdev does not get an equal share of the load
(b) Why is a large raidz2 so bad? When I use a standard Linux box with
hardware raid6 over 16 disks I usually get more bandwidth and at least
about the same small file performance
(c) Would the use of several smaller vdev would help much? And which
layout would be a good compromise for getting space as well as
performance and reliability? 46 disks have so few prime factors
Thanks a lot
Carsten
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
