Here's an update:
I thought that the error message
arcmsr0: too many outstanding commands
might be due to a Scsi queue being over ran
The areca driver has
#*define* ARCMSR_MAX_OUTSTANDING_CMD
<http://src.opensolaris.org/source/s?defs=ARCMSR_MAX_OUTSTANDING_CMD> 256
What might be happening is each raid set results in a new instance of the areca
driver getting loaded so perhaps
the scsi queue on the card is just get over ran as each drive is getting a
queue depth of 256, as such I tested with sd_max_throttle:16
(16 Drives * 16 Queues = 256)
I verified sd_max_throttle got set via:
r...@yoda:~/solaris-install-stuff# echo "sd_max_throttle/D" |mdb -k
sd_max_throttle:
sd_max_throttle:16
I see that if I run this script to create a bunch of small files I can make a
lot of drives jump to degrade in a hurry.
#!/bin/bash
dir=/backup/homebackup/junk
mkdir -p $dir
cd $dir
date
printf "Creating 10000 1k files in $dir \n"
i=10000
while [ $i -ge 0 ]
do
j=`expr $i - 1`
dd if=/dev/zero of=$i count=1 bs=1k &> /dev/null
i=$j
done
date
i=10000
printf "Deleting 10000 1k files in $dir \n"
while [ $i -ge 0 ]
do
j=`expr $i - 1`
rm $i
i=$j
done
date
Before running the script:
r...@yoda:~# zpool status
pool: backup
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
raidz1 ONLINE 0 0 2
c4t2d0 ONLINE 0 0 0
c4t3d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c4t5d0 ONLINE 0 0 0
c4t6d0 ONLINE 0 0 0
c4t7d0 ONLINE 0 0 0
c4t8d0 ONLINE 0 0 0
raidz1 ONLINE 0 0 2
c4t9d0 ONLINE 0 0 0
c4t10d0 ONLINE 0 0 0
c4t11d0 ONLINE 0 0 0
c4t12d0 ONLINE 0 0 0
c4t13d0 ONLINE 0 0 0
c4t14d0 ONLINE 0 0 0
c4t15d0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0
c4t1d0s0 ONLINE 0 0 0
errors: No known data errors
AFTER running the script:
r...@yoda:~/solaris-install-stuff# zpool status -v
pool: backup
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backup DEGRADED 0 0 5
raidz1 DEGRADED 0 0 14
c4t2d0 DEGRADED 0 0 0 too many errors
c4t3d0 ONLINE 0 0 0
c4t4d0 ONLINE 0 0 0
c4t5d0 DEGRADED 0 0 0 too many errors
c4t6d0 ONLINE 0 0 0
c4t7d0 DEGRADED 0 0 0 too many errors
c4t8d0 DEGRADED 0 0 0 too many errors
raidz1 DEGRADED 0 0 12
c4t9d0 DEGRADED 0 0 0 too many errors
c4t10d0 DEGRADED 0 0 0 too many errors
c4t11d0 DEGRADED 0 0 0 too many errors
c4t12d0 DEGRADED 0 0 0 too many errors
c4t13d0 DEGRADED 0 0 0 too many errors
c4t14d0 DEGRADED 0 0 0 too many errors
c4t15d0 DEGRADED 0 0 1 too many errors
errors: Permanent errors have been detected in the following files:
backup/homebackup:<0x0>
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0s0 ONLINE 0 0 0
c4t1d0s0 ONLINE 0 0 0
errors: No known data errors
BTW
I called Seagate to check the drive firmware. They confirm that Firmware
version 3.AEK is the latest for the drives I have. This is the version
running on all 16 of my drives.
I'm about out of ideas to try.
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
