Hi Cindy,
Thanks for the advice. This is just a little old Gateway PC provisioned as an
informal workgroup server. The main storage is two SATA drives in an external
enclosure, connected to a Sil3132 PCIe eSATA controller. The OS is snv_134b,
upgraded from snv_111a.
I can't identify a cause in particular. The box has been running for several
months without much oversight. It's possible that the two eSATA cables got
reconnected to different ports after a recent move.
The backup has been made and I will try the export & import, per your advice
(if zpool command works - it does again at the moment, no reboot!). I will also
try switching the eSATA cables to opposite ports.
Thanks,
Alex
Command output follows:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c5t1d0 <ATA-WDC WD5000AAKS-0-1D05-465.76GB>
/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0
1. c8d0 <DEFAULT cyl 9726 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0
2. c9d0 <DEFAULT cyl 38910 alt 2 hd 255 sec 63>
/pci@0,0/pci-ide@1f,2/ide@1/cmdk@0,0
3. c11t0d0 <WD-Ext HDD 1021-2002-931.51GB>
/pci@0,0/pci107b,5058@1a,7/storage@1/disk@0,0
# zpool history tank
History for 'tank':
2010-06-18.15:14:16 zpool create tank c13t0d0
2011-05-07.02:00:07 zpool scrub tank
2011-05-14.02:00:08 zpool scrub tank
2011-05-21.02:00:12 zpool scrub tank
<a million 'zfs snapshot' and 'zfs destroy' events from zfs-auto-snap omitted>
# zdb -l /dev/dsk/c5t1d0s0
--------------------------------------------
LABEL 0
--------------------------------------------
version: 14
name: 'tank'
state: 0
txg: 3374337
pool_guid: 6242690959503408617
hostid: 8697169
hostname: 'wdssandbox'
top_guid: 17982590661103377266
guid: 1717308203478351258
vdev_children: 1
vdev_tree:
type: 'mirror'
id: 0
guid: 17982590661103377266
whole_disk: 0
metaslab_array: 23
metaslab_shift: 32
ashift: 9
asize: 500094468096
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1717308203478351258
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 27
children[1]:
type: 'disk'
id: 1
guid: 9267693216478869057
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 893
--------------------------------------------
LABEL 1
--------------------------------------------
version: 14
name: 'tank'
state: 0
txg: 3374337
pool_guid: 6242690959503408617
hostid: 8697169
hostname: 'wdssandbox'
top_guid: 17982590661103377266
guid: 1717308203478351258
vdev_children: 1
vdev_tree:
type: 'mirror'
id: 0
guid: 17982590661103377266
whole_disk: 0
metaslab_array: 23
metaslab_shift: 32
ashift: 9
asize: 500094468096
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1717308203478351258
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 27
children[1]:
type: 'disk'
id: 1
guid: 9267693216478869057
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 893
--------------------------------------------
LABEL 2
--------------------------------------------
version: 14
name: 'tank'
state: 0
txg: 3374337
pool_guid: 6242690959503408617
hostid: 8697169
hostname: 'wdssandbox'
top_guid: 17982590661103377266
guid: 1717308203478351258
vdev_children: 1
vdev_tree:
type: 'mirror'
id: 0
guid: 17982590661103377266
whole_disk: 0
metaslab_array: 23
metaslab_shift: 32
ashift: 9
asize: 500094468096
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1717308203478351258
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 27
children[1]:
type: 'disk'
id: 1
guid: 9267693216478869057
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 893
--------------------------------------------
LABEL 3
--------------------------------------------
version: 14
name: 'tank'
state: 0
txg: 3374337
pool_guid: 6242690959503408617
hostid: 8697169
hostname: 'wdssandbox'
top_guid: 17982590661103377266
guid: 1717308203478351258
vdev_children: 1
vdev_tree:
type: 'mirror'
id: 0
guid: 17982590661103377266
whole_disk: 0
metaslab_array: 23
metaslab_shift: 32
ashift: 9
asize: 500094468096
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 1717308203478351258
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 27
children[1]:
type: 'disk'
id: 1
guid: 9267693216478869057
path: '/dev/dsk/c5t1d0s0'
devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a'
phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a'
whole_disk: 1
DTL: 893
On May 20, 2011, at 8:34 AM, Cindy Swearingen wrote:
> Hi Alex
>
> More scary than interesting to me.
>
> What kind of hardware and which Solaris release?
>
> Do you know what steps lead up to this problem? Any recent hardware
> changes?
>
> This output should tell you which disks were in this pool originally:
>
> # zpool history tank
>
> If the history identifies tank's actual disks, maybe you can determine
> which disk is masquerading as c5t1d0.
>
> If that doesn't work, accessing the individual disk entries in format
> should tell which one is the problem, if its only one.
>
> I would like to see the output of this command:
>
> # zdb -l /dev/dsk/c5t1d0s0
>
> Make sure you have a good backup of your data. If you need to pull a
> disk to check cabling, or rule out controller issues, you should
> probably export this pool first. Have a good backup.
>
> Others have resolved minor device issues by exporting/importing the
> pool but with format/zpool commands hanging on your system, I'm not
> confident that this operation will work for you.
>
> Thanks,
>
> Cindy
>
> On 05/19/11 12:17, Alex wrote:
>> I thought this was interesting - it looks like we have a failing drive in
>> our mirror, but the two device nodes in the mirror are the same:
>> pool: tank
>> state: DEGRADED
>> status: One or more devices could not be used because the label is missing or
>> invalid. Sufficient replicas exist for the pool to continue
>> functioning in a degraded state.
>> action: Replace the device using 'zpool replace'.
>> see: http://www.sun.com/msg/ZFS-8000-4J
>> scrub: scrub completed after 1h9m with 0 errors on Sat May 14 03:09:45 2011
>> config:
>> NAME STATE READ WRITE CKSUM
>> tank DEGRADED 0 0 0
>> mirror-0 DEGRADED 0 0 0
>> c5t1d0 ONLINE 0 0 0
>> c5t1d0 FAULTED 0 0 0 corrupted data
>> c5t1d0 does indeed only appear once in the "format" list. I wonder how to go
>> about correcting this if I can't uniquely identify the failing drive.
>> "format" takes forever to spill its guts, and the zpool commands all
>> hang...... clearly there is hardware error here, probably causing that, but
>> not sure how to identify which disk to pull.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
