Hi Alex, If the hardware and cables were moved around then this is probably the root cause of your problem. You should see if you can move the devices/cabling back to what they were before the move.
The zpool history output provides the original device name, which isn't c5t1d0, either: # zpool create tank c13t0d0 You might grep the zpool history output to find out which disk was eventually attached, like this: # zpool history | grep attach But its clear from the zdb -l output, that the devid for this particular device changed, which we've seen happen on some hardware. If the devid persists, ZFS can follow the devid of the device even if its physical path changes and is able to recover more gracefully. If you continue to use this hardware for your storage pool, you should export the pool before making any kind of hardware change. Thanks, Cindy On 05/21/11 18:05, Alex Dolski wrote:
Hi Cindy, Thanks for the advice. This is just a little old Gateway PC provisioned as an informal workgroup server. The main storage is two SATA drives in an external enclosure, connected to a Sil3132 PCIe eSATA controller. The OS is snv_134b, upgraded from snv_111a. I can't identify a cause in particular. The box has been running for several months without much oversight. It's possible that the two eSATA cables got reconnected to different ports after a recent move. The backup has been made and I will try the export & import, per your advice (if zpool command works - it does again at the moment, no reboot!). I will also try switching the eSATA cables to opposite ports. Thanks, Alex Command output follows: # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c5t1d0 <ATA-WDC WD5000AAKS-0-1D05-465.76GB> /pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0 1. c8d0 <DEFAULT cyl 9726 alt 2 hd 255 sec 63> /pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0 2. c9d0 <DEFAULT cyl 38910 alt 2 hd 255 sec 63> /pci@0,0/pci-ide@1f,2/ide@1/cmdk@0,0 3. c11t0d0 <WD-Ext HDD 1021-2002-931.51GB> /pci@0,0/pci107b,5058@1a,7/storage@1/disk@0,0 # zpool history tank History for 'tank': 2010-06-18.15:14:16 zpool create tank c13t0d0 2011-05-07.02:00:07 zpool scrub tank 2011-05-14.02:00:08 zpool scrub tank 2011-05-21.02:00:12 zpool scrub tank <a million 'zfs snapshot' and 'zfs destroy' events from zfs-auto-snap omitted> # zdb -l /dev/dsk/c5t1d0s0 -------------------------------------------- LABEL 0 -------------------------------------------- version: 14 name: 'tank' state: 0 txg: 3374337 pool_guid: 6242690959503408617 hostid: 8697169 hostname: 'wdssandbox' top_guid: 17982590661103377266 guid: 1717308203478351258 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 17982590661103377266 whole_disk: 0 metaslab_array: 23 metaslab_shift: 32 ashift: 9 asize: 500094468096 is_log: 0 children[0]: type: 'disk' id: 0 guid: 1717308203478351258 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 27 children[1]: type: 'disk' id: 1 guid: 9267693216478869057 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 893 -------------------------------------------- LABEL 1 -------------------------------------------- version: 14 name: 'tank' state: 0 txg: 3374337 pool_guid: 6242690959503408617 hostid: 8697169 hostname: 'wdssandbox' top_guid: 17982590661103377266 guid: 1717308203478351258 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 17982590661103377266 whole_disk: 0 metaslab_array: 23 metaslab_shift: 32 ashift: 9 asize: 500094468096 is_log: 0 children[0]: type: 'disk' id: 0 guid: 1717308203478351258 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 27 children[1]: type: 'disk' id: 1 guid: 9267693216478869057 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 893 -------------------------------------------- LABEL 2 -------------------------------------------- version: 14 name: 'tank' state: 0 txg: 3374337 pool_guid: 6242690959503408617 hostid: 8697169 hostname: 'wdssandbox' top_guid: 17982590661103377266 guid: 1717308203478351258 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 17982590661103377266 whole_disk: 0 metaslab_array: 23 metaslab_shift: 32 ashift: 9 asize: 500094468096 is_log: 0 children[0]: type: 'disk' id: 0 guid: 1717308203478351258 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 27 children[1]: type: 'disk' id: 1 guid: 9267693216478869057 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 893 -------------------------------------------- LABEL 3 -------------------------------------------- version: 14 name: 'tank' state: 0 txg: 3374337 pool_guid: 6242690959503408617 hostid: 8697169 hostname: 'wdssandbox' top_guid: 17982590661103377266 guid: 1717308203478351258 vdev_children: 1 vdev_tree: type: 'mirror' id: 0 guid: 17982590661103377266 whole_disk: 0 metaslab_array: 23 metaslab_shift: 32 ashift: 9 asize: 500094468096 is_log: 0 children[0]: type: 'disk' id: 0 guid: 1717308203478351258 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1939879/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 27 children[1]: type: 'disk' id: 1 guid: 9267693216478869057 path: '/dev/dsk/c5t1d0s0' devid: 'id1,sd@SATA_____WDC_WD5000AAKS-0_____WD-WCAWF1769949/a' phys_path: '/pci@0,0/pci8086,2845@1c,3/pci1095,3132@0/disk@1,0:a' whole_disk: 1 DTL: 893 On May 20, 2011, at 8:34 AM, Cindy Swearingen wrote:Hi Alex More scary than interesting to me. What kind of hardware and which Solaris release? Do you know what steps lead up to this problem? Any recent hardware changes? This output should tell you which disks were in this pool originally: # zpool history tank If the history identifies tank's actual disks, maybe you can determine which disk is masquerading as c5t1d0. If that doesn't work, accessing the individual disk entries in format should tell which one is the problem, if its only one. I would like to see the output of this command: # zdb -l /dev/dsk/c5t1d0s0 Make sure you have a good backup of your data. If you need to pull a disk to check cabling, or rule out controller issues, you should probably export this pool first. Have a good backup. Others have resolved minor device issues by exporting/importing the pool but with format/zpool commands hanging on your system, I'm not confident that this operation will work for you. Thanks, Cindy On 05/19/11 12:17, Alex wrote:I thought this was interesting - it looks like we have a failing drive in our mirror, but the two device nodes in the mirror are the same: pool: tank state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: scrub completed after 1h9m with 0 errors on Sat May 14 03:09:45 2011 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 c5t1d0 ONLINE 0 0 0 c5t1d0 FAULTED 0 0 0 corrupted data c5t1d0 does indeed only appear once in the "format" list. I wonder how to go about correcting this if I can't uniquely identify the failing drive. "format" takes forever to spill its guts, and the zpool commands all hang...... clearly there is hardware error here, probably causing that, but not sure how to identify which disk to pull.
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
