Re: [zfs-discuss] Any rhyme or reason to disk dev names?
Dear Matthew, Some methods were not mentioned that have been discussed on zfs-discuss@ in the past several times. Useful for list archives at least: . One easy way to detect disk locations is to make the lights blink if you have lights per drive in your chassis. But it does not scale. For example something similar to dd if=/dev/dsk/c0t5000CCA228C0E488d0 of=/dev/null . Other methods have been discussed (check the archives), for example: http://mail.opensolaris.org/pipermail/zfs-discuss/2011-June/048982.html Quoting one example from above post by Richard: > For the later Nexenta, OpenSolaris or Solaris 11 Express releases the mpt_sas driver will try to light the > OK2RM (ok to remove) LED for a disk when you use cfgadm to disconnect the paths. Apparently this also > works for SATA disks in an enclosure that manages SATA disks. The process is documented very nicely > by Cindy in the ZFS Admin Guide. However, there are a number of enclosures that do not have an OK2RM > LED. YMMV. If you got other good pointers off-list, may I suggest to make a summary for the list as this question pops up often. Regards, Ville From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Matthew R. Wilson Sent: Thursday, December 22, 2011 1:39 AM To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] Any rhyme or reason to disk dev names? Thank you for all of the good pointers, everyone. croinfo and diskinfo don't give me any output, but that's not surprising since this is a home-built system. But it's good to know those utilities exist for production hardware. Making the association between the disk serial number and target number by matching them up in the iostat -En or prtconf output looks like it will work for me. Thanks again! -Matthew On Wed, Dec 21, 2011 at 8:21 AM, Garrett D'Amore wrote: On Dec 21, 2011, at 3:14 AM, James C. McPherson wrote: > On 21/12/11 05:58 PM, Matthew R. Wilson wrote: >> Hello, >> >> I am curious to know if there is an easy way to guess or identify the >> device names of disks. Previously the /dev/dsk/c0t0d0s0 system made sense >> to me... I had a SATA controller card with 8 ports, and they showed up >> with the numbers 1-8 in the "t" position of the device name. >> >> But I just built a new system with two LSI SAS HBAs in it, and my device >> names are along the lines of: >> /dev/dsk/c0t5000CCA228C0E488d0 >> >> I could not find any correlation between that identifier and the a) >> controller the disk was plugged in to, or b) the port number on the >> controller. The only way I could make a mapping of device name to >> controller port was to add one drive at a time, reboot the system, and run >> "format" to see which new disk name shows up. >> >> I'm guessing there's a better way, but I can't find any obvious answer as >> to how to determine which port on my LSI controller card will correspond >> with which seemingly random device name. Can anyone offer any suggestions >> on a way to predict the device naming, or at least get the system to list >> the disks after I insert one without rebooting? > > Hi Matthew, > By default the names for disks attached via mpt_sas(7d), or > mpt(7d) if your disks are new enough, is to use their WWN > as reported in the SCSI INQUIRY Page83 response. > > The old paradigm you refer to is based on the physical id > of the device on a parallel SCSI bus. That doesn't scale > with SAS, and is something we're trying to move away from. More to the point, on SAS and other similar busses, there simply *isn't* such a thing as a simple target number. The old numbering scheme from parallel SCSI was suitable when you could have only 7 or 15 or so devices on a single bus. With modern busses you can have many thousands of devices on the same fabric. So we address them by WWN. - Garrett > > If you'd like some info about how we use devids and guids, > please refer to my presentation > > http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf > > > For your particular configuration, if you note the serial > number and WWN of the device before you insert them, you > can match that up with info from iostat -En and/or prtconf -v. > > > hth, > James C. McPherson > -- > Oracle > http://www.jmcp.homeunix.com/blog > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Corrupt Array
Hi guys, after a scrub my raidz array status showed: # zpool status pool: pool state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scan: scrub repaired 85.5K in 1h21m with 0 errors on Mon Dec 19 06:24:25 2011 config: NAMESTATE READ WRITE CKSUM poolONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 ad18ONLINE 0 0 1 ad19ONLINE 0 0 0 ad10ONLINE 0 0 1 ad4 ONLINE 0 0 0 errors: No known data errors I assume the checksum counts are current and irreconcilable. (Why does the scan say 'repaired with 0 errors' then?). What should one do at this point? I rebooted and ran another scrub, this time it came up with 0 errors and 0 checksum counts, what does that mean? I then backed up the array, kicked out ad18 and resilvered it from scratch: # zpool status pool: pool state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: resilvered 218G in 1h25m with 14 errors on Wed Dec 21 14:48:47 2011 config: NAME STATE READ WRITE CKSUM pool DEGRADED 0 014 raidz1-0 DEGRADED 0 028 replacing-0 OFFLINE 0 0 0 ad18/old OFFLINE 0 0 0 ad18 ONLINE 0 0 0 ad19 ONLINE 0 0 0 ad10 ONLINE 0 0 0 ad4 ONLINE 0 0 0 errors: 11 data errors, use '-v' for a list and 'zpool status -v' gives me a list of affected files. I assume I delete those files, then follow the same procedure on ad10? # uname -a FreeBSD file 8.2-STABLE FreeBSD 8.2-STABLE #0: Sat Nov 12 17:51:22 SAST 2011 root@file:/usr/obj/usr/src/sys/COWNEL amd64 ZFS filesystem version 5 ZFS storage pool version 28 ps. I did get 1 disk alert in the logs during this whole process, half an hour before resilvering: Dec 21 12:41:48 file kernel: ad10: WARNING - READ_DMA48 UDMA ICRC error (retrying request) LBA=306763504 Dec 21 12:41:48 file kernel: ad10: FAILURE - READ_DMA48 status=51 error=10 LBA=306763504 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
Thank you for all of the good pointers, everyone. croinfo and diskinfo don't give me any output, but that's not surprising since this is a home-built system. But it's good to know those utilities exist for production hardware. Making the association between the disk serial number and target number by matching them up in the iostat -En or prtconf output looks like it will work for me. Thanks again! -Matthew On Wed, Dec 21, 2011 at 8:21 AM, Garrett D'Amore wrote: > > On Dec 21, 2011, at 3:14 AM, James C. McPherson wrote: > > > On 21/12/11 05:58 PM, Matthew R. Wilson wrote: > >> Hello, > >> > >> I am curious to know if there is an easy way to guess or identify the > >> device names of disks. Previously the /dev/dsk/c0t0d0s0 system made > sense > >> to me... I had a SATA controller card with 8 ports, and they showed up > >> with the numbers 1-8 in the "t" position of the device name. > >> > >> But I just built a new system with two LSI SAS HBAs in it, and my device > >> names are along the lines of: > >> /dev/dsk/c0t5000CCA228C0E488d0 > >> > >> I could not find any correlation between that identifier and the a) > >> controller the disk was plugged in to, or b) the port number on the > >> controller. The only way I could make a mapping of device name to > >> controller port was to add one drive at a time, reboot the system, and > run > >> "format" to see which new disk name shows up. > >> > >> I'm guessing there's a better way, but I can't find any obvious answer > as > >> to how to determine which port on my LSI controller card will correspond > >> with which seemingly random device name. Can anyone offer any > suggestions > >> on a way to predict the device naming, or at least get the system to > list > >> the disks after I insert one without rebooting? > > > > Hi Matthew, > > By default the names for disks attached via mpt_sas(7d), or > > mpt(7d) if your disks are new enough, is to use their WWN > > as reported in the SCSI INQUIRY Page83 response. > > > > The old paradigm you refer to is based on the physical id > > of the device on a parallel SCSI bus. That doesn't scale > > with SAS, and is something we're trying to move away from. > > More to the point, on SAS and other similar busses, there simply *isn't* > such a thing as a simple target number. The old numbering scheme from > parallel SCSI was suitable when you could have only 7 or 15 or so devices > on a single bus. With modern busses you can have many thousands of devices > on the same fabric. So we address them by WWN. > >- Garrett > > > > If you'd like some info about how we use devids and guids, > > please refer to my presentation > > > > http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf > > > > > > For your particular configuration, if you note the serial > > number and WWN of the device before you insert them, you > > can match that up with info from iostat -En and/or prtconf -v. > > > > > > hth, > > James C. McPherson > > -- > > Oracle > > http://www.jmcp.homeunix.com/blog > > ___ > > zfs-discuss mailing list > > zfs-discuss@opensolaris.org > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
On Dec 21, 2011, at 3:14 AM, James C. McPherson wrote: > On 21/12/11 05:58 PM, Matthew R. Wilson wrote: >> Hello, >> >> I am curious to know if there is an easy way to guess or identify the >> device names of disks. Previously the /dev/dsk/c0t0d0s0 system made sense >> to me... I had a SATA controller card with 8 ports, and they showed up >> with the numbers 1-8 in the "t" position of the device name. >> >> But I just built a new system with two LSI SAS HBAs in it, and my device >> names are along the lines of: >> /dev/dsk/c0t5000CCA228C0E488d0 >> >> I could not find any correlation between that identifier and the a) >> controller the disk was plugged in to, or b) the port number on the >> controller. The only way I could make a mapping of device name to >> controller port was to add one drive at a time, reboot the system, and run >> "format" to see which new disk name shows up. >> >> I'm guessing there's a better way, but I can't find any obvious answer as >> to how to determine which port on my LSI controller card will correspond >> with which seemingly random device name. Can anyone offer any suggestions >> on a way to predict the device naming, or at least get the system to list >> the disks after I insert one without rebooting? > > Hi Matthew, > By default the names for disks attached via mpt_sas(7d), or > mpt(7d) if your disks are new enough, is to use their WWN > as reported in the SCSI INQUIRY Page83 response. > > The old paradigm you refer to is based on the physical id > of the device on a parallel SCSI bus. That doesn't scale > with SAS, and is something we're trying to move away from. More to the point, on SAS and other similar busses, there simply *isn't* such a thing as a simple target number. The old numbering scheme from parallel SCSI was suitable when you could have only 7 or 15 or so devices on a single bus. With modern busses you can have many thousands of devices on the same fabric. So we address them by WWN. - Garrett > > If you'd like some info about how we use devids and guids, > please refer to my presentation > > http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf > > > For your particular configuration, if you note the serial > number and WWN of the device before you insert them, you > can match that up with info from iostat -En and/or prtconf -v. > > > hth, > James C. McPherson > -- > Oracle > http://www.jmcp.homeunix.com/blog > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
On 2011-12-21 09:22, v...@bb-c.de wrote: I am curious to know if there is an easy way to guess or identify the device names of disks. Have a look at the file /etc/path_to_inst. There you will find all device instances managed by a particular driver. The first entry of each line is the physical device. If you then look in /dev/rdsk and check which symbolic link of the form ctds points to this physical device, you have your match. One caveat is that if you move disks around, /etc/path_to_inst will grow, and there is no guarantee that any device listed in this file is really present in the running system. HTH -- Volker Take a look at 'diskinfo' - should be in Solaris 10U10 and Solaris 11. Cheers, Henrik ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
On Dec 21, 2011, at 2:58, "Matthew R. Wilson" wrote: > Can anyone offer any suggestions on a way to predict the device naming, or at > least get the system to list the disks after I insert one without rebooting? You have gotten some good responses that should help you out. However, you shouldn't have to reboot to see the new disks try devfsadm. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
On Wed, Dec 21, 2011 at 1:58 AM, Matthew R. Wilson wrote: > Hello, > > I am curious to know if there is an easy way to guess or identify the device > names of disks. Previously the /dev/dsk/c0t0d0s0 system made sense to me... > I had a SATA controller card with 8 ports, and they showed up with the > numbers 1-8 in the "t" position of the device name. > > But I just built a new system with two LSI SAS HBAs in it, and my device > names are along the lines of: > /dev/dsk/c0t5000CCA228C0E488d0 > > I could not find any correlation between that identifier and the a) > controller the disk was plugged in to, or b) the port number on the > controller. The only way I could make a mapping of device name to controller > port was to add one drive at a time, reboot the system, and run "format" to > see which new disk name shows up. > > I'm guessing there's a better way, but I can't find any obvious answer as to > how to determine which port on my LSI controller card will correspond with > which seemingly random device name. Can anyone offer any suggestions on a > way to predict the device naming, or at least get the system to list the > disks after I insert one without rebooting? Depending on the hardware you are using, you may be able to benefit from croinfo. $ croinfo D:devchassis-path t:occupant-type c:occupant-compdev - --- - /dev/chassis//SYS/SASBP/HDD0/disk disk c0t5000CCA012B66E90d0 /dev/chassis//SYS/SASBP/HDD1/disk disk c0t5000CCA012B68AC8d0 The text in the left column represents text that should be printed on the corresponding disk slots. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
On 21/12/11 05:58 PM, Matthew R. Wilson wrote: Hello, I am curious to know if there is an easy way to guess or identify the device names of disks. Previously the /dev/dsk/c0t0d0s0 system made sense to me... I had a SATA controller card with 8 ports, and they showed up with the numbers 1-8 in the "t" position of the device name. But I just built a new system with two LSI SAS HBAs in it, and my device names are along the lines of: /dev/dsk/c0t5000CCA228C0E488d0 I could not find any correlation between that identifier and the a) controller the disk was plugged in to, or b) the port number on the controller. The only way I could make a mapping of device name to controller port was to add one drive at a time, reboot the system, and run "format" to see which new disk name shows up. I'm guessing there's a better way, but I can't find any obvious answer as to how to determine which port on my LSI controller card will correspond with which seemingly random device name. Can anyone offer any suggestions on a way to predict the device naming, or at least get the system to list the disks after I insert one without rebooting? Hi Matthew, By default the names for disks attached via mpt_sas(7d), or mpt(7d) if your disks are new enough, is to use their WWN as reported in the SCSI INQUIRY Page83 response. The old paradigm you refer to is based on the physical id of the device on a parallel SCSI bus. That doesn't scale with SAS, and is something we're trying to move away from. If you'd like some info about how we use devids and guids, please refer to my presentation http://www.jmcp.homeunix.com/~jmcp/WhatIsAGuid.pdf For your particular configuration, if you note the serial number and WWN of the device before you insert them, you can match that up with info from iostat -En and/or prtconf -v. hth, James C. McPherson -- Oracle http://www.jmcp.homeunix.com/blog ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Any rhyme or reason to disk dev names?
> I am curious to know if there is an easy way to guess or identify > the device names of disks. Have a look at the file /etc/path_to_inst. There you will find all device instances managed by a particular driver. The first entry of each line is the physical device. If you then look in /dev/rdsk and check which symbolic link of the form ctds points to this physical device, you have your match. One caveat is that if you move disks around, /etc/path_to_inst will grow, and there is no guarantee that any device listed in this file is really present in the running system. HTH -- Volker -- Volker A. Brandt Consulting and Support for Oracle Solaris Brandt & Brandt Computer GmbH WWW: http://www.bb-c.de/ Am Wiesenpfad 6, 53340 Meckenheim, GERMANYEmail: v...@bb-c.de Handelsregister: Amtsgericht Bonn, HRB 10513 Schuhgröße: 46 Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt "When logic and proportion have fallen sloppy dead" ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
