On 8/11/10 10:21 AM, Jeff Bacon wrote:
Wow, sounds familiar - binderedondat. I thought it was just when using
expanders... guess it's just anything 1068-based. Lost a 20TB pool to
having the controller basically just hose up what it was doing and write
scragged data to the disk.
1) The suggestion using the serial number of the drive to trace back to
what's connected to what is good, assuming you can pull drives to look
at their serial numbers.
Except that this could cause more problems if you happen to
pull the wrong one in the middle of a resilver operation. Or
anything, really.
2) One thing I've done over the years is, given that I often use the
same motherboards, is physically map out the PCI slot addresses -
/dev/cfg/c2 ../../devices/p...@0,0/pci8086,3...@3/pci1000,3...@0:scsi
/dev/cfg/c3
../../devices/p...@7a,0/pci8086,3...@5/pci1000,3...@0/ip...@f:scsi
/dev/cfg/c4
../../devices/p...@7a,0/pci8086,3...@5/pci1000,3...@0/ip...@v0:scsi
/dev/cfg/c5
../../devices/p...@7a,0/pci8086,3...@7/pci1000,3...@0/ip...@f:scsi
/dev/cfg/c6
../../devices/p...@7a,0/pci8086,3...@7/pci1000,3...@0/ip...@v0:scsi
/dev/cfg/c7 ../../devices/p...@7a,0/pci8086,3...@9/pci1000,3...@0:scsi
^^^^^^^^^^^^^^^ this part will
correspond with a physical slot
^^^^^^^^^ if you have a SM dual-IOH board,
these represent the two IOH-36s
^ on a single-IOH board,
I've noted that this often
corresponds to the
physical slot number (it's "unit-address" from DDI)
So far, it's involved "stick a card in a slot/reboot/reconfig/see what
address it's at/note it down" or other forms of reverse engineering.
Handy to have occasionally. If you're doing a BYO, taking the time up
front to figure this out is a Good Idea.
This is what FMA's libtopo solves for you. Fairly well, too:
# /usr/lib/fm/fmd/fmtopo -V
....
hc://:product-id=Sun-Ultra-40-M2-Workstation:server-id=blinder:chassis-id=0802FMY00N/motherboard=0/hostbridge=1/pciexrc=1/pciexbus=2/pciexdev=0/pciexfn=0
group: protocol version: 1 stability: Private/Private
resource fmri
hc://:product-id=Sun-Ultra-40-M2-Workstation:server-id=blinder:chassis-id=0802FMY00N/motherboard=0/hostbridge=1/pciexrc=1/pciexbus=2/pciexdev=0/pciexfn=0
label string PCIE0 Slot
FRU fmri
hc://:product-id=Sun-Ultra-40-M2-Workstation:server-id=blinder:chassis-id=0802FMY00N/motherboard=0/hostbridge=1/pciexrc=1/pciexbus=2/pciexdev=0
ASRU fmri dev:////p...@0,0/pci10de,3...@a/pci1000,3...@0
group: authority version: 1 stability: Private/Private
product-id string Sun-Ultra-40-M2-Workstation
chassis-id string 0802FMY00N
server-id string blinder
group: io version: 1 stability: Private/Private
dev string /p...@0,0/pci10de,3...@a/pci1000,3...@0
driver string mpt
module fmri mod:///mod-name=mpt/mod-id=57
group: pci version: 1 stability: Private/Private
device-id string 58
extended-capabilities string pciexdev
class-code string 10000
vendor-id string 1000
assigned-addresses uint32[] [ 2164391952 0 16384 0 256 2197946388 0
2686517248 0 16384 2197946396 0 2686451712 0 65536 ]
Note the label property.
3) Get a copy of "lsiutil" for solaris (available from LSI's site) -
it's an easy way to check out the controller and see if it's there or
whether it sees the drives or what.
(There is a newer version of lsiutil that supports the 2008s...
strangely, it's not available from the LSI site. Their tech support
didn't even know it existed when I asked. I got my copy off someone on
hardforum.)
Sigh. Sun (and now Oracle) didn't distribute lsiutil due at least in
part to the likelihood of customers killing their hbas. Which, having
used lsiutil (to recover from failed operations) is depressingly easy.
The replacement is sas2ircu. I believe the same caveats apply.
4) Things you didn't want to know: the LSI1068 actually has a very small
write cache on board. So if you manage a certain set of situations
(namely, setting the "device i/o timeout" in the BIOS to something other
than 0, then having a SATA drive blow up in a certain way such that it
hangs for longer than the timeout you set), the mpt driver (it seems)
can get impatient and re-initialize the controller, or that's what it
looks like. Great way to scrag a volume. :(
The 1068 also has a limitation of 122 devices for its logical target-id
concept. But we don't talk about that in polite company :-)
Please, go and have a poke around the output from libtopo. I think you'll
be pleasantly surprised at what you can discover with it.
McB
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
