Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Did anybody ever find out if this option worked? Does setting this hidden option mean that drives are always listed and named according to the order they are physically connected? Ross > Hi Kent: > > So, in using the lsiutil utility, what do I find, > but the following option: (this was under the hidden option 15) > 12. Change (enable/disable) persistence > > I have not had a chance to try it, but it says > the default is enabled. Let me know if you try it. > > Paul This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Hi Kent: So, in using the lsiutil utility, what do I find, but the following option: (this was under the hidden option 15) 12. Change (enable/disable) persistence I have not had a chance to try it, but it says the default is Enabled. Let me know if you try it. Paul Kent Watsen wrote: > > > > Paul Jochum wrote: >>What the lsiutil does for me is clear the persistent mapping for >> all of the drives on a card. > Since James confirms that I'm doomed to ad hoc methods tracking > device-ids to bays, I'm interested in knowing if somehow your ability > to clear the persistent mapping for *all* of the drives on the card > somehow gets you to a stable-state? Seems to me that, if the tool is > run from the BIOS, like the LSI Configuration Utility, then it kind > defeats the uptime that I'm trying to achieve by having a RAID system > in the first place... Is it, by chance, a user land program? > >> I don't know of a way to disable the mapping completely (but that >> does sound like a nice option). Since SUN is reselling this card now >> (that is how I got my cards), I wonder if they can put in a request >> to LSI to provide this enhancement? > Yes, please! - can someone from SUN please request LSI to add a > feature to selectively disable their card's persistent drive mapping > feature? > > > Thanks, > Kent > > > > > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Paul Jochum wrote: >What the lsiutil does for me is clear the persistent mapping for > all of the drives on a card. Since James confirms that I'm doomed to ad hoc methods tracking device-ids to bays, I'm interested in knowing if somehow your ability to clear the persistent mapping for *all* of the drives on the card somehow gets you to a stable-state? Seems to me that, if the tool is run from the BIOS, like the LSI Configuration Utility, then it kind defeats the uptime that I'm trying to achieve by having a RAID system in the first place... Is it, by chance, a user land program? > I don't know of a way to disable the mapping completely (but that does > sound like a nice option). Since SUN is reselling this card now (that > is how I got my cards), I wonder if they can put in a request to LSI > to provide this enhancement? Yes, please! - can someone from SUN please request LSI to add a feature to selectively disable their card's persistent drive mapping feature? Thanks, Kent ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Kent Watsen wrote:
> Eric Schrock wrote:
>> For x86 systems, you can use ipmitool to manipulate the led state
>> (ipmitool sunoem led ...). On older galaxy systems, you can only set the
>> fail LED ('io.hdd0.led'), as the ok2rm LED is not physically connected
>> to anything. On newer systems, you can set both the 'fail' and 'okrm'
>> LEDs. You cannot change the activity LED except by manually sending the
>> 'set sensor reading' IPMI command (not available via impitool).
>>
>> For external enclosures, you'll need a SES control program.
>>
>> Both of these problems are being worked on under the FMA sensor
>> framework to create a unified view through libtopo. Until that's
>> complete, you'll be stuck using ad hoc methods.
>>
> Hi Eric,
>
> I've looked at your blog and have tried your suggestions, but I need a
> little more advice.
>
> I am on an x86 system running SXCE svn_74 - the system has 6 SAS
> backplanes but, according to the integrator, no "real" scsi enclosure
> services. according to the man page, I found that I could use `sdr
> elist generic` to list LEDs, but that command doesn't return any output:
>
> # ipmitool sdr elist generic
> # /* there was no output */
>
> Is it not returning sensor ids because I don't have real scsi enclosure
> services? Is there anything I can do or am I doomed to ad hoc methods
> forever?
The version of ipmitool which we're shipping with snv
has this comment at the start of libipmi.h ::
(usr/src/lib/libipmi/common/libipmi.h)
/*
* Private interfaces for communicating with attached services over IPMI. This
* library is designed for system software communicating with Sun-supported
* service processors over /dev/bmc. It is not a generic IPMI library.
*
* Documentation references refer to "Intelligent Platform Management Interface
* Specification Second Generation v2.0", document revision 1.0 with Februrary
* 15, 2006 Markup from "IPMI v2.0 Addenda, Errata, and Clarifications Revision
* 3".
*/
Further, unless you have a /dev/bmc device you won't get
anything back from the shipped ipmitool.
Unfortunately that means you are stuck - for the moment
at least - with ad hoc measures.
James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp http://www.jmcp.homeunix.com/blog
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Eric Schrock wrote:
> For x86 systems, you can use ipmitool to manipulate the led state
> (ipmitool sunoem led ...). On older galaxy systems, you can only set the
> fail LED ('io.hdd0.led'), as the ok2rm LED is not physically connected
> to anything. On newer systems, you can set both the 'fail' and 'okrm'
> LEDs. You cannot change the activity LED except by manually sending the
> 'set sensor reading' IPMI command (not available via impitool).
>
> For external enclosures, you'll need a SES control program.
>
> Both of these problems are being worked on under the FMA sensor
> framework to create a unified view through libtopo. Until that's
> complete, you'll be stuck using ad hoc methods.
>
Hi Eric,
I've looked at your blog and have tried your suggestions, but I need a
little more advice.
I am on an x86 system running SXCE svn_74 - the system has 6 SAS
backplanes but, according to the integrator, no "real" scsi enclosure
services. according to the man page, I found that I could use `sdr
elist generic` to list LEDs, but that command doesn't return any output:
# ipmitool sdr elist generic
# /* there was no output */
Is it not returning sensor ids because I don't have real scsi enclosure
services? Is there anything I can do or am I doomed to ad hoc methods
forever?
Thanks,
Kent
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Kent Watsen wrote: > Given that manually tracking shifting ids doesn't sound appealing to > me, would using a SATA controller like the AOC-SAT2-MV8 resolve the > issue? Given that I currently only have one LSI HBA - I'd need to get 2 > more for all 24 drives ---or--- I could get 3 of these SATA controllers > plus 6 discrete-to-8087 reverse breakout cables. Going down the > LSI-route would cost about $600 while going down the AOC-SAT2-MV8 would > cost about $400. I understand that the SATA controllers are less > performant, but I'd gladly exchange some performance that I'm likely to > never need to simplify my administrative overhead... > So, I picked up an AOC-SAT2-MV8 off eBay for not too much and then I got a 4xSATA to one SFF-8087 cable to connect it to one one my six backplanes. But, as fortune would have it, the cable I bought has SATA connectors that are physically too big to plug into the AOC-SAT2-MV8 - since the AOC-SAT2-MV8 stacks two SATA connectors on top of each other... Any chance anyone knows of a 4xSATA to SFF-8087 reverse breakout cable that will plug into the AOC-SAT2-MV8? Alternatively, anyone know how hard it would be to splice the SATA cables that came with the AOC-SAT2-MV8 into the cable I bought? Also, the cable I got actually had 5 cables going into the SFF-8087 - 4 SATA and another that is rectangular and has holes for either 7 or 8 pins (not sure if its for 7 or 8 pins as one of the holes appears a bit different). Any idea what this fifth cable is for? Thanks, Kent ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
On Wed, Dec 12, 2007 at 09:22:07AM -0800, Richard Elling wrote:
>
> For Sun systems, we have 3 LEDs on the drives:
> 1. Ready to remove (blue)
> 2. Service required (amber)
> 3. OK/Activity (green)
>
> So there must be a way to set the ready to remove LED from Solaris.
> In the old days, we could use luxadm(1m). Does that still work, or is
> there some new equivalent?
>
For x86 systems, you can use ipmitool to manipulate the led state
(ipmitool sunoem led ...). On older galaxy systems, you can only set the
fail LED ('io.hdd0.led'), as the ok2rm LED is not physically connected
to anything. On newer systems, you can set both the 'fail' and 'okrm'
LEDs. You cannot change the activity LED except by manually sending the
'set sensor reading' IPMI command (not available via impitool).
For external enclosures, you'll need a SES control program.
Both of these problems are being worked on under the FMA sensor
framework to create a unified view through libtopo. Until that's
complete, you'll be stuck using ad hoc methods.
- Eric
--
Eric Schrock, FishWorkshttp://blogs.sun.com/eschrock
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
James C. McPherson wrote: > > Now here's where things get murky. > > At this point in time at least (it may change!) Solaris' mpt > driver uses LSI's logical target id mapping method. This is > *NOT* an enclosure/slot naming method - at least, not from the > OS' point of view. Additionally, unless you're using an actual > real SCSI Enclosure Services (ses) device, there's no enclosure > to provide enclosure/slot mapping with either. > > Since mpt uses logical target id, therefore the target id which > Solaris sees _will definitely change_ if you swap a disk. > JMCP, For Sun systems, we have 3 LEDs on the drives: 1. Ready to remove (blue) 2. Service required (amber) 3. OK/Activity (green) So there must be a way to set the ready to remove LED from Solaris. In the old days, we could use luxadm(1m). Does that still work, or is there some new equivalent? -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Hi Kent: What the lsiutil does for me is clear the persistent mapping for all of the drives on a card. I don't know of a way to disable the mapping completely (but that does sound like a nice option). Since SUN is reselling this card now (that is how I got my cards), I wonder if they can put in a request to LSI to provide this enhancement? Paul Kent Watsen wrote: > > Hi Paul, > > Already in my LSI Configuration Utility I have an option to clear the > persistent mapping for drives not present, but then the card resumes > its normal persistent-mapping logic. What I really want is to disable > to persistent mapping logic completely - is the `lsiutil` doing that > for you? > > Thanks, > Kent > > > > Paul Jochum wrote: >> Hi Kent: >> >> I have run into the same problem before, and have worked with LSI and >> SUN support to fix it. LSI calls this "persistant drive mapping", >> and here is how to clear it >> >> 1) obtain the latest version of the program "lsiutil" from LSI. They >> don't seem to have the Solaris versions on their website, but I got >> it by email when entering a ticket into their support system. I know >> that they have a version for Solaris x86 (and I believe a Sparc >> version also). The version I currently have is: "LSI Logic MPT >> Configuration Utility, Version 1.52, September 7, 2007" >> >> 2) Execute the lsiutil program on your target box. >>a) first it will ask you to select which card to use (I have >> multiple cards in my machine, don't know if it will ask if you only >> have 1 card in your box) >> b) then you need to select option 15 (it is a hidden option, not >> shown on the menu) >> c) then you select option 10 (Clear all persistant mappings) >> d) then option 0 multiple times to get out of the program >> e) I normally than reboot the box, and the next time it comes up, >> the drives are back in order. >> e) or (instead of rebooting) option 99, to reset the chip (causes >> new mappings to be established), then option 8 (to verify lower >> target IDs), then "devfsadm". After devfsadm completes, lsiutil >> option 42 should display valid device names (in /dev/rdsk), and >> "format" should find the devices so that you can label them. >> >> Hope this helps. I happened to need it last night again (I normally >> have to run it after re-imaging a box, assuming that I don't want to >> save the data that was on those drives). >> >> Paul Jochum >> >> >> This message posted from opensolaris.org >> ___ >> zfs-discuss mailing list >> zfs-discuss@opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> >> ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Hi Paul, Already in my LSI Configuration Utility I have an option to clear the persistent mapping for drives not present, but then the card resumes its normal persistent-mapping logic. What I really want is to disable to persistent mapping logic completely - is the `lsiutil` doing that for you? Thanks, Kent Paul Jochum wrote: > Hi Kent: > > I have run into the same problem before, and have worked with LSI and SUN > support to fix it. LSI calls this "persistant drive mapping", and here is > how to clear it > > 1) obtain the latest version of the program "lsiutil" from LSI. They don't > seem to have the Solaris versions on their website, but I got it by email > when entering a ticket into their support system. I know that they have a > version for Solaris x86 (and I believe a Sparc version also). The version I > currently have is: "LSI Logic MPT Configuration Utility, Version 1.52, > September 7, 2007" > > 2) Execute the lsiutil program on your target box. >a) first it will ask you to select which card to use (I have multiple > cards in my machine, don't know if it will ask if you only have 1 card in > your box) > b) then you need to select option 15 (it is a hidden option, not shown on > the menu) > c) then you select option 10 (Clear all persistant mappings) > d) then option 0 multiple times to get out of the program > e) I normally than reboot the box, and the next time it comes up, the > drives are back in order. > e) or (instead of rebooting) option 99, to reset the chip (causes new > mappings to be established), then option 8 (to verify lower target IDs), then > "devfsadm". After devfsadm completes, lsiutil option 42 should display valid > device names (in /dev/rdsk), and "format" should find the devices so that you > can label them. > > Hope this helps. I happened to need it last night again (I normally have to > run it after re-imaging a box, assuming that I don't want to save the data > that was on those drives). > > Paul Jochum > > > This message posted from opensolaris.org > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Hi Kent: I have run into the same problem before, and have worked with LSI and SUN support to fix it. LSI calls this "persistant drive mapping", and here is how to clear it 1) obtain the latest version of the program "lsiutil" from LSI. They don't seem to have the Solaris versions on their website, but I got it by email when entering a ticket into their support system. I know that they have a version for Solaris x86 (and I believe a Sparc version also). The version I currently have is: "LSI Logic MPT Configuration Utility, Version 1.52, September 7, 2007" 2) Execute the lsiutil program on your target box. a) first it will ask you to select which card to use (I have multiple cards in my machine, don't know if it will ask if you only have 1 card in your box) b) then you need to select option 15 (it is a hidden option, not shown on the menu) c) then you select option 10 (Clear all persistant mappings) d) then option 0 multiple times to get out of the program e) I normally than reboot the box, and the next time it comes up, the drives are back in order. e) or (instead of rebooting) option 99, to reset the chip (causes new mappings to be established), then option 8 (to verify lower target IDs), then "devfsadm". After devfsadm completes, lsiutil option 42 should display valid device names (in /dev/rdsk), and "format" should find the devices so that you can label them. Hope this helps. I happened to need it last night again (I normally have to run it after re-imaging a box, assuming that I don't want to save the data that was on those drives). Paul Jochum This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Wow, how fortunate for me that you are on this list! I guess I do have a follow-up question... If each new drive gets a new id when plugged into the system - and I learn to discover that drive's id using dmesg or iostat and use `zfs replace ` correctly - when a drive fails, what will it take for me to physically find it. I'm hoping there is a command, like dd, that I can use to make that drive's LED blink, but I don't know if I can trust that `dd` will work at all when the drive is failed! Since I don't have enclosure-services, does that mean my best option is to manually track id-to-bay mappings? (envision a clip-board hanging on rack) Given that manually tracking shifting ids doesn't sound appealing to me, would using a SATA controller like the AOC-SAT2-MV8 resolve the issue? Given that I currently only have one LSI HBA - I'd need to get 2 more for all 24 drives ---or--- I could get 3 of these SATA controllers plus 6 discrete-to-8087 reverse breakout cables. Going down the LSI-route would cost about $600 while going down the AOC-SAT2-MV8 would cost about $400. I understand that the SATA controllers are less performant, but I'd gladly exchange some performance that I'm likely to never need to simplify my administrative overhead... Thanks, Kent James C. McPherson wrote: > > Hi Kent, > I'm one of the team that works on Solaris' mpt driver, which we > recently enhanced to deliver mpxio support with SAS. I have a bit > of knowledge about your issue :-) > > Kent Watsen wrote: >> Based on recommendations from this list, I asked the company that >> built my box to use an LSI SAS3081E controller. >> >> The first problem I noticed was that the drive-numbers were ordered >> incorrectly. That is, given that my system has 24 bays (6 rows, 4 >> bays/row), the drive numbers from top-to-bottom & left-to-right were >> 6, 1, 0, 2, 4, 5 - even though when the system boots, each drive is >> scanned in perfect order (I can tell by watching the LEDs blink). >> >> I contacted LSI tech support and they explained: >> >> >> SAS treats device IDs differently than SCSI. LSI SAS controllers >> "remember" devices in the order they were discovered by the controller. >> This memory is persistent across power cycles. It is based on the world >> wide name (WWN) given uniquely to every SAS device. This allows your >> boot device to remain your boot device no matter where it migrates in >> the SAS topology. >> >> In order to clear the memory of existing devices you need at least one >> device that will not be present in your final configuration. Re-boot >> the machine and enter the LSI configuration utility (CTRL-C). Then find >> your way to SAS Topology. To see "more" options, press CTRL-M. Choose >> the option to clear all non-present device IDs. This clears the >> persistent memory of all devices not present at that time. Exchange the >> drives. The system will now remember the order it finds the drives >> after the next boot cycle. > > > Firstly, yes, the LSI SAS hbas do use persistent mapping, > with a "logical target id" by default. This is where the > hba does the translation between the physical disk device's > SAS address (which you'll see in "prtconf -v" as the devid), > and an essentially arbitrary target number which gets passed > up to the OS - in this case Solaris. > > The support person @ LSI was correct about deleting all those > mappings. > > Yes, the controller is being smart and tracking the actual > device rather than a particular bay/slot mapping. This isn't > so bad, mostly. The effect for you is that you can't assume > that the replaced device is going to have the same target > number as the old one (in fact, I'd call that quite unlikely) > so you'll have to see what the new device name is by checking > your dmesg or iostat -En output. > > > > >> Sure enough, I was able to physical reorder my drives so they were 0, >> 1, 2, 4, 5, 6 - so, appearantly, the company that put my system >> together moved the drives around after they were initially scanned. >> But where is 3? (answer below). Then I tried another test: >> >> 1. make first disk blink >> >> # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 >> 10+0 records in >> 10+0 records out >> >> 2. pull disk '0' out and replace it with a brand new disk >> >> # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 >> dd: /dev/dsk/c2t0d0p0: open: No such file or directory >> >> 3. scratch head and try again with '3' (I had previously cleared >> the LSI's controllers memory) >> >> # run dd if=/dev/dsk/c2t3d0p0 of=/dev/null count=10 >> 10+0 records in >> 10+0 records out >> >> So, it seems my SAS controller is being too smart for its own good - >> it tracks the drives themselves, not the drive-bays. If I hot-swap a >> brand new drive into a bay, Solaris will see it as a new disk, not a >> replacement for the old disk. How can ZFS support this?
Re: [zfs-discuss] LSI SAS3081E = unstable drive numbers?
Hi Kent, I'm one of the team that works on Solaris' mpt driver, which we recently enhanced to deliver mpxio support with SAS. I have a bit of knowledge about your issue :-) Kent Watsen wrote: > Based on recommendations from this list, I asked the company that built > my box to use an LSI SAS3081E controller. > > The first problem I noticed was that the drive-numbers were ordered > incorrectly. That is, given that my system has 24 bays (6 rows, 4 > bays/row), the drive numbers from top-to-bottom & left-to-right were 6, > 1, 0, 2, 4, 5 - even though when the system boots, each drive is scanned > in perfect order (I can tell by watching the LEDs blink). > > I contacted LSI tech support and they explained: > > > SAS treats device IDs differently than SCSI. LSI SAS controllers > "remember" devices in the order they were discovered by the controller. > This memory is persistent across power cycles. It is based on the world > wide name (WWN) given uniquely to every SAS device. This allows your > boot device to remain your boot device no matter where it migrates in > the SAS topology. > > In order to clear the memory of existing devices you need at least one > device that will not be present in your final configuration. Re-boot > the machine and enter the LSI configuration utility (CTRL-C). Then find > your way to SAS Topology. To see "more" options, press CTRL-M. Choose > the option to clear all non-present device IDs. This clears the > persistent memory of all devices not present at that time. Exchange the > drives. The system will now remember the order it finds the drives > after the next boot cycle. > Firstly, yes, the LSI SAS hbas do use persistent mapping, with a "logical target id" by default. This is where the hba does the translation between the physical disk device's SAS address (which you'll see in "prtconf -v" as the devid), and an essentially arbitrary target number which gets passed up to the OS - in this case Solaris. The support person @ LSI was correct about deleting all those mappings. Yes, the controller is being smart and tracking the actual device rather than a particular bay/slot mapping. This isn't so bad, mostly. The effect for you is that you can't assume that the replaced device is going to have the same target number as the old one (in fact, I'd call that quite unlikely) so you'll have to see what the new device name is by checking your dmesg or iostat -En output. > Sure enough, I was able to physical reorder my drives so they were 0, 1, > 2, 4, 5, 6 - so, appearantly, the company that put my system together > moved the drives around after they were initially scanned. But where is > 3? (answer below). Then I tried another test: > > 1. make first disk blink > > # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 > 10+0 records in > 10+0 records out > > 2. pull disk '0' out and replace it with a brand new disk > > # run dd if=/dev/dsk/c2t0d0p0 of=/dev/null count=10 > dd: /dev/dsk/c2t0d0p0: open: No such file or directory > > 3. scratch head and try again with '3' (I had previously cleared the > LSI's controllers memory) > > # run dd if=/dev/dsk/c2t3d0p0 of=/dev/null count=10 > 10+0 records in > 10+0 records out > > So, it seems my SAS controller is being too smart for its own good - it > tracks the drives themselves, not the drive-bays. If I hot-swap a brand > new drive into a bay, Solaris will see it as a new disk, not a > replacement for the old disk. How can ZFS support this? I asked the > LSI tech support again and got: > > > I don't have the knowledge to answer that, so I'll just say > this: most vendors, including Sun, set up the SAS HBA to use > "enclosure/slot" naming, which means that if a drive is > swapped, it does NOT get a new name (after all, the enclosure > and slot did not change). > Now here's where things get murky. At this point in time at least (it may change!) Solaris' mpt driver uses LSI's logical target id mapping method. This is *NOT* an enclosure/slot naming method - at least, not from the OS' point of view. Additionally, unless you're using an actual real SCSI Enclosure Services (ses) device, there's no enclosure to provide enclosure/slot mapping with either. Since mpt uses logical target id, therefore the target id which Solaris sees _will definitely change_ if you swap a disk. (I'm a tad annoyed that the LSI support person appears to have made an assumption based on a total lack of understanding about how Solaris' mpt driver works). (My assumption here is that you're using Solaris' mpt(7d) driver rather than LSI's itmpt driver) So how do you use your system and its up-to-24 drives with ZFS? (a) ensure that you note what Solaris's idea of the target id is when you replace a drive, then (b) use "zpool replace" to tell ZFS what to do with the new device in your enclosure. I hope the above helps you along the way...
