On Apr 17, 2013, at 4:56 PM, Outback Dingo wrote:
On Wed, Apr 17, 2013 at 7:29 PM, Teske, Devin <devin.te...@fisglobal.com<mailto:devin.te...@fisglobal.com>> wrote: On Apr 17, 2013, at 4:10 PM, Outback Dingo wrote: On Wed, Apr 17, 2013 at 6:39 PM, Teske, Devin <devin.te...@fisglobal.com<mailto:devin.te...@fisglobal.com>> wrote: On Apr 17, 2013, at 3:26 PM, Outback Dingo wrote: > Ok, maybe im at a loss here in the way my brain is viewing this > > we have a box, its got 2 nodes in the chassis, and 32 sata drives > attached to a SATA/SAS backplane via 4 (2 per node) LSI MPT SAS2 cards > should i not logically be seeing 4 controllers X #drive count ?? > > camcontrol devlist shows 32 devices, daX,passX and sesX,passX > > <SEAGATE ST33000650SS 0004> at scbus0 target 9 lun 0 (da0,pass0) > <STORBRICK-3 1400> at scbus0 target 10 lun 0 (ses0,pass1) > <SEAGATE ST33000650SS 0004> at scbus0 target 11 lun 0 (da1,pass2) > <STORBRICK-1 1400> at scbus0 target 12 lun 0 (ses1,pass3) > <SEAGATE ST33000650SS 0004> at scbus0 target 13 lun 0 (da2,pass4) > <STORBRICK-2 1400> at scbus0 target 14 lun 0 (ses2,pass5) > <SEAGATE ST33000650SS 0004> at scbus0 target 15 lun 0 (da3,pass6) > <STORBRICK-4 1400> at scbus0 target 16 lun 0 (ses3,pass7) > <SEAGATE ST33000650SS 0004> at scbus0 target 17 lun 0 (da4,pass8) > <STORBRICK-6 1400> at scbus0 target 18 lun 0 (ses4,pass9) > <SEAGATE ST33000650SS 0004> at scbus0 target 19 lun 0 (da5,pass10) > <STORBRICK-0 1400> at scbus0 target 20 lun 0 (ses5,pass11) > <SEAGATE ST33000650SS 0004> at scbus0 target 21 lun 0 (da6,pass12) > <STORBRICK-7 1400> at scbus0 target 22 lun 0 (ses6,pass13) > <SEAGATE ST33000650SS 0004> at scbus0 target 23 lun 0 (da7,pass14) > <STORBRICK-5 1400> at scbus0 target 24 lun 0 (ses7,pass15) > <SEAGATE ST9300605SS 0004> at scbus1 target 0 lun 0 (da8,pass16) > <SEAGATE ST9300605SS 0004> at scbus1 target 1 lun 0 (da9,pass17) > <STORBRICK-3 1400> at scbus8 target 10 lun 0 (ses8,pass19) > <SEAGATE ST33000650SS 0004> at scbus8 target 11 lun 0 (da11,pass20) > <STORBRICK-1 1400> at scbus8 target 12 lun 0 (ses9,pass21) > <SEAGATE ST33000650SS 0004> at scbus8 target 13 lun 0 (da12,pass22) > <STORBRICK-2 1400> at scbus8 target 14 lun 0 (ses10,pass23) > <SEAGATE ST33000650SS 0004> at scbus8 target 15 lun 0 (da13,pass24) > <STORBRICK-4 1400> at scbus8 target 16 lun 0 (ses11,pass25) > <SEAGATE ST33000650SS 0004> at scbus8 target 17 lun 0 (da14,pass26) > <STORBRICK-6 1400> at scbus8 target 18 lun 0 (ses12,pass27) > <SEAGATE ST33000650SS 0004> at scbus8 target 19 lun 0 (da15,pass28) > <STORBRICK-0 1400> at scbus8 target 20 lun 0 (ses13,pass29) > <SEAGATE ST33000650SS 0004> at scbus8 target 21 lun 0 (da16,pass30) > <STORBRICK-7 1400> at scbus8 target 22 lun 0 (ses14,pass31) > <SEAGATE ST33000650SS 0004> at scbus8 target 23 lun 0 (da17,pass32) > <STORBRICK-5 1400> at scbus8 target 24 lun 0 (ses15,pass33) > <USB 2.0 Flash Drive 8.07> at scbus9 target 0 lun 0 (da18,pass34) > > > we would like to create a zpool from all the devices, that in theory if > nodeA failed > then nodeB could force import the pool, gmultipath (which you mention in the subject) is the appropriate tool for this, but there's no need for an import of the pool if you build the pool out of multipath devices. In our experience, we can pull a cable and zfs continues working just fine. In other words, don't build the pool out of the devices, put a gmultipath label on each device and then use /dev/multipath/LABEL for the zpool devices. > nodeA and NodeB are attached through > dual LSI controllers, to the SATA/SAS backplane. but i cant seem to create > a zpool from sesX or passX devices, i can however create a 16 drive zp0ol > on either node, from any daX device. what did i miss? ive looked at > gmirror, and also ses documents. Any insight is appreciated, thanks in > advance. gmirror is the wrong tool, gmultipath is what you want. The basic task is to use "gmultipath label FOO da#" to write a cookie on the disk (used to identify new/existing paths during GOEM "taste" events for example). After you've labeled the da# devices with gmultipath you say "gmultipath status" to see the components of each label and you use "multipath/LABEL" as your disk name when creating the zpool (these correspond directly to /dev/multipath/LABEL, but "zpool create …" or "zpool add …" allow you to omit the leading "/dev"). sanity check me on node A i did zpool destroy master gmultipath label FOO da0 gmultipath status Name Status Components multipath/FOO DEGRADED da0 (ACTIVE) multipath/FOO-619648737 DEGRADED da1 (ACTIVE) multipath/FOO-191725652 DEGRADED da2 (ACTIVE) multipath/FOO-1539342315 DEGRADED da3 (ACTIVE) multipath/FOO-1276041606 DEGRADED da4 (ACTIVE) multipath/FOO-2000832198 DEGRADED da5 (ACTIVE) multipath/FOO-1285640577 DEGRADED da6 (ACTIVE) multipath/FOO-1816092574 DEGRADED da7 (ACTIVE) multipath/FOO-1102254444 DEGRADED da8 (ACTIVE) multipath/FOO-330300690 DEGRADED da9 (ACTIVE) multipath/FOO-92140635 DEGRADED da10 (ACTIVE) multipath/FOO-855257672 DEGRADED da11 (ACTIVE) multipath/FOO-1003634134 DEGRADED da12 (ACTIVE) multipath/FOO-2449862 DEGRADED da13 (ACTIVE) multipath/FOO-1137080233 DEGRADED da14 (ACTIVE) multipath/FOO-1696804371 DEGRADED da15 (ACTIVE) multipath/FOO-1304457562 DEGRADED da16 (ACTIVE) multipath/FOO-912159854 DEGRADED da17 (ACTIVE) now on node B i should do the same? reboot both nodes and i should be able "see" 32 multipath/FOO deices to create a pool from ? It appears from the above output that you labeled all of the block devices (da0 through da17) with the same label. This is not what you want. Use "gmultipath clear FOO" on each of the block devices and have another go using unique values. For example: gmultipath label SATA_LUN01 da0 gmultipath label SATA_LUN02 da1 gmultipath label SATA_LUN03 da2 gmultipath label SATA_LUN04 da3 gmultipath label SATA_LUN05 da4 gmultipath label SATA_LUN06 da5 gmultipath label SATA_LUN07 da6 gmultipath label SATA_LUN08 da7 gmultipath label SATA_LUN09 da8 gmultipath label SATA_LUN10 da9 gmultipath label SATA_LUN11 da10 gmultipath label SATA_LUN12 da11 gmultipath label SATA_LUN13 da12 gmultipath label SATA_LUN14 da13 gmultipath label SATA_LUN15 da14 gmultipath label SATA_LUN16 da15 gmultipath label SATA_LUN17 da16 gmultipath label SATA_LUN18 da17 .. Then "gmultipath status" should show your unique labels each with a single component. Then you would do: zpool create master multipath/SATA_LUN{01,02,03,04,05,06,…} ahh ok got it, and probably on the other node gmultipath label SATA_LUN19 da0 gmultipath label SATA_LUN20 da1 -------------------snip------------------------------ gmultipath label SATA_LUN36 da15 No. You do not need to label the other "node" Since the "gmultipath label …" command writes data to the disk, you do not need to label the disk multiple times (and in fact would be an error to). Rather, as the system is probing and adding disks, it will automatically detect multiple paths based on this data stored on the disk. Read: If da0 and another da# device are indeed two paths to the same device, then as those devices are probed by the kernel, "gmultipath status" will dynamically show the newly discovered paths. If, after labeling all the devices on a single path you find that "gmultipath status" still shows only one component for each label, try rebooting. If still after a reboot "gmultipath status" only shows a single component for each label, then clearly you are not configured (hardware wise) for multiple paths to the same components (and this may be where the "gmultipath" versus "gmirror" nit that I caught in your original post comes into play -- maybe "gmultipath" was the wrong thing to put in the subject if you don't have multiple paths to the same components but instead have a mirrored set of components that you want to gmirror all your data to a second pool -- if that ends up being the case, then I would actually recommend a zfs send/receive cron-job based on snapshots to utilize the performance of ZFS Copy On Write rather than perhaps gmirror; but your mileage may vary). -- Devin then create the zpool from the "36" multipath devices? so if i create a 36 drive multipath zpool on nodeA when it fails do i just import it to nodeB i was thinking to use carp for failover..... so nodeB would continue nfs sessions and import the zpool to nodeB -- Devin _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you. _______________________________________________ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"