Hello Team, Could you please help me out here.
-Udai *From:* lustre-discuss <lustre-discuss-boun...@lists.lustre.org> *On Behalf > Of *Udai Sharma > *Sent:* Friday, August 23, 2019 4:40 PM > *To:* lustre-discuss@lists.lustre.org > *Subject:* [SPAMMY (6.924)][lustre-discuss] Lustre in HA-LVM Cluster issue > > > > Hi Team, > > > > Starting with topology and configurations: > > > > --------------------------- > > Topology: > > > > [HA1]< -----[N3]----- >[N4] > > | | | > > | | | > > -----[Client]---------- > > > > [N1,N2] = HA1 --> OSTs > > N3 --- > MGS > > N4 --- > MDT > > > > N1 -> 3 Logical volumes [OST1,OST2,OST3] > > N2 -> 3 Logical volumes [OST4,OST5,OST6] > > N3 -> 1 Logical volume [MGT1] > > N4 -> 1 Logical volume [MDT1] > > ------------------------------------------ > > > > N3 [MGS] > > > > Created Zpool, formatted and Mounted it. > > > > zpool create -f -O canmount=off -o multihost=on -o cachefile=none lustre > /dev/mgs/mgs01 > > mkfs.lustre --reformat --mgs --backfstype=zfs lustre/mgs01 > > mount.lustre lustre/mgs01 /mnt/mgs/ > > > > ------------------------------------------ > > > > N4 [MDT] > > > > Created Zpool, formatted and Mounted it. > > > > zpool create -f -O canmount=off -o multihost=on -o cachefile=none lustre > /dev/mdt/mdt01 > > mkfs.lustre --reformat --mdt --fsname=lustre --index=0 > --mgsnode=10.2.2.202@tcp1 --backfstype=zfs lustre/mdt01 > > mount.lustre lustre/mdt01 /mnt/mdt > > > > ---------------------------------------- > > > > HA1 [HA-LVM system] > > > > *N1 [OST1,OST2,OST3]* > > > > Created Zpool, formatted and Mounted it. > > zpool create lustre -f -O canmount=off -o multihost=on -o cachefile=none > /dev/vg_e/thinvolume1 /dev/vg_e/thinvolume2 /dev/vg_e/thinvolume3 > > mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=111 > --mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 > lustre/ost01 ; mount.lustre lustre/ost01 > /mnt/ost01/ > > mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=222 > --mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 > lustre/ost02 ; mount.lustre lustre/ost02 /mnt/ost02/ > > mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=333 > --mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 > lustre/ost03 ; mount.lustre lustre/ost03 /mnt/ost03/ > > df -h | grep lustre > > lustre/ost01 287G 3.0M 287G 1% /mnt/ost01 > > lustre/ost02 287G 3.0M 287G 1% /mnt/ost02 > > lustre/ost03 287G 3.0M 287G 1% /mnt/ost03 > > > > > > *N2 [OST4,OST5,OST6]* > > > > Created Zpool, formatted and Mounted it. > > zpool create -f -O canmount=off -o multihost=on -o cachefile=none lustre > /dev/vg_p/thinvolume1 /dev/vg_p/thinvolume2 /dev/vg_p/thinvolume3 > > mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=444 > --mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 > lustre/ost04 ; mount.lustre lustre/ost04 > /mnt/ost04 > > > mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=555 > --mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 > lustre/ost05 ; mount.lustre lustre/ost05 /mnt/ost05 > > mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=666 > --mgsnode=10.2.2.202@tcp1 --servicenode=10.2.2.239@tcp1:10.2.2.241@tcp1 > lustre/ost06 ; mount.lustre lustre/ost06 /mnt/ost06 > > > > df -h | grep lustre > > lustre/ost04 287G 3.0M 287G 1% /mnt/ost04 > > lustre/ost05 287G 3.0M 287G 1% /mnt/ost05 > > lustre/ost06 287G 3.0M 287G 1% /mnt/ost06 > > > > *Created PCS cluster over HA1. * > > > > Resource Group: electron > > vg_e (ocf::heartbeat:LVM): Started gp-electron > > zfs-pool-electron (ocf::heartbeat:ZFS): Started electron > > lustre-ost1 (ocf::heartbeat:Lustre): Started electron > > lustre-ost2 (ocf::heartbeat:Lustre): Started electron > > lustre-ost3 (ocf::heartbeat:Lustre): Started electron > > Resource Group: proton > > vg_p (ocf::heartbeat:LVM): Started gp-proton > > zfs-pool-proton (ocf::heartbeat:ZFS): Started proton > > lustre-ost4 (ocf::heartbeat:Lustre): Started proton > > lustre-ost5 (ocf::heartbeat:Lustre): Started proton > > lustre-ost6 (ocf::heartbeat:Lustre): Started proton > > > > ---------------------------------------- > > > > Client: > > > > # mount | grep lustre > > 10.2.2.202@tcp1:/lustre on /lustre type lustre (rw,lazystatfs) > > > > #lfs osts > > OBDS: > > 1: lustre-OST0001_UUID INACTIVE > > 2: lustre-OST0002_UUID INACTIVE > > 3: lustre-OST0003_UUID INACTIVE > > 4: lustre-OST0004_UUID INACTIVE > > 5: lustre-OST0005_UUID INACTIVE > > 6: lustre-OST0006_UUID INACTIVE > > 10: lustre-OST000a_UUID INACTIVE > > 11: lustre-OST000b_UUID INACTIVE > > 20: lustre-OST0014_UUID INACTIVE > > 22: lustre-OST0016_UUID INACTIVE > > 30: lustre-OST001e_UUID INACTIVE > > 33: lustre-OST0021_UUID INACTIVE > > 40: lustre-OST0028_UUID INACTIVE > > 44: lustre-OST002c_UUID INACTIVE > > 50: lustre-OST0032_UUID INACTIVE > > 55: lustre-OST0037_UUID INACTIVE > > 60: lustre-OST003c_UUID INACTIVE > > 66: lustre-OST0042_UUID INACTIVE > > 111: lustre-OST006f_UUID ACTIVE > > 222: lustre-OST00de_UUID ACTIVE > > 333: lustre-OST014d_UUID ACTIVE > > 444: lustre-OST01bc_UUID ACTIVE > > 555: lustre-OST022b_UUID ACTIVE > > 666: lustre-OST029a_UUID ACTIVE > > > > # lfs mdts > > MDTS: > > 0: lustre-MDT0000_UUID ACTIVE > > > > > > > > OSTs are part of active-passive HA-LVM cluster. As per configuration, in > case of resource failure, RA moves the resource to other node. It’s working > fine. > > > > Issues seen: > > 1. OST00* are going to INACTIVE state if the corresponding disk in > unmounted from the OST server, it never becomes active even-though if it > mounted again. Had to format every time with –index to get the volumes > listed again. Hence, so many inactive nodes in the ‘lctl dl’ output. > 2. Recovery_status at N4[MDT] is always inactive. How to enable > recovery, in case of failure. > 3. How to re-activate the INACTIVE objects at OSTs? > 4. In case of failover of HA resource services gets moved to other > peer, but IOs, iozone or dd, never resumes. > > > > Please advise. > > > > PS: lctl 2.12.2 > > > > Thanks in advance. > > Udai > > > > >
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org