Re: [lustre-discuss] MGS+MDT migration to a new storage using LVM tools
Thanks Andreas for your detailed reply. I took your advice on the MDT naming. As now the migration is complete I want to share some major problems I had on the way. I don't know where to point the blame, the Lustre kernel, e2fprogs, srp tools, multipath version, lvm version and moving from 512 blocks to 4096. But as soon as I created the mirror, the server went into a kernel panic, core dump loop. I managed to stop it only by breaking the mirror from another server connected to the same storage. It took me a full day to recover the system. Today I restarted the process, this time from a different server ,not running Lustre, which I already used for a vm lun lvm migration. The exact same procedure ran flawlessly and I needed only to refresh the lvm on the MDS to be able to mount the migrated mdt. Cheers, David On Sun, Jul 19, 2020 at 12:27 PM Andreas Dilger wrote: > On Jul 19, 2020, at 12:41 AM, David Cohen > wrote: > > > > Hi, > > We have a combined MGS+MDT and I'm looking for a migration to new > storage with a minimal disruption to the running jobs on the cluster. > > > > Can anyone find problems in the scenario below and/or suggest another > solution? > > I would appreciate also "no problems" replies to reassure the scenario > before I proceed. > > > > Current configuration: > > The mdt is a logical volume in a lustre_pool VG on a /dev/mapper/MDT0001 > PV > > I've been running Lustre on LVM at home for many years, and have done > pvmove > of the underlying storage to new devices without any problems. > > > Migration plan: > > Add /dev/mapper/MDT0002 new disk (multipath) > > I would really recommend that you *not* use MDT0002 as the name of the PV. > This is very confusing because the MDT itself (at the Lustre level) is > almost certainly named "-MDT", and if you ever add new MDTs to > this filesystem it will be confusing as to which *Lustre* MDT is on which > underlying PV. Instead, I'd take the opportunity to name this "MDT" to > match the actual Lustre MDT target name. > > > extend the VG: > > pvcreate /dev/mapper/MDT0002 > > vgextend lustre_pool /dev/mapper/MDT0002 > > mirror the mdt to the new disk: > > lvconvert -m 1 /dev/lustre_pool/TECH_MDT /dev/mapper/MDT0002 > > I typically just use "pvmove", but doing this by adding a mirror and then > splitting it off is probably safer. That would still leave you with a full > copy of the MDT on the original PV if something happened in the middle. > > > wait the mirrored disk to sync: > > lvs -o+devices > > when it's fully synced unmount the MDT, remove the old disk from the > mirror: > > lvconvert -m 0 /dev/lustre_pool/TECH_MDT /dev/mapper/MDT0001 > > and remove the old disk from the pool: > > vgreduce lustre_pool /dev/mapper/MDT0001 > > pvremove /dev/mapper/MDT0001 > > remount the MDT and let the clients few minutes to recover the > connection. > > In my experience with pvmove, there is no need to do anything with the > clients, > as long as you are not also moving the MDT to a new server, since the > LVM/DM > operations are totally transparent to both the Lustre server and client. > > After my pvmove (your "lvconvert -m 0"), I would just vgreduce the old PV > from > the VG, and then leave it in the system (internal HDD) until the next time > I > needed to shut down the server. If you have hot-plug capability for the > PVs, > then you don't even need to wait for that. > > Cheers, Andreas > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] MGS+MDT migration to a new storage using LVM tools
On Jul 19, 2020, at 12:41 AM, David Cohen wrote: > > Hi, > We have a combined MGS+MDT and I'm looking for a migration to new storage > with a minimal disruption to the running jobs on the cluster. > > Can anyone find problems in the scenario below and/or suggest another > solution? > I would appreciate also "no problems" replies to reassure the scenario before > I proceed. > > Current configuration: > The mdt is a logical volume in a lustre_pool VG on a /dev/mapper/MDT0001 PV I've been running Lustre on LVM at home for many years, and have done pvmove of the underlying storage to new devices without any problems. > Migration plan: > Add /dev/mapper/MDT0002 new disk (multipath) I would really recommend that you *not* use MDT0002 as the name of the PV. This is very confusing because the MDT itself (at the Lustre level) is almost certainly named "-MDT", and if you ever add new MDTs to this filesystem it will be confusing as to which *Lustre* MDT is on which underlying PV. Instead, I'd take the opportunity to name this "MDT" to match the actual Lustre MDT target name. > extend the VG: > pvcreate /dev/mapper/MDT0002 > vgextend lustre_pool /dev/mapper/MDT0002 > mirror the mdt to the new disk: > lvconvert -m 1 /dev/lustre_pool/TECH_MDT /dev/mapper/MDT0002 I typically just use "pvmove", but doing this by adding a mirror and then splitting it off is probably safer. That would still leave you with a full copy of the MDT on the original PV if something happened in the middle. > wait the mirrored disk to sync: > lvs -o+devices > when it's fully synced unmount the MDT, remove the old disk from the mirror: > lvconvert -m 0 /dev/lustre_pool/TECH_MDT /dev/mapper/MDT0001 > and remove the old disk from the pool: > vgreduce lustre_pool /dev/mapper/MDT0001 > pvremove /dev/mapper/MDT0001 > remount the MDT and let the clients few minutes to recover the connection. In my experience with pvmove, there is no need to do anything with the clients, as long as you are not also moving the MDT to a new server, since the LVM/DM operations are totally transparent to both the Lustre server and client. After my pvmove (your "lvconvert -m 0"), I would just vgreduce the old PV from the VG, and then leave it in the system (internal HDD) until the next time I needed to shut down the server. If you have hot-plug capability for the PVs, then you don't even need to wait for that. Cheers, Andreas signature.asc Description: Message signed with OpenPGP ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] MGS+MDT migration to a new storage using LVM tools
Hi, We have a combined MGS+MDT and I'm looking for a migration to new storage with a minimal disruption to the running jobs on the cluster. Can anyone find problems in the scenario below and/or suggest another solution? I would appreciate also "no problems" replies to reassure the scenario before I proceed. Current configuration: The mdt is a logical volume in a lustre_pool VG on a /dev/mapper/MDT0001 PV Migration plan: Add /dev/mapper/MDT0002 new disk (multipath) extend the VG: pvcreate /dev/mapper/MDT0002 vgextend lustre_pool /dev/mapper/MDT0002 mirror the mdt to the new disk: lvconvert -m 1 /dev/lustre_pool/TECH_MDT /dev/mapper/MDT0002 wait the mirrored disk to sync: lvs -o+devices when it's fully synced unmount the MDT, remove the old disk from the mirror: lvconvert -m 0 /dev/lustre_pool/TECH_MDT /dev/mapper/MDT0001 and remove the old disk from the pool: vgreduce lustre_pool /dev/mapper/MDT0001 pvremove /dev/mapper/MDT0001 remount the MDT and let the clients few minutes to recover the connection. Thanks, David ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
