Re: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3
and one RAID-1 array >> of two 480 GB SATA SSD for metadata). We use Lustre to essentially have a >> consistent "relatively large and fast" file system able do to MPI-IO >> correctly. Our system is sufficient for our small team. As I said before, we >> plan in the next weeks to switch from CentOS 7.9 with Lustre 2.12.9 and >> MOFED 4.9 LTS to Lustre 2.15.3, MOFED 5.8 LTS with Alma/RHEL 9.2 on the head >> and compute nodes and Alma/RHEL 8.8 on the file server (Lustre) node. >> >> Let's hope we don't hit too many problems during this large update ! >> >> Thanks, >> Martin Audet >> >> >> P.S. Why do you want to stay with MOFED 4.9 ? MOFED 4.9 include UCX 1.8.0 >> which is considered problematic (ex: memory corruption) by OpenMPI libraries >> version > 4.1.0 (they refuse to compile, hopefully we never encountered this >> problem to my knowledge). More recent MOFED also come bundled with xpmem >> which is supposed to accelerate intro-node communications (zero copy >> mechanism) way better than CMA or KNEM (and xpmem is supported by both >> OpenMPI and mpich). >> >> From: Tung-Han Hsieh >>> Sent: September 26, 2023 9:57 PM >>> To: Audet, Martin >>> Cc: lustre-discuss@lists.lustre.org >>> Subject: Re: [lustre-discuss] Cannot mount MDT after upgrading from Lustre >>> 2.12.6 to 2.15.3 >>> >>> ***Attention*** This email originated from outside of the NRC. >>> ***Attention*** Ce courriel provient de l'extérieur du CNRC. >>> >>> OK. Let me provide more details about the upgrade from 2.12.X to 2.15.X. >>> >>> We have several production clusters. Due to different requirements and >>> hardware, each has slightly different Lustre configurations. Our Linux >>> distribution is Debian. Sometimes due to hardware spec, we need to use >>> different Linux kernel versions. So we always have to compile Kernel >>> (vanilla Linux kernel), MLNX_OFED, e2fsprog / ZFS, and Lustre. Here I >>> temporarily skip the compilation (that involved a lot of details), just >>> mention the upgrade procedure for two different cases. >>> >>> 1. Two of our clusters have Lustre-2.12.6, with MDT with ZFS backend, and >>> OST with ZFS backend, where ZFS is 0.7.13. From the compatibility list >>> mentioned in documentation, the upgrade involves both ZFS upgrade to 2.0.7 >>> and Lustre upgrade to 2.15.X. It was very smooth and successful. The >>> procedure is: >>> 1) Compile and install new versions of Linux kernel, MLNX_OFED, ZFS, >>> and Lustre. Depending on the hardware, some servers we use Linux kernel >>> 4.19.X + MLNX_OFED-4.6, or Linux kernel 4.19.X + MLNX_OFED-4.9, or Linux >>> kernel 5.4.X + MLNX_OFED-4.9. They are all compatible to Lustre-2.15.X and >>> ZFS-2.0.7. >>> 2) After installation of the new software, start up ZFS first: >>> ``` >>> modprobe zfs >>> zpool import >>> zpool status >>> ``` >>> There will be messages about ZFS pool needs to be upgraded to >>> enable new features. But after upgrade, the ZFS pool will not be compatible >>> to the old ZFS version anymore. The command to upgrade ZFS pool is (for all >>> the MDT and OST pools): >>> ``` >>> zpool upgrade >>> zpool status >>> ``` >>> Checking zpool status again, the warning messages were >>> disappeared. >>> 3) I checked the Lustre Operation Manual again, and saw that in the >>> recent version it does not need to run tunefs.lustre --writeconf command. >>> Sorry that it was my fault. But in our case, we run this command whenever >>> doing Lustre upgrade. Note that to run it, we have to run it in all the >>> Lustre MDT and OST pools. It cleared out the logs and will regenerate it >>> when mounting Lustre. >>> >>> 2. One of our clusters have Lustre-2.12.6, with MDT with ldiskfs backend, >>> and OST with ZFS backend, where ZFS is 0.7.13. We have to upgrade it to >>> e2fsprogs-1.47.0, ZFS-2.0.7 and Lustre-2.15.3. The upgrade of OST part is >>> exactly the same as the above, so I don't repeat. The major challenge is >>> MDT with ldiskfs. What I have done is >>> 1) After installing all the new version of the software, running >>> tunefs.lustre --writeconf (for all MDT and OST). Probably this is the wrong >>> step for the upgrade to Lustre-2.15.X. >>> 2) According to Lustre Operation manual chapter 17, t
Re: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3
ar to ZFS, instead of copying from Linux kernel and applying patches to convert to ldiskfs code. Doing this, it is expected that ldiskfs could be less dependent on the minor revisions of Linux kernel, to bring us more freedom of choosing a suitable Linux kernel version in our applications. Best Regards, T.H.Hsieh Audet, Martin 於 2023年9月28日 週四 上午2:02寫道： > Hello Hsieh, > > > Thanks for sharing your experience in response to my request. Thanks also > for the clarification concerning --writeconf. Since I didn't see that in > the documentation for major updates, I thought I missed something. Let's > hope that other will follow and that more information becomes available. > > > From your message I noticed that you mentioned that compiling Lustre with > ldiskfs as a back end is a nightmare. You argue that Lustre code is not > self contained and essentially consist of a set of patches over the kernel > ext4 sources. While I don't contest this I have noticed that hopefully the > number and the volume of patches tend to decrease with time from RHEL 7 to > 8 and finally to 9. With RHEL 9.2 I think there is only two patches > remaining. There is also the patchless option for servers. I don't know if > you have any experience with that. Older documentation warned about a > performance penalty when taking advantage of this feature... > > > From what you describe I see that your systems are way larger than ours. > As I said our cluster is small and our Lustre installation is minimal (24 > compute nodes each with two 12 cores CPUs, Infiniband EDR, 192 GB RAM per > node, one similarly configure head and file server node, the file server > node host the Lustre MGS, MDS and OSS servers, it is connected to two > RAID-6 arrays consisting each of six 8 TB NLSAS HDD for data and one RAID-1 > array of two 480 GB SATA SSD for metadata). We use Lustre to essentially > have a consistent "relatively large and fast" file system able do to MPI-IO > correctly. Our system is sufficient for our small team. As I said before, > we plan in the next weeks to switch from CentOS 7.9 with Lustre 2.12.9 and > MOFED 4.9 LTS to Lustre 2.15.3, MOFED 5.8 LTS with Alma/RHEL 9.2 on the > head and compute nodes and Alma/RHEL 8.8 on the file server (Lustre) node. > > > Let's hope we don't hit too many problems during this large update ! > > > Thanks, > > Martin Audet > > P.S. Why do you want to stay with MOFED 4.9 ? MOFED 4.9 include UCX 1.8.0 > which is considered problematic (ex: memory corruption) by OpenMPI > libraries version > 4.1.0 (they refuse to compile, hopefully we never > encountered this problem to my knowledge). More recent MOFED also come > bundled with xpmem which is supposed to accelerate intro-node > communications (zero copy mechanism) way better than CMA or KNEM (and xpmem > is supported by both OpenMPI and mpich). > > ------ > *From:* Tung-Han Hsieh > *Sent:* September 26, 2023 9:57 PM > *To:* Audet, Martin > *Cc:* lustre-discuss@lists.lustre.org > *Subject:* Re: [lustre-discuss] Cannot mount MDT after upgrading from > Lustre 2.12.6 to 2.15.3 > > > Attention*** This email originated from outside of the NRC. > ***Attention*** Ce courriel provient de l'extérieur du CNRC.* > OK. Let me provide more details about the upgrade from 2.12.X to 2.15.X. > > We have several production clusters. Due to different requirements and > hardware, each has slightly different Lustre configurations. Our Linux > distribution is Debian. Sometimes due to hardware spec, we need to use > different Linux kernel versions. So we always have to compile Kernel > (vanilla Linux kernel), MLNX_OFED, e2fsprog / ZFS, and Lustre. Here I > temporarily skip the compilation (that involved a lot of details), just > mention the upgrade procedure for two different cases. > > 1. Two of our clusters have Lustre-2.12.6, with MDT with ZFS backend, and > OST with ZFS backend, where ZFS is 0.7.13. From the compatibility list > mentioned in documentation, the upgrade involves both ZFS upgrade to 2.0.7 > and Lustre upgrade to 2.15.X. It was very smooth and successful. The > procedure is: > 1) Compile and install new versions of Linux kernel, MLNX_OFED, ZFS, > and Lustre. Depending on the hardware, some servers we use Linux kernel > 4.19.X + MLNX_OFED-4.6, or Linux kernel 4.19.X + MLNX_OFED-4.9, or Linux > kernel 5.4.X + MLNX_OFED-4.9. They are all compatible to Lustre-2.15.X and > ZFS-2.0.7. > 2) After installation of the new software, start up ZFS first: > ``` > modprobe zfs > zpool import > zpool status > ``` > There will be messages about ZFS pool needs to be upgraded to > enable new features. But after upgrade, the ZFS pool will not be compatible > to the ol
Re: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3
On Sep 26, 2023, at 13:44, Audet, Martin via lustre-discuss mailto:lustre-discuss@lists.lustre.org>> wrote: Hello all, I would appreciate if the community would give more attention to this issue because upgrading from 2.12.x to 2.15.x, two LTS versions, is something that we can expect many cluster admin will try to do in the next few months... Who, in particular, is "the community"? That term implies a collective effort, and I'd welcome feedback from your testing of the upgrade process. It is definitely possible for an individual to install Lustre 2.12.9 on one or more VMs (or better, use a clone of your current server OS image), format a small test filesystem with the current configuration, copy some data into it, and then follow your planned process to upgrade to 2.15.3 (which should mostly just be "unmount everything, install new RPMs, mount"). That is prudent system administration to test the process in advance of changing your production system. We ourselves plan to upgrade a small Lustre (production) system from 2.12.9 to 2.15.3 in the next couple of weeks... After seeing problems reports like this we start feeling a bit nervous... The documentation for doing this major update appears to me as not very specific... Patches with improvements to the process described in the manual are welcome. Please see https://wiki.lustre.org/Lustre_Manual_Changes for details on how to submit your contributions. In this document for example, https://doc.lustre.org/lustre_manual.xhtml#upgradinglustre , the update process appears not so difficult and there is no mention of using "tunefs.lustre --writeconf" for this kind of update. Or am I missing something ? I think you are answering your own question here... The documented upgrade process has no mention of running "writeconf", but it was run for an unknown reason. This introduced an unknown problem with the configuration files that prevented the target from mounting. Then, rather than re-running writeconf to fix the configuration files, the entire MDT was copied to a new storage device (a large no-op IMHO, since any issue with the MDT config files would be copied along with it) and writeconf was run again to regenerate the configs, which could have been done just as easily on the original MDT. So the relatively straight forward upgrade process was turned into a complicated process for no apparent reason. There have been 2.12->2.15 upgrades done already in production without issues, and this is also tested continuously during development. Of course there are a wide variety of different configurations, features, and hardware on which Lustre is run, and it isn't possible to test even a fraction of all configurations. I don't think one problem report on the mailing list is an indication that there are fundamental issues with the upgrade process. Cheers, Andreas Thanks in advance for providing more tips for this kind of update. Martin Audet From: lustre-discuss mailto:lustre-discuss-boun...@lists.lustre.org>> on behalf of Tung-Han Hsieh via lustre-discuss mailto:lustre-discuss@lists.lustre.org>> Sent: September 23, 2023 2:20 PM To: lustre-discuss@lists.lustre.org<mailto:lustre-discuss@lists.lustre.org> Subject: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3 ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Dear All, Today we tried to upgrade Lustre file system from version 2.12.6 to 2.15.3. But after the work, we cannot mount MDT successfully. Our MDT is ldiskfs backend. The procedure of upgrade is 1. Install the new version of e2fsprogs-1.47.0 2. Install Lustre-2.15.3 3. After reboot, run: tunefs.lustre --writeconf /dev/md0 Then when mounting MDT, we got the error message in dmesg: === [11662.434724] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.584593] Lustre: 3440:0:(scrub.c:189:scrub_file_load()) chome-MDT: reset scrub OI count for format change (LU-16655) [11666.036253] Lustre: MGS: Logs for fs chome were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11666.523144] Lustre: chome-MDT: Imperative Recovery not enabled, recovery window 300-900 [11666.594098] LustreError: 3440:0:(mdd_device.c:1355:mdd_prepare()) chome-MDD: get default LMV of root failed: rc = -2 [11666.594291] LustreError: 3440:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [11666.594951] Lustre: Failing over chome-MDT [11672.868438] Lustre: 3440:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1695492248/real 1695492248] req@5dfd9b53 x1777852464760
Re: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3
://doc.lustre.org/lustre_manual.xhtml#upgradinglustre , the update > process appears not so difficult and there is no mention of using > "tunefs.lustre > --writeconf" for this kind of update. > > > Or am I missing something ? > > > Thanks in advance for providing more tips for this kind of update. > > > Martin Audet > -- > *From:* lustre-discuss on > behalf of Tung-Han Hsieh via lustre-discuss < > lustre-discuss@lists.lustre.org> > *Sent:* September 23, 2023 2:20 PM > *To:* lustre-discuss@lists.lustre.org > *Subject:* [lustre-discuss] Cannot mount MDT after upgrading from Lustre > 2.12.6 to 2.15.3 > > > Attention*** This email originated from outside of the NRC. > ***Attention*** Ce courriel provient de l'extérieur du CNRC.* > Dear All, > > Today we tried to upgrade Lustre file system from version 2.12.6 to > 2.15.3. But after the work, we cannot mount MDT successfully. Our MDT is > ldiskfs backend. The procedure of upgrade is > > 1. Install the new version of e2fsprogs-1.47.0 > 2. Install Lustre-2.15.3 > 3. After reboot, run: tunefs.lustre --writeconf /dev/md0 > > Then when mounting MDT, we got the error message in dmesg: > > === > [11662.434724] LDISKFS-fs (md0): mounted filesystem with ordered data > mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc > [11662.584593] Lustre: 3440:0:(scrub.c:189:scrub_file_load()) > chome-MDT: reset scrub OI count for format change (LU-16655) > [11666.036253] Lustre: MGS: Logs for fs chome were removed by user > request. All servers must be restarted in order to regenerate the logs: rc > = 0 > [11666.523144] Lustre: chome-MDT: Imperative Recovery not enabled, > recovery window 300-900 > [11666.594098] LustreError: 3440:0:(mdd_device.c:1355:mdd_prepare()) > chome-MDD: get default LMV of root failed: rc = -2 > [11666.594291] LustreError: > 3440:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start > targets: -2 > [11666.594951] Lustre: Failing over chome-MDT > [11672.868438] Lustre: 3440:0:(client.c:2295:ptlrpc_expire_one_request()) > @@@ Request sent has timed out for slow reply: [sent 1695492248/real > 1695492248] req@5dfd9b53 x1777852464760768/t0(0) > o251->MGC192.168.32.240@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl > 1695492254 ref 2 fl Rpc:XNQr/0/ rc 0/-1 job:'' > [11672.925905] Lustre: server umount chome-MDT complete > [11672.926036] LustreError: 3440:0:(super25.c:183:lustre_fill_super()) > llite: Unable to mount : rc = -2 > [11872.893970] LDISKFS-fs (md0): mounted filesystem with ordered data > mode. Opts: (null) > > > Could anyone help to solve this problem ? Sorry that it is really urgent. > > Thank you very much. > > T.H.Hsieh > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3
Hello all, I would appreciate if the community would give more attention to this issue because upgrading from 2.12.x to 2.15.x, two LTS versions, is something that we can expect many cluster admin will try to do in the next few months... We ourselves plan to upgrade a small Lustre (production) system from 2.12.9 to 2.15.3 in the next couple of weeks... After seeing problems reports like this we start feeling a bit nervous... The documentation for doing this major update appears to me as not very specific... In this document for example, https://doc.lustre.org/lustre_manual.xhtml#upgradinglustre , the update process appears not so difficult and there is no mention of using "tunefs.lustre --writeconf" for this kind of update. Or am I missing something ? Thanks in advance for providing more tips for this kind of update. Martin Audet From: lustre-discuss on behalf of Tung-Han Hsieh via lustre-discuss Sent: September 23, 2023 2:20 PM To: lustre-discuss@lists.lustre.org Subject: [lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3 ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Dear All, Today we tried to upgrade Lustre file system from version 2.12.6 to 2.15.3. But after the work, we cannot mount MDT successfully. Our MDT is ldiskfs backend. The procedure of upgrade is 1. Install the new version of e2fsprogs-1.47.0 2. Install Lustre-2.15.3 3. After reboot, run: tunefs.lustre --writeconf /dev/md0 Then when mounting MDT, we got the error message in dmesg: === [11662.434724] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.584593] Lustre: 3440:0:(scrub.c:189:scrub_file_load()) chome-MDT: reset scrub OI count for format change (LU-16655) [11666.036253] Lustre: MGS: Logs for fs chome were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11666.523144] Lustre: chome-MDT: Imperative Recovery not enabled, recovery window 300-900 [11666.594098] LustreError: 3440:0:(mdd_device.c:1355:mdd_prepare()) chome-MDD: get default LMV of root failed: rc = -2 [11666.594291] LustreError: 3440:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [11666.594951] Lustre: Failing over chome-MDT [11672.868438] Lustre: 3440:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1695492248/real 1695492248] req@5dfd9b53 x1777852464760768/t0(0) o251->MGC192.168.32.240@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1695492254 ref 2 fl Rpc:XNQr/0/ rc 0/-1 job:'' [11672.925905] Lustre: server umount chome-MDT complete [11672.926036] LustreError: 3440:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [11872.893970] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: (null) Could anyone help to solve this problem ? Sorry that it is really urgent. Thank you very much. T.H.Hsieh ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Cannot mount MDT after upgrading from Lustre 2.12.6 to 2.15.3
Dear All, Today we tried to upgrade Lustre file system from version 2.12.6 to 2.15.3. But after the work, we cannot mount MDT successfully. Our MDT is ldiskfs backend. The procedure of upgrade is 1. Install the new version of e2fsprogs-1.47.0 2. Install Lustre-2.15.3 3. After reboot, run: tunefs.lustre --writeconf /dev/md0 Then when mounting MDT, we got the error message in dmesg: === [11662.434724] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11662.584593] Lustre: 3440:0:(scrub.c:189:scrub_file_load()) chome-MDT: reset scrub OI count for format change (LU-16655) [11666.036253] Lustre: MGS: Logs for fs chome were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11666.523144] Lustre: chome-MDT: Imperative Recovery not enabled, recovery window 300-900 [11666.594098] LustreError: 3440:0:(mdd_device.c:1355:mdd_prepare()) chome-MDD: get default LMV of root failed: rc = -2 [11666.594291] LustreError: 3440:0:(obd_mount_server.c:2027:server_fill_super()) Unable to start targets: -2 [11666.594951] Lustre: Failing over chome-MDT [11672.868438] Lustre: 3440:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1695492248/real 1695492248] req@5dfd9b53 x1777852464760768/t0(0) o251->MGC192.168.32.240@o2ib@0@lo:26/25 lens 224/224 e 0 to 1 dl 1695492254 ref 2 fl Rpc:XNQr/0/ rc 0/-1 job:'' [11672.925905] Lustre: server umount chome-MDT complete [11672.926036] LustreError: 3440:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount : rc = -2 [11872.893970] LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: (null) Could anyone help to solve this problem ? Sorry that it is really urgent. Thank you very much. T.H.Hsieh ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
