Hi Andreas,
Thank you so much for the information provided. Without it, we might
have struggled to find a solution for an extended period.
I attempted to rebuild the OI using LFSCK, but experienced hang-ups and
Lustre crashes while executing "lctl lfsck_start." Later, we were able
to address the OI issue by using the "-o resetoi" option during MDT
mounting:
mount -o resetoi /mdt00
After resolving OI issue, I was then able to run LFSCK to conduct a
comprehensive scan of the MDT.
Thanks again,
Jane
On 2023-05-03 14:52, Andreas Dilger wrote:
This looks like https://jira.whamcloud.com/browse/LU-16655 [2] causing
problems after the upgrade from 2.12.x to 2.15.[012] breaking the
Object Index files.
A patch for this has already been landed to b2_15 and will be included
in 2.15.3. If you've hit this issue, then you need to backup/delete
the OI files (off of Lustre) and run OI Scrub to rebuild them.
I believe the OI Scrub/rebuild is described in the Lustre Manual.
Cheers, Andreas
On May 3, 2023, at 09:30, Colin Faber via lustre-discuss
<lustre-discuss@lists.lustre.org> wrote:
Hi,
What does your client log indicate? (dmesg / syslog)
On Wed, May 3, 2023, 7:32 AM Jane Liu via lustre-discuss
<lustre-discuss@lists.lustre.org> wrote:
Hello,
I'm writing to ask for your help on one issue we observed after a
major
upgrade of a large Lustre system from RHEL7 + 2.12.9 to RHEL8 +
2.15.2.
Basically we preserved MDT disk (VDisk on a VM) and also all OST
disk
(JBOD) in RHEL7 and then reinstalled RHEL8 OS and then attached
those
preserved disks to RHEL8 OS. However, I met an issue after the OS
upgrade and lustre installation.
I believe the issue is related to metadata.
The old MDS was a virtual machine, and the MDT vdisk was preserved
during the upgrade. When a new VM was created with the same
hostname and
IP, the preserved MDT vdisk was attached to it. Everything seemed
fine
initially. However, after the client mount was completed, the file
listing displayed question marks, as shown below:
[root@experimds01 ~]# mount -t lustre 11.22.33.44@tcp:/experi01
/mntlustre/
[root@experimds01 ~]# cd /mntlustre/
[root@experimds01 mntlustre]# ls -l
ls: cannot access 'experipro': No such file or directory
ls: cannot access 'admin': No such file or directory
ls: cannot access 'test4': No such file or directory
ls: cannot access 'test3': No such file or directory
total 0
d????????? ? ? ? ? ? admin
d????????? ? ? ? ? ? experipro
-????????? ? ? ? ? ? test3
-????????? ? ? ? ? ? test4
I shut down the MDT and ran "e2fsck -p
/dev/mapper/experimds01-experimds01". It reported "primary
superblock
features different from
backup, check forced."
[root@experimds01 ~]# e2fsck -p
/dev/mapper/experimds01-experimds01
experi01-MDT0000 primary superblock features different from
backup,
check forced.
experi01-MDT0000: 9493348/429444224 files (0.5% non-contiguous),
109369520/268428864 blocks
Running e2fsck again showed that the filesystem was clean.
[root@experimds01 /]# e2fsck -p
/dev/mapper/experimds01-experimds01
experi01-MDT0000: clean, 9493378/429444224 files,
109369610/268428864
blocks
However, the issue persisted. The file listing continued to
display
question marks.
Do you have any idea what could be causing this problem and how to
fix
it? By the way, I have an e2image backup of the MDT from the
RHEL7 system just in case we need fix it using the backup. Also,
after
the upgrade, the command "lfs df" shows that all OSTs and MDT
are fine.
Thank you in advance for your assistance.
Best regards,
Jane
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org [1]
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org [1]
Links:
------
[1]
https://urldefense.com/v3/__http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org__;!!P4SdNyxKAPE!EjGkARsk0Yp3SMoPZG3ShFH-7yqJP8eiiu01CbPIWhkz9haU5QrbeCbYwZNFupImfJ2pe5WFfJT-Ci0S-Qw0nsps$
[2]
https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-16655__;!!P4SdNyxKAPE!EjGkARsk0Yp3SMoPZG3ShFH-7yqJP8eiiu01CbPIWhkz9haU5QrbeCbYwZNFupImfJ2pe5WFfJT-Ci0S-fnmcNE6$
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
