Hello Mr. Andreas Dilger, I am looping you with the hope that my issue will be fixed.
Regards, Ihsan Ur Rahman On Tue, Oct 21, 2025 at 12:06 AM Ihsan Ur Rahman <[email protected]> wrote: > Hello Lustre community, > > One of the OSTs failed. We have copied the data using rsync and > reformatted the OST with the following command: > > mkfs.lustre --ost --index=29 --replace --reformat --fsname=lustre > --backfstype=ldiskfs --mgsnode=X.X.X.X@o2ib --mkfsoptions="-i 7000" > /dev/sdd > > > The OST has been deactivated on both the client side and the MDS side. We > mounted it with the following command and copied the data back using rsync: > > mount -t ldiskfs /dev/sdd /mnt/ost29 > > > Once the copying was completed, I tried to mount it as a Lustre OST, but I > encountered the following errors: > > # mount -t lustre /dev/sdd /mnt/29ost > mount.lustre: mount /dev/sdd at /mnt/29ost failed: File exists > > > When I tried a second time, I got this error: > > # mount -t lustre /dev/sdd /mnt/29ost > mount.lustre: mount /dev/sdd at /mnt/29ost failed: Operation now in > progress > > > I also tried: > > # mount -t lustre -o abort_recov /dev/sdd /mnt/ost29 > mount.lustre: mount /dev/sdd at /mnt/ost29 failed: File exists > > > On the OSS, when I run: > > # lctl dl | grep OST001d > 12 UP osd-ldiskfs lustre-OST001d-osd lustre-OST001d-osd_UUID 3 > > > The OST still appears. I tried to remove it using: > > lctl --device lustre-OST001d-osd cleanup > lctl --device lustre-OST001d-osd detach > > > However, when I run mount -t lustre /dev/sdd /mnt/ost29 again, OST001d > appears once more. > > I have also found Lustre-related errors in the OSS kernel logs. > > [74585.030434] LustreError: 15c-8: MGC10.1x.x.x.132@o2ib: The >> configuration from log 'lustre-OST001d' failed (-115). This may be the >> result of communication errors between this node and the MGS, a bad >> configuration, or other errors. See the syslog for more information. >> [74585.034979] LustreError: >> 8737:0:(obd_mount_server.c:1397:server_start_targets()) failed to start >> server lustre-OST001d: -115 >> [74585.036522] LustreError: >> 8737:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start >> targets: -115 >> [74585.037997] LustreError: 8737:0:(obd_config.c:610:class_cleanup()) >> Device 13 not setup >> [74585.039532] Lustre: server umount lustre-OST001d complete >> [74585.039537] LustreError: 8737:0:(obd_mount.c:1608:lustre_fill_super()) >> Unable to mount /dev/sdd (-115) >> [74744.284257] LDISKFS-fs (sdd): file extents enabled, maximum tree >> depth=5 >> [74745.013194] LDISKFS-fs (sdd): mounted filesystem with ordered data >> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc >> [74745.477281] LustreError: >> 8763:0:(obd_mount_server.c:110:server_register_mount()) Already registered >> lustre-OST001d >> [74745.478835] LustreError: >> 8763:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start >> targets: -17 >> [74745.480369] LustreError: >> 8763:0:(obd_mount_server.c:1600:server_put_super()) no obd lustre-OST001d >> [74746.236387] Lustre: server umount lustre-OST001d complete >> [74746.236393] LustreError: 8763:0:(obd_mount.c:1608:lustre_fill_super()) >> Unable to mount (-17) >> [74751.705099] LDISKFS-fs (sdd): file extents enabled, maximum tree >> depth=5 >> [74752.436188] LDISKFS-fs (sdd): mounted filesystem with ordered data >> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc >> [74752.902198] LustreError: 8776:0:(llog_osd.c:987:llog_osd_next_block()) >> lustre-OST001d-osd: invalid llog tail at log id >> [0xb46f60d6:0xa1cf2803:0x0]:0 offset 20672 last_rec idx 0 tail idx >> 2250830848lrt len 2048 read_size 4288 >> [74752.905177] LustreError: 8776:0:(llog.c:1419:llog_backup()) >> MGC10.19.4.132@o2ib: failed to backup log lustre-OST001d: rc = -22 >> [74752.908120] LustreError: >> 8776:0:(mgc_request.c:1879:mgc_llog_local_copy()) MGC10.x.x.x.132@o2ib: >> failed to copy remote log lustre-OST001d: rc = -22 >> [74752.916309] LustreError: >> 8788:0:(nodemap_storage.c:206:nodemap_cache_find_create()) cannot destroy >> nodemap index: rc = -115 >> [74753.304620] LustreError: 8788:0:(obd_config.c:559:class_setup()) setup >> lustre-OST001d failed (-115) >> [74753.306167] LustreError: >> 8788:0:(obd_config.c:1835:class_config_llog_handler()) MGC10.19.4.132@o2ib: >> cfg command failed: rc = -115 >> [74753.309113] Lustre: cmd=cf003 0:lustre-OST001d 1:dev 2:0 3:f >> >> [74753.309227] LustreError: 15c-8: MGC10.19.4.132@o2ib: The >> configuration from log 'lustre-OST001d' failed (-115). This may be the >> result of communication errors between this node and the MGS, a bad >> configuration, or other errors. See the syslog for more information. >> [74753.313693] LustreError: >> 8776:0:(obd_mount_server.c:1397:server_start_targets()) failed to start >> server lustre-OST001d: -115 >> [74753.315270] LustreError: 8776:0:(obd_config.c:610:class_cleanup()) >> Device 13 not setup >> [75312.334648] LDISKFS-fs (sdd): file extents enabled, maximum tree >> depth=5 >> [75313.081093] LDISKFS-fs (sdd): mounted filesystem with ordered data >> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc >> [75313.081112] LustreError: 8822:0:(osd_handler.c:7815:osd_mount()) >> lustre-OST001d-osd: failed to set lma on /dev/sdd root inode >> [75313.082671] LustreError: 8822:0:(obd_config.c:559:class_setup()) setup >> lustre-OST001d-osd failed (-30) >> [75313.084163] LustreError: >> 8822:0:(obd_mount.c:202:lustre_start_simple()) lustre-OST001d-osd setup >> error -30 >> [75313.085645] LustreError: >> 8822:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on >> /dev/sdd: -30 >> [75313.087122] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super()) >> Unable to mount (-30) >> [75313.088583] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super()) >> Skipped 1 previous similar message >> [75330.678571] LDISKFS-fs (sdd): file extents enabled, maximum tree >> depth=5 >> [75331.407996] LDISKFS-fs (sdd): mounted filesystem with ordered data >> mode. Opts: errors=remount-ro,no_mbcache,nodelalloc >> [75331.408015] LustreError: 8837:0:(osd_handler.c:7815:osd_mount()) >> lustre-OST001d-osd: failed to set lma on /dev/sdd root inode >> [75331.409564] LustreError: 8837:0:(obd_config.c:559:class_setup()) setup >> lustre-OST001d-osd failed (-30) >> [75331.411040] LustreError: >> 8837:0:(obd_mount.c:202:lustre_start_simple()) lustre-OST001d-osd setup >> error -30 >> [75331.412528] LustreError: >> 8837:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on >> /dev/sdd: -30 > > > > You guys' support will be highly appreciated as we are stuck very badly. > lustre version is 2.12.6 . > > Regards, > > Ihsan Ur Rahman > > > > >
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
