Hello Lustre community, One of the OSTs failed. We have copied the data using rsync and reformatted the OST with the following command:
mkfs.lustre --ost --index=29 --replace --reformat --fsname=lustre --backfstype=ldiskfs --mgsnode=X.X.X.X@o2ib --mkfsoptions="-i 7000" /dev/sdd The OST has been deactivated on both the client side and the MDS side. We mounted it with the following command and copied the data back using rsync: mount -t ldiskfs /dev/sdd /mnt/ost29 Once the copying was completed, I tried to mount it as a Lustre OST, but I encountered the following errors: # mount -t lustre /dev/sdd /mnt/29ost mount.lustre: mount /dev/sdd at /mnt/29ost failed: File exists When I tried a second time, I got this error: # mount -t lustre /dev/sdd /mnt/29ost mount.lustre: mount /dev/sdd at /mnt/29ost failed: Operation now in progress I also tried: # mount -t lustre -o abort_recov /dev/sdd /mnt/ost29 mount.lustre: mount /dev/sdd at /mnt/ost29 failed: File exists On the OSS, when I run: # lctl dl | grep OST001d 12 UP osd-ldiskfs lustre-OST001d-osd lustre-OST001d-osd_UUID 3 The OST still appears. I tried to remove it using: lctl --device lustre-OST001d-osd cleanup lctl --device lustre-OST001d-osd detach However, when I run mount -t lustre /dev/sdd /mnt/ost29 again, OST001d appears once more. I have also found Lustre-related errors in the OSS kernel logs. [74585.030434] LustreError: 15c-8: MGC10.1x.x.x.132@o2ib: The configuration > from log 'lustre-OST001d' failed (-115). This may be the result of > communication errors between this node and the MGS, a bad configuration, or > other errors. See the syslog for more information. > [74585.034979] LustreError: > 8737:0:(obd_mount_server.c:1397:server_start_targets()) failed to start > server lustre-OST001d: -115 > [74585.036522] LustreError: > 8737:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start > targets: -115 > [74585.037997] LustreError: 8737:0:(obd_config.c:610:class_cleanup()) > Device 13 not setup > [74585.039532] Lustre: server umount lustre-OST001d complete > [74585.039537] LustreError: 8737:0:(obd_mount.c:1608:lustre_fill_super()) > Unable to mount /dev/sdd (-115) > [74744.284257] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5 > [74745.013194] LDISKFS-fs (sdd): mounted filesystem with ordered data > mode. Opts: errors=remount-ro,no_mbcache,nodelalloc > [74745.477281] LustreError: > 8763:0:(obd_mount_server.c:110:server_register_mount()) Already registered > lustre-OST001d > [74745.478835] LustreError: > 8763:0:(obd_mount_server.c:1992:server_fill_super()) Unable to start > targets: -17 > [74745.480369] LustreError: > 8763:0:(obd_mount_server.c:1600:server_put_super()) no obd lustre-OST001d > [74746.236387] Lustre: server umount lustre-OST001d complete > [74746.236393] LustreError: 8763:0:(obd_mount.c:1608:lustre_fill_super()) > Unable to mount (-17) > [74751.705099] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5 > [74752.436188] LDISKFS-fs (sdd): mounted filesystem with ordered data > mode. Opts: errors=remount-ro,no_mbcache,nodelalloc > [74752.902198] LustreError: 8776:0:(llog_osd.c:987:llog_osd_next_block()) > lustre-OST001d-osd: invalid llog tail at log id > [0xb46f60d6:0xa1cf2803:0x0]:0 offset 20672 last_rec idx 0 tail idx > 2250830848lrt len 2048 read_size 4288 > [74752.905177] LustreError: 8776:0:(llog.c:1419:llog_backup()) > MGC10.19.4.132@o2ib: failed to backup log lustre-OST001d: rc = -22 > [74752.908120] LustreError: > 8776:0:(mgc_request.c:1879:mgc_llog_local_copy()) MGC10.x.x.x.132@o2ib: > failed to copy remote log lustre-OST001d: rc = -22 > [74752.916309] LustreError: > 8788:0:(nodemap_storage.c:206:nodemap_cache_find_create()) cannot destroy > nodemap index: rc = -115 > [74753.304620] LustreError: 8788:0:(obd_config.c:559:class_setup()) setup > lustre-OST001d failed (-115) > [74753.306167] LustreError: > 8788:0:(obd_config.c:1835:class_config_llog_handler()) MGC10.19.4.132@o2ib: > cfg command failed: rc = -115 > [74753.309113] Lustre: cmd=cf003 0:lustre-OST001d 1:dev 2:0 3:f > > [74753.309227] LustreError: 15c-8: MGC10.19.4.132@o2ib: The configuration > from log 'lustre-OST001d' failed (-115). This may be the result of > communication errors between this node and the MGS, a bad configuration, or > other errors. See the syslog for more information. > [74753.313693] LustreError: > 8776:0:(obd_mount_server.c:1397:server_start_targets()) failed to start > server lustre-OST001d: -115 > [74753.315270] LustreError: 8776:0:(obd_config.c:610:class_cleanup()) > Device 13 not setup > [75312.334648] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5 > [75313.081093] LDISKFS-fs (sdd): mounted filesystem with ordered data > mode. Opts: errors=remount-ro,no_mbcache,nodelalloc > [75313.081112] LustreError: 8822:0:(osd_handler.c:7815:osd_mount()) > lustre-OST001d-osd: failed to set lma on /dev/sdd root inode > [75313.082671] LustreError: 8822:0:(obd_config.c:559:class_setup()) setup > lustre-OST001d-osd failed (-30) > [75313.084163] LustreError: 8822:0:(obd_mount.c:202:lustre_start_simple()) > lustre-OST001d-osd setup error -30 > [75313.085645] LustreError: > 8822:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on > /dev/sdd: -30 > [75313.087122] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super()) > Unable to mount (-30) > [75313.088583] LustreError: 8822:0:(obd_mount.c:1608:lustre_fill_super()) > Skipped 1 previous similar message > [75330.678571] LDISKFS-fs (sdd): file extents enabled, maximum tree depth=5 > [75331.407996] LDISKFS-fs (sdd): mounted filesystem with ordered data > mode. Opts: errors=remount-ro,no_mbcache,nodelalloc > [75331.408015] LustreError: 8837:0:(osd_handler.c:7815:osd_mount()) > lustre-OST001d-osd: failed to set lma on /dev/sdd root inode > [75331.409564] LustreError: 8837:0:(obd_config.c:559:class_setup()) setup > lustre-OST001d-osd failed (-30) > [75331.411040] LustreError: 8837:0:(obd_mount.c:202:lustre_start_simple()) > lustre-OST001d-osd setup error -30 > [75331.412528] LustreError: > 8837:0:(obd_mount_server.c:1958:server_fill_super()) Unable to start osd on > /dev/sdd: -30 You guys' support will be highly appreciated as we are stuck very badly. lustre version is 2.12.6 . Regards, Ihsan Ur Rahman
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
