Thank you. I checked all those files with "ll_decode_filter_fid" and "lfs fid2path":
On OSS servers: # umount /lustre/ost01 # mount -t ldiskfs /dev/mapper/ost01 /mnt/ # find /mnt/O/0/d*/ -type f /mnt/O/0/d11/450605003 /mnt/O/0/d12/450605484 . . . # ll_decode_filter_fid /mnt/O/0/d11/450605003 /mnt/O/0/d11/450605003: parent=[0x200019425:0x733f:0x0] stripe=0 stripe_size=1048576 stripe_count=1 layout_version=0 range=0 # umount /mnt # mount -t lustre /dev/mapper/ost01 /lustre/ost01 On client: # lfs fid2path /cluster [0x200019425:0x733f:0x0] lfs fid2path: cannot find /cluster [0x200019425:0x733f:0x0]: No such file or directory I got a reply "No such file or directory" for all those unknown files. So I can assume those are "stray objects from deleted files". Best regards, Pavlo Khmel > On 20 Jan 2024, at 04:45, Andreas Dilger <adil...@whamcloud.com> wrote: > > > >> On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss >> <lustre-discuss@lists.lustre.org <mailto:lustre-discuss@lists.lustre.org>> >> wrote: >> >> Hi, >> >> I'm trying to remove 4 OSTs. >> >> # lfs osts >> OBDS: >> 0: cluster-OST0000_UUID ACTIVE >> 1: cluster-OST0001_UUID ACTIVE >> 2: cluster-OST0002_UUID ACTIVE >> 3: cluster-OST0003_UUID ACTIVE >> . . . >> >> I moved all files to other OSTs. "lfs find" cannot find any files on these 4 >> OSTs. >> >> # time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster >> >> real 936m8.528s >> user 13m48.298s >> sys 210m1.245s >> >> But still: 2624 inods are in use and 14.5G total size. >> >> # lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003 >> cluster-OST0000_UUID 4293438576 644 4293437932 1% /cluster[OST:0] >> cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1] >> cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2] >> cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3] >> >> # lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003 >> cluster-OST0000_UUID 29.2T 3.8G 27.6T 1% /cluster[OST:0] >> cluster-OST0001_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:1] >> cluster-OST0002_UUID 29.2T 3.3G 27.6T 1% /cluster[OST:2] >> cluster-OST0003_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:3] >> >> I tried to check the file-system for errors: >> >> # umount /lustre/ost01 >> # e2fsck -fy /dev/mapper/ost01 >> >> and >> >> # lctl lfsck_start --device cluster-OST0001 >> # lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub >> . . . >> status: completed >> >> I tried to mount OST as ldiskfs and there are several files in /O/0/d*/ >> >> # umount /lustre/ost01 >> # mount -t ldiskfs /dev/mapper/ost01 /mnt/ >> # ls -Rhl /mnt/O/0/d*/ >> . . . >> /mnt/O/0/d11/: >> -rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003 >> /mnt/O/0/d12/: >> -rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028 >> -rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484 >> . . . >> >> Is it expected behavior? Is it save to delere OST even with those files? > > > You can run the debugfs "stat" command to print the "fid" xattr and it will > print the MDT > parent FID for use with "lfs fid2path" on the client to see if there are any > files related > to these objects. You could also run "ll_decode_filter_fid" to do the same > thing on the > mounted ldiskfs filesystem. > > It is likely that there are a few stray objects from deleted files, but hard > to say for sure. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > >
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org