Thanks for digging up the LUs that I somehow didn't manage to find.
Also, I noticed that the XFS trace was for the case where Lustre is fast
(walking/creating the tree and rename files).
I've attached the other case (dest: project0/d1) for XFS. If I'm
interpreting it right it seems that on XFS mv is using reflinks for the
copy:
ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = 0
That explains why it is fast on XFS and a full copy on Lustre which
doesn't support this operation.
I'm not sure I see the logic in the mv implementation. It should be
possible to handle both cases by walking/creating the tree and rename
files. I will dig deeper into the mv code to figure out what is going on.
Cheers,
Hans Henrik
On 17/12/2025 21.15, Andreas Dilger wrote:
Looking at the strace output, it appears in all cases that the *directory*
rename is returning EXDEV, as it should be the case (to preserve the projid
for that directory tree). In the XFS case, it is refusing to do the file
rename across projects also (returning EXDEV and forcing userspace to do the
file copy, which is very inefficient), while Lustre allows the file rename
to succeed (just reassinging the project quota usage).
In the lustre-mv-copy case, it appears that "mv" is not trying to rename the
file again after it creates the target directory, as it does in the lustre-mv
case. Instead it just copies the file, which seems like a bug in "mv".
It looks like the renameat2() flags have not been implemented yet:
https://jira.whamcloud.com/browse/LU-12272 - RENAME_NOREPLACE
https://jira.whamcloud.com/browse/LU-11557 - RENAME_EXCHANGE
https://jira.whamcloud.com/browse/LU-19720 - RENAME_WHITEOUT
Probably implementing RENAME_NOREPLACE support would be the most important
and useful flag, since "mv" is always trying this first. RENAME_EXCHANGE
could be useful for some applications, while RENAME_WHITEOUT is for overlayfs.
Cheers, Andreas
On Dec 17, 2025, at 05:54, Hans Henrik Happe via
lustre-discuss<[email protected]> wrote:
To be more specific:
Server:
- Lustre 2.15.7
- Rocky Linux 8.10 (4.18.0-553.53.1.el8_10.x86_64)
Client:
- Lustre 2.15.7, but same outcome on 2.15.5
- Rocky Linux 9.6 (5.14.0-570.42.2.el9_6.x86_64)
- glibc-2.34-168.el9_6.23.x86_64
- coreutils-8.32-39.el9.x86_64
On 17/12/2025 13.22, Hans Henrik Happe via lustre-discuss wrote:
It's 2.15.7.
On 17/12/2025 03.36, Andreas Dilger wrote:
There have definitely been some fixes in this area.
What version of Lustre are you running?
On Dec 16, 2025, at 07:39, Hans Henrik Happe via
lustre-discuss<[email protected]> wrote:
Hi,
It seems like the inherited project quota issue [1] with mv has been resolved
with XFS in RHEL9. However, Lustre is a bit strange.
Given a "src" and "dst"(empty) dir these two ways of calling mv behaves
differently:
mv src dst/ (work: no copy)
mv src dst/src (copy, then delete)
I've attached an strace output for both Lustre and XFS. It seems like mv is
handling the fact that Lustre don't have renameat2 a bit differently.
Before I dig further. Is this behavior known?
Cheers,
Hans Henrik
[1]http://lists.lustre.org/pipermail/lustre-discuss-lustre.org/2023-February/018511.html
<lustre-mv.strace><xfs-mv.strace><lustre-mv-copy.strace>_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Cheers,
Hans Henrik
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Andreas Dilger
Principal Lustre Architect
[email protected]
renameat2(AT_FDCWD, "project1/d1", AT_FDCWD, "project0/d1", RENAME_NOREPLACE) =
-1 EXDEV (Invalid cross-device link)
newfstatat(AT_FDCWD, "project0/d1", 0x7ffecf195440, 0) = -1 ENOENT (No such
file or directory)
newfstatat(AT_FDCWD, "project1/d1", {st_mode=S_IFDIR|0755, st_size=26, ...},
AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "project0/d1", 0x7ffecf194fa0, AT_SYMLINK_NOFOLLOW) = -1
ENOENT (No such file or directory)
rmdir("project0/d1") = -1 ENOENT (No such file or directory)
mkdir("project0/d1", 0700) = 0
newfstatat(AT_FDCWD, "project0/d1", {st_mode=S_IFDIR|0700, st_size=6, ...},
AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "project1/d1", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=26, ...}) = 0
getdents64(3, 0x561e74912300 /* 4 entries */, 32768) = 96
getdents64(3, 0x561e74912300 /* 0 entries */, 32768) = 0
close(3) = 0
newfstatat(AT_FDCWD, "project1/d1/d2", {st_mode=S_IFDIR|0755, st_size=16, ...},
AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "project0/d1/d2", 0x7ffecf194a50, AT_SYMLINK_NOFOLLOW) =
-1 ENOENT (No such file or directory)
rmdir("project0/d1/d2") = -1 ENOENT (No such file or directory)
mkdir("project0/d1/d2", 0700) = 0
newfstatat(AT_FDCWD, "project0/d1/d2", {st_mode=S_IFDIR|0700, st_size=6, ...},
AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "project1/d1/d2", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) =
3
fstat(3, {st_mode=S_IFDIR|0755, st_size=16, ...}) = 0
getdents64(3, 0x561e74912300 /* 3 entries */, 32768) = 72
getdents64(3, 0x561e74912300 /* 0 entries */, 32768) = 0
close(3) = 0
newfstatat(AT_FDCWD, "project1/d1/d2/f2", {st_mode=S_IFREG|0644,
st_size=2147483648, ...}, AT_SYMLINK_NOFOLLOW) = 0
newfstatat(AT_FDCWD, "project0/d1/d2/f2", 0x7ffecf194500, AT_SYMLINK_NOFOLLOW)
= -1 ENOENT (No such file or directory)
unlink("project0/d1/d2/f2") = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "project1/d1/d2/f2", O_RDONLY|O_NOFOLLOW) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2147483648, ...}) = 0
openat(AT_FDCWD, "project0/d1/d2/f2", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4
fstat(4, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = 0
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
