Unfortunately, I don't have any more time to figure this out. I opted to just install lustre 2.15.1, because that is the one that seems to work for me.
I have attached my install procedure for lustre 2.15.7, is there something I should be doing differently? Gr, Martin Balvers ________________________________ From: Andreas Dilger <[email protected]> Sent: Friday, August 01, 2025 22:58 To: BALVERS Martin <[email protected]> Cc: lustre-discuss <[email protected]> Subject: Re: [lustre-discuss] No ZFS compression on OST's when using PFL WARNING - EXTERNAL SENDER - BE CYBERSAFE You are changing a number of different things with each of your tests, not just Lustre. It could be ZFS 2.1.5->2.1.15 or Alma 8.6->8.10 is also causing issues... Hard to say for sure without apples-to-apples comparisons. It should be possible to build Lustre 2.15.5 on Alma 8.6 + ZFS 2.1.5 to help isolate the issue. There was LU-19193<https://urldefense.com/v3/__https://jira.whamcloud.com/browse/LU-19193__;!!OUGTln_Lrg!Ugs-QNv1jVaQ0FZzMnQ9mLWiu0QW-B3iSDiiXoIU2XwGrg2ab0Uh-J-krG6w9pYI3ffwwwLgjxtjhLeNfA$> filed recently about ZFS and blocksize detection, which might hurt ZFS compression, but that was never landed on 2.15. Cheers, Andreas On Aug 1, 2025, at 06:33, BALVERS Martin via lustre-discuss <[email protected]> wrote: I ended up installing lustre 2.15.1 on the new servers. Now compression works as expected when PFL is enabled. What I have tested so far is: AlmaLinux 9.4 - Lustre 2.16.1 - ZFS 2.1.16 - No compression with PFL AlmaLinux 8.10 - Lustre 2.15.7 - ZFS 2.2.8 - No compression with PFL AlmaLinux 8.10 - Lustre 2.15.5 - ZFS 2.1.15 - No compression with PFL AlmaLinux 8.6 - Lustre 2.15.1 - ZFS 2.1.5 - PFL + compression works as expected I tried installing 2.15.2, 2.15.3 and 2.15.4 but that failed with ksym errors. It seems that somewhere between lustre 2.15.1 and 2.15.5 the PFL + compression breaks. gr, Martin Balvers Cheers, Andreas — Andreas Dilger Lustre Principal Architect Whamcloud/DDN Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits. This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.
### SETUP NODES # Install AlmaLinux 8.10 from minimal iso and configure networking etc. # For any server, disable selinux! nano /etc/selinux/config SELINUX=disabled dnf install epel-release dnf install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm # Disable [zfs] and enable [zfs-kmod] repos nano /etc/yum.repos.d/zfs.repo dnf install zfs nano /etc/yum.repos.d/lustre.repo [lustre-server] name=lustre-server baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.7/el8.10/server/ enabled=1 exclude=*debuginfo* gpgcheck=0 # to allow port 988 firewall-cmd --zone=public --add-port 988/tcp --permanent && firewall-cmd --reload # create zfs.conf file # by default zfs takes max 50% memory, set zfs_arc_max to 80-90% of total memory (in bytes) Use 'free -b' tot find the total memory in bytes nano /etc/modprobe.d/zfs.conf ### OSS # Optimized ZFS settings for Lustre OST with HDD mirrors and 265GB RAM options zfs zfs_arc_max=227633266688 options zfs zfs_arc_min=8589934592 options zfs zfs_prefetch_disable=1 options zfs metaslab_debug_unload=0 options zfs zfs_vdev_cache_size=0 options zfs zfs_read_chunk_size=1048576 options zfs zfs_vdev_async_write_min_active=10 options zfs zfs_vdev_async_write_max_active=30 options zfs zfs_vdev_async_write_active_min_dirty_percent=15 options zfs zfs_vdev_scheduler=deadline reboot dnf install lustre kmod-lustre kmod-lustre-osd-zfs ### setup OSS # create the zfs pool zpool create -o ashift=12 -f lustre -o cachefile=none -O acltype=posixacl -O compression=zstd -O xattr=sa -O canmount=off mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf mirror /dev/sdg /dev/sdh mirror /dev/sdi /dev/sdj mirror /dev/sdk /dev/sdl # Create the lustre fs. Index needs to be unique for each OST mkfs.lustre --reformat --ost --backfstype=zfs --fsname=lustre --index=<i> --mgsnode=192.168.6.1@tcp lustre/ost mkdir /mnt/ost # mounting the pool starts lustre mount -t lustre_tgt -v lustre/ost /mnt/ost # Make sure compression is enabled: zfs set compression=zstd lustre/ost ## From a client lfs setstripe -E 1M -L mdt -E 128M -c 1 -S 4M -E 2G -c 2 -z 128M -E -1 -c -1 -z 256M /mnt/lustre_new
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
