Re: XFS vs Ext4
On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote: > > We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We > have used ext4 for the past 10 years, however we are considering using XFS > for its better performance with larger files. Which file system do you use > for your lab? > Historical background. XFS filesystem with the companion XLV logical volume manager (aka "partitioning tool") came to Linux from SGI IRIX, where it was developed circa late-1990-ies. XFS was copied to Linux verbatim (initially with shims and kludges, later, fully integrated). XLV was reimplemented as LVM. The EXT series of filesystems were developed together with the linux kernel (first ext filesystem may have originated with MINIX, look it up). As improvements were made, journaling, no need to fsck after crash, online grow/shrink, etc, they were renamed ext2/ext3/ext4 and they are still largely compatible between themselves. For many purposes, both filesystems are obsoleted by ZFS, which added: - added metadata and data checksums - to detect silent bit rot on current-generation HDDs and SSDs - added online filesystem check - for broken data, gives you list of filenames instead of inode numbers - added "built-in" mirroring - together with checksums, online fsck (zfs scrub) and monthly zfs scrub cron job, allows automatic healing of bit rot. - added "built-in" raid-5 and raid-6 - again, together with checksums and online fsck, allows automatic healing and robust operation in presence of disk bad sectors, I/O errors, corruption and single-disk failure. - other goodies like snapshots, large ram cache, dedup, online compression, etc are taken for granted for current generation filesystems. On current generation HDDs and SSds, use of bare XFS and ext4 is dangerous, SSD failure or "HDD grows bad sectors" will destroy your data completely. On current generation HDDs, use of mirrored XFS and ext4 is dangerous (using mdadm or LVM mirroring), (a) bit rot inevitably causes differences between data between the two disks. Lacking checksums, mdadm and LVM mirroring cannot decide which of the two copies is the correct one. (b) after a crash, mirror rebuild fill fail if both disks happen to have bad sectors (or throw random I/O errors). Ditto for RAID5 and RAID6, probability of RAID rebuild failing because multiple disks have have sectors and I/O errors goes up with the number of disks. ZFS was invented to resolve all these problems. (BTRFS was invented as a NIH erzatz ZFS, is still incomplete wrt RAID5/RAID6). Bottom line, if you can, use ZFS. Current Ubuntu installer has a button "install on ZFS", use it! -- Konstantin Olchanski Data Acquisition Systems: The Bytes Must Flow! Email: olchansk-at-triumf-dot-ca Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada
Re: XFS vs Ext4
On Tue, Dec 5, 2023 at 1:51 AM Paul Robert Marino wrote: > With NFS always go with EXT4 because NFS isn't compatible with 64bit inodes, so you need to disable a flag with XFS for "inode64" which means on files over 2GB XFS' will need to create multiple inodes instead of 1 at the beginning of the file which hurts it's performance. Can you provide details on this, or a reference? Even NFSv2 had fixed-size 32-byte/256-bit file handles. NFSv3 (1995) made them variable size and up to 64 bytes / 512 bits. Linux NFS may have taken a while to catch up, but I am pretty sure anything from the past decade will be fine with 64 bit inodes. We have been using XFS over NFS for at least 10 years now, with partitions in the hundreds of terabytes and individual files in the hundreds of gigabytes. No problem. I have no idea why anyone would consider ext4 for any serious application. - Pat
Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4
Although CIS compliance (for "security" of assets) may be a requirement, it is not infallible and often gives a false sense of being secure from compromises. Example: https://urldefense.proofpoint.com/v2/url?u=https-3A__techcrunch.com_2023_03_16_cisa-2Dnation-2Dstate-2Dhackers-2Dbreach-2Dfederal-2Dagency=DwIDaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=6PchYtJmUCDWyWETQdqlMWUVc480pT-DgabXWg_139Oa_NTleIYH_GXzCxcXwLCt=T_qwom_-eNAdMLSWz93PZyzR-ZfY2DhKqc79YZ4mCqE= and there are some discussions at Black Hat. Whenever a compromise on a CIS compliant system is accomplished (sometimes by a nation-state clandestine service operation/agent), vulnerabilities are discovered that have to be corrected in the compliance testing applications. Given how ancient SL is at this time, I am curious under which situations it is still used -- dedicated systems that cannot be changed to a more "current" "enterprise" Linux (such as Ubuntu LTS)? -Original Message- From: Nico Kadel-Garcia [mailto:nka...@gmail.com] Sent: Tuesday, December 05, 2023 01:19 PM -08 To: Laura Hild Cc: scientific-linux-users Subject: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4 On Tue, Dec 5, 2023 at 9:06 AM Laura Hild wrote: No! No. No LVM! Bad admin, no biscuit! [...] *Bad* admin. Where's my squirt bottle? Yeah, I wrote "if you're a slice-and-dicer" for a reason. One big root is a fine option, but it's not the situation I was imagining where one is concerned with shrinkability. I think having hard limits on the growth of certain subtrees can be helpful, and sometimes different mount options. I'm Kickstarting rather than imaging, so I don't have a problem including the hostname in the name of the Volume Group. Everyone has different objectives (and I'm skeptical lack of LVM would have adequately protected you from your predecessors' and colleagues' :)). Various options for various filesystems is considered a big deal CIS compliance. I consider it a completely destabilizing waste of time better spent elsewhere. Avoiding LVM and its tendency to use identical volume names in VM images is a problem in cloud or VM environments where you may wish to mount and access a previous snapshot of the same VM. Itt's much easier to cut sections off a long rope than try to tie a bunch of short pieces to make the rope you need, when trying to tie up loose ends.
Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4
On Tue, Dec 5, 2023 at 9:06 AM Laura Hild wrote: > > > No! No. No LVM! Bad admin, no biscuit! > > [...] > > *Bad* admin. Where's my squirt bottle? > > Yeah, I wrote "if you're a slice-and-dicer" for a reason. One big root is a > fine option, but it's not the situation I was imagining where one is > concerned with shrinkability. I think having hard limits on the growth of > certain subtrees can be helpful, and sometimes different mount options. I'm > Kickstarting rather than imaging, so I don't have a problem including the > hostname in the name of the Volume Group. Everyone has different objectives > (and I'm skeptical lack of LVM would have adequately protected you from your > predecessors' and colleagues' :)). Various options for various filesystems is considered a big deal CIS compliance. I consider it a completely destabilizing waste of time better spent elsewhere. Avoiding LVM and its tendency to use identical volume names in VM images is a problem in cloud or VM environments where you may wish to mount and access a previous snapshot of the same VM. Itt's much easier to cut sections off a long rope than try to tie a bunch of short pieces to make the rope you need, when trying to tie up loose ends. >
Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4
> No! No. No LVM! Bad admin, no biscuit! > [...] > *Bad* admin. Where's my squirt bottle? Yeah, I wrote "if you're a slice-and-dicer" for a reason. One big root is a fine option, but it's not the situation I was imagining where one is concerned with shrinkability. I think having hard limits on the growth of certain subtrees can be helpful, and sometimes different mount options. I'm Kickstarting rather than imaging, so I don't have a problem including the hostname in the name of the Volume Group. Everyone has different objectives (and I'm skeptical lack of LVM would have adequately protected you from your predecessors' and colleagues' :)).
Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4
We use EXT4 mostly. We find it to be more robust for lab machines, and more efficient for VMs. XFS is limited to cases where its more advanced features are useful. We treat our user facing systems as being "disposable", if they develop a software/filesystem issue we usually wipe and reinstall rather than try to fix same. Important storage is mounted from separate storage servers.
Re: XFS vs Ext4
XFS is a from the ground up journaling filesystem, whereas EXT4 is still a journal tacked on filesystem. That's said EXT4 has caught up to XFS and in some specific cases exceeded its performance but not all. The short version is it depends on what you are doing both have pros and cons, Here are some short examples If I'm doing a Gluster object store I will always use XFS because Gluster is optimized for XFS, If it's an NFS volume then EXT4 is usually better because of inode compatibility issues which can be worked around in XFS but tank the performance. If I'm using it for temp space for compiles or an ETL, XFS will probably do better because it will reduce the IOPS around the handling of inodes. Extremely large volumes (100TB+ on a SAN or in a RAID) I always will use XFS. Desktop pick the one you know the best, but if undelete is important to you use EXT4 (XFS has no mechanism for undeleting files). if you are dealing with an embedded device that boots off of an SD card or EMMC, XFS if possible because how it handles inodes puts less wear on it over time. High risk of file corruption due to unreliable power or hardware always XFS The long version There is a huge efficiency gain when deleting files in XFS. It's noticeably faster when freeing up space from deleting files; this is because instead of having an inode per block it only creates 1 inode at the beginning of every file as needed for legacy application backward support. Another positive side effect of his formatting an XFS filesystem is faster because it never pre-creates inodes during the formatting process, infact the only thing mkfs.xfs does is create a couple of redundant journals. The 1 inode per file is also the reason XFS puts less wear on SSD's, This difference wont make a noticeable impact on the life of a business class SSD, and probably not in a decent consumed grade SSD, but for SD cards and EMMC being used as a boot device this impact can be huge especially if that device isn't using tmpfs for /tmp or is storing /var on the SSD and is writing a lot of logs and temp files. Risk of file system corruption due to unplanned power outages XFS, because it keeps multiple copies of the journal, ignores the inodes for recovery and generally is self healing. While they both do an amazing job at recovering from this kind of situation, EXT4 still trusts the inodes over its single copy of the journal and as a result is more likely to have significant file corruption issues sooner. fsck.xfs normally only does a comparison of the copies of the journal to fix journal corruption issues and will usually roll back file changes that were incomplete at the time of the power loss. By the way XFS does this automatically when the volumes are mounted if the checksums of the copies of the journal don't match so essentially it's normally a placebo command which you should never need to run unless you are dealing with bad sectors in the physical media. EXT4 handles things way differently on mount and as a result you may need to manually run fsck.ext4 after a power loss to recover and it may not recover as well. Essentially if you want to create and delete files quickly XFS is definitely the way to go, the easiest way you can see this is by deleting a large multi GB file and running the df command multiple times EXT4 will take a few seconds to free the space and XFS will free it immediately. XFS can also be tuned to optimize the performance in general on RAID volumes if it is told some details about the RAID when the filesystem is created. https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_6_html_performance-5Ftuning-5Fguide_s-2Dstorage-2Dxfs=DwIFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=J0XUyOuCLK8Iw3r5uS2peX0cFvg-VzKQz417xJp_NKS1rBAJsaP7fmbiyaCWwCz3=anjmxPLaNFBFOW2Ma6LUp1D5fZRlbef28388r3x2kAA= https://urldefense.proofpoint.com/v2/url?u=https-3A__xfs.org_index.php_XFS-5FFAQ-23Q-3A-5FHow-5Fto-5Fcalculate-5Fthe-5Fcorrect-5Fsunit.2Cswidth-5Fvalues-5Ffor-5Foptimal-5Fperformance=DwIFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=J0XUyOuCLK8Iw3r5uS2peX0cFvg-VzKQz417xJp_NKS1rBAJsaP7fmbiyaCWwCz3=NaUHHrZJZOKhn5EanoQKy78vFjwVakMN-Cm3kB3NdCw= Also XFS has incremental backup capabilities built in, it rudimentary but its there https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_7_html_storage-5Fadministration-5Fguide_xfsbackuprestore=DwIFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=J0XUyOuCLK8Iw3r5uS2peX0cFvg-VzKQz417xJp_NKS1rBAJsaP7fmbiyaCWwCz3=JsxStjzjv_BahpqpOc2VZyEHPZIn73ex9tpkznid0_s= . With NFS always go with EXT4 because NFS isn't compatible with 64bit inodes, so you need to disable a flag with XFS for "inode64" which means on files over 2GB XFS' will need to create multiple
