Re: XFS vs Ext4

[Konstantin Olchanski]

For your system, I would not agonize over choice of ext4 or XFS,
in practice you will see little difference.

Some practical advice for your system:

- bump the OS + home dirs from 1TB to 2TB (incremental cost of two 1TB SSD vs 
two 2TB SSD is small)
- run this system on UPS power. your 200 TB data array will be RAID-6 with 
12-16 of 20 TB HDDs, probability of one HDD failure is high, raid rebuild time 
will be about 2 days, if power goes out during rebuild, Bad Things Will Happen.
- we have been using XFS for large data arrays since late 1990-ies (on SGI 
machines), it is very reliable, it will not corrupt itself (unless you have 
defective hardware, ZFS was developed to deal with that, checksums, 
self-healing, etc).
- your server should have ECC DRAM. this is a standard feature for all 
server-class machines, use it. all our server machines have 64 GB of ECC memory.

If I were building this system, I would make both the 2TB SSD array and the 
200TB data array ZFS.

Also you do not say how you will back up your data. You must have both backups 
and archives. Backups protect you against oops, I deleted wrong file", archives 
protected you against "oops, I deleted wrong file 2 years ago".

Without backups and archives, if you have a fire, a flood, a crypto-ransomware 
attack,
if your server is stolen, you lose everything.


K.O.


On Wed, Dec 06, 2023 at 11:37:54AM -0500, Edward Zuniga wrote:
> Cc'ing supervisor to loop him in as well.
> 
> On Wed, Dec 6, 2023, 9:18 AM Edward Zuniga  wrote:
> 
> > Thanks everyone for the feedback! I've learned so much from reading the
> > discussions.
> >
> > For our application, we will have a LAN with a single server (1TB RAID1
> > array for OS, 200TB RAID5 array for data) and up to 16 workstations (1TB
> > RAID1 array for OS). Our IT department is more familiar with Rocky Linux 8,
> > which I assume will perform the same as AlmaLinux 8. Some of our MRI
> > processing can take weeks to finish, so we need a system that is very
> > reliable. We also work with individual files in the hundreds of gigabytes.
> >
> > While reading the Red Hat 8 manual
> >  >  >,
> > I found a few possible issues regarding XFS. I'm curious to see if anyone
> > has experienced these as well.
> >
> > 1. Metadata error behaviorIn ext4, you can configure the behavior when
> > the file system encounters metadata errors. The default behavior is to
> > simply continue the operation. When XFS encounters an unrecoverable
> > metadata error, it shuts down the file system and returns the EFSCORRUPTED
> >  error.*This could be problematic for processing that takes several
> > weeks.*2. Inode numbers
> >
> > The ext4 file system does not support more than 232 inodes.
> >
> > XFS dynamically allocates inodes. An XFS file system cannot run out of
> > inodes as long as there is free space on the file system.
> >
> > Certain applications cannot properly handle inode numbers larger than 232 on
> > an XFS file system. These applications might cause the failure of 32-bit
> > stat calls with the EOVERFLOW return value. Inode number exceed 232 under
> > the following conditions:
> >
> >- The file system is larger than 1 TiB with 256-byte inodes.
> >- The file system is larger than 2 TiB with 512-byte inodes.
> >
> > If your application fails with large inode numbers, mount the XFS file
> > system with the -o inode32 option to enforce inode numbers below 232.
> > Note that using inode32 does not affect inodes that are already allocated
> > with 64-bit numbers.
> > *Has anyone encountered this issue? *3. The Red Hat 8 manual also warns
> > that using xfs_repair -L might cause significant file system damage and
> > data loss and should only be used as a last resort. The manual does not
> > mention a similar warning about using e2fsck to repair an ext4 file system.
> > Has anyone experienced issues repairing a corrupt XFS file system?
> > Thanks,Eddie
> >
> > On Tue, Dec 5, 2023 at 8:46 PM Konstantin Olchanski 
> > wrote:
> >
> >> On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote:
> >> >
> >> > We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
> >> > have used ext4 for the past 10 years, however we are considering using
> >> XFS
> >> > for its better performance with larger files. Which file system do you
> >> use
> >> > for your lab?
> >> >
> >>
> >> Historical background.
> >>
> >> XFS filesystem with the companion XLV logical volume manager (aka
> >> "partitioning tool")
> >> came to Linux from SGI IRIX, where it was developed circa late-1990-ies.

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Patrick Riehecky]

On Wed, 2023-12-06 at 09:18 -0500, Edward Zuniga wrote:
Thanks everyone for the feedback! I've learned so much from reading the 
discussions.

For our application, we will have a LAN with a single server (1TB RAID1 array 
for OS, 200TB RAID5 array for data) and up to 16 workstations (1TB RAID1 array 
for OS). Our IT department is more familiar with Rocky Linux 8, which I assume 
will perform the same as AlmaLinux 8. Some of our MRI processing can take weeks 
to finish, so we need a system that is very reliable. We also work with 
individual files in the hundreds of gigabytes.

While reading the Red Hat 8 
manual,
 I found a few possible issues regarding XFS. I'm curious to see if anyone has 
experienced these as well.

1. Metadata error behavior
In ext4, you can configure the behavior when the file system encounters 
metadata errors. The default behavior is to simply continue the operation. When 
XFS encounters an unrecoverable metadata error, it shuts down the file system 
and returns the EFSCORRUPTED error.
This could be problematic for processing that takes several weeks.

In the rare issues I've hit with xfs metadata, xfs_repair has always been able 
to save me.  I've needed it maybe a dozen times in the last 20 years.  The 
repairs have been very fast in my experience - about 1 minute on an 8Tb volume. 
 Repair time seems to scale at O(ln(n)) based on my research.

2. Inode numbers

The ext4 file system does not support more than 232 inodes.

XFS dynamically allocates inodes. An XFS file system cannot run out of inodes 
as long as there is free space on the file system.

Certain applications cannot properly handle inode numbers larger than 232 on an 
XFS file system. These applications might cause the failure of 32-bit stat 
calls with the EOVERFLOW return value. Inode number exceed 232 under the 
following conditions:

  *   The file system is larger than 1 TiB with 256-byte inodes.
  *   The file system is larger than 2 TiB with 512-byte inodes.

If your application fails with large inode numbers, mount the XFS file system 
with the -o inode32 option to enforce inode numbers below 232. Note that using 
inode32 does not affect inodes that are already allocated with 64-bit numbers.

To be honest, I've only ever hit issues with 64 bit inodes on 32 bit kernels.  
I'm not sure I've truly stressed the space, but I've got some pretty decent 
volumes (30Tb) with a whole lot of files and not hit any issues.  Your mileage 
may vary.

Has anyone encountered this issue?
3. The Red Hat 8 manual also warns that using xfs_repair -L might cause 
significant file system damage and data loss and should only be used as a last 
resort. The manual does not mention a similar warning about using e2fsck to 
repair an ext4 file system. Has anyone experienced issues repairing a corrupt 
XFS file system?


xfs_repair -L is fairly scary as it zeros out the transaction log.  I'd reach 
out to the XFS folks before running it.  I've only needed a normal xfs_repair 
in the past, and that pretty infrequently.


On Tue, Dec 5, 2023 at 8:46 PM Konstantin Olchanski 
mailto:olcha...@triumf.ca>> wrote:
On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote:
>
> We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
> have used ext4 for the past 10 years, however we are considering using XFS
> for its better performance with larger files. Which file system do you use
> for your lab?
>

Historical background.

XFS filesystem with the companion XLV logical volume manager (aka "partitioning 
tool")
came to Linux from SGI IRIX, where it was developed circa late-1990-ies. XFS 
was copied
to Linux verbatim (initially with shims and kludges, later, fully integrated).
XLV was reimplemented as LVM.

The EXT series of filesystems were developed together with the linux kernel 
(first ext
filesystem may have originated with MINIX, look it up). As improvements were 
made,
journaling, no need to fsck after crash, online grow/shrink, etc, they were
renamed ext2/ext3/ext4 and they are still largely compatible between themselves.

For many purposes, both filesystems are obsoleted by ZFS, which added:

- added metadata and data checksums - to detect silent bit rot on 
current-generation HDDs and SSDs
- added online filesystem check - for broken data, gives you list of filenames 
instead of inode numbers
- added "built-in" mirroring - together with checksums, online fsck (zfs scrub) 
and monthly zfs scrub cron job, allows automatic healing of bit rot.
- added "built-in" raid-5 and raid-6 - again, together 

Re: XFS vs Ext4

[Miles ONeal]

We've never had any problems - these or any of the others mentioned. We've used 
XFS on single HDD and SSD physical workstations, but have since migrated those 
to VMs, so they're on hardware RAID now, as are most of our systems - whether 
VM or bare metal. Since we've not encountered a corrupt XFS, we haven't had to 
repair one.

The servers I run stay up 24/7 and are critical to engineering. When we moved 
the application from Solaris to Linux 8-9 years ago, I picked XFS because ZFS 
wasn't fully developed and supported on Linux yet. To date, I've seen no reason 
to switch. Performance has been good. File sizes are all over the map. We also 
have apps that run for days. I'm told by people I trust that ZFS consumes a lot 
of RAM; we already use a lot of RAM between applications and buffering.

I'm not seeing any recent benchmarks of ZFS vs XFS. I wish I had the budget and 
time to do that myself.

Had ZFS been in great shape 8-9 years ago, I'd likely have gone with it since 
we'd been using it on Solaris. At this point, we have no compelling reason to 
change. AFAICT, ZFS is not currently supported by Redhat.

From: owner-scientific-linux-us...@listserv.fnal.gov 
 on behalf of Edward Zuniga 

Sent: Wednesday, December 6, 2023 08:18
To: Edward Zuniga ; SCIENTIFIC-LINUX-USERS@fnal.gov 

Cc: Konstantin Olchanski 
Subject: Re: XFS vs Ext4

Thanks everyone for the feedback! I've learned so much from reading the 
discussions. For our application, we will have a LAN with a single server (1TB 
RAID1 array for OS, 200TB RAID5 array for data) and up to 16 workstations (1TB 
RAID1
ZjQcmQRYFpfptBannerStart
This Message is from an Untrusted Sender
You have not previously corresponded with this sender.
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/DQ3KfwI!vitvTwL4hccUFPgO7_zsG9fIHck32yvqmmKQ0z74P6iI4BcqAIzErd15oPXoC2DA1ThRRm3EZTHXaiR14ZmieRi557Cxejlgjo2MNO22AxAEeEbamD0fEdlMIwMa$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd
Thanks everyone for the feedback! I've learned so much from reading the 
discussions.

For our application, we will have a LAN with a single server (1TB RAID1 array 
for OS, 200TB RAID5 array for data) and up to 16 workstations (1TB RAID1 array 
for OS). Our IT department is more familiar with Rocky Linux 8, which I assume 
will perform the same as AlmaLinux 8. Some of our MRI processing can take weeks 
to finish, so we need a system that is very reliable. We also work with 
individual files in the hundreds of gigabytes.

While reading the Red Hat 8 
manual<https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_8_html_managing-5Ffile-5Fsystems_overview-2Dof-2Davailable-2Dfile-2Dsystems-5Fmanaging-2Dfile-2Dsystems=DwMFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=l2u4y6HnoaoxnKdKpP6TSgfeihqn5-vjvZKxF5i2ORzU4Y8_oxPPbTRAt842L9HF=x-uSSeA1IYnWWjSgPr_wxJC5HkceMyQWxtnRfc9p6uM=>,
 I found a few possible issues regarding XFS. I'm curious to see if anyone has 
experienced these as well.

1. Metadata error behavior
In ext4, you can configure the behavior when the file system encounters 
metadata errors. The default behavior is to simply continue the operation. When 
XFS encounters an unrecoverable metadata error, it shuts down the file system 
and returns the EFSCORRUPTED error.
This could be problematic for processing that takes several weeks.
2. Inode numbers

The ext4 file system does not support more than 232 inodes.

XFS dynamically allocates inodes. An XFS file system cannot run out of inodes 
as long as there is free space on the file system.

Certain applications cannot properly handle inode numbers larger than 232 on an 
XFS file system. These applications might cause the failure of 32-bit stat 
calls with the EOVERFLOW return value. Inode number exceed 232 under the 
following conditions:

  *   The file system is larger than 1 TiB with 256-byte inodes.
  *   The file system is larger than 2 TiB with 512-byte inodes.

If your application fails with large inode numbers, mount the XFS file system 
with the -o inode32 option to enforce inode numbers below 232. Note that using 
inode32 does not affect inodes that are already allocated with 64-bit numbers.

Has anyone encountered this issue?
3. The Red Hat 8 manual also warns that using xfs_repair -L might cause 
significant file system damage and data loss and should only be used as a last 
resort. The manual does not mention a similar warning about using e2fsck to 
repair an ext4 file system. Has anyone experienced issues repairing a corrupt 
XFS file system?
Thanks,
Eddie


On Tue, Dec 5, 2023 at 8:46 PM Konstantin Olchanski 
mailto:olcha...@triumf.ca>> wrote:
On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote:
>
> We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
> have used ext4 for the past 10 years, however we are considering using XFS
&

Re: XFS vs Ext4

[Edward Zuniga]

Cc'ing supervisor to loop him in as well.

On Wed, Dec 6, 2023, 9:18 AM Edward Zuniga  wrote:

> Thanks everyone for the feedback! I've learned so much from reading the
> discussions.
>
> For our application, we will have a LAN with a single server (1TB RAID1
> array for OS, 200TB RAID5 array for data) and up to 16 workstations (1TB
> RAID1 array for OS). Our IT department is more familiar with Rocky Linux 8,
> which I assume will perform the same as AlmaLinux 8. Some of our MRI
> processing can take weeks to finish, so we need a system that is very
> reliable. We also work with individual files in the hundreds of gigabytes.
>
> While reading the Red Hat 8 manual
>   >,
> I found a few possible issues regarding XFS. I'm curious to see if anyone
> has experienced these as well.
>
> 1. Metadata error behaviorIn ext4, you can configure the behavior when
> the file system encounters metadata errors. The default behavior is to
> simply continue the operation. When XFS encounters an unrecoverable
> metadata error, it shuts down the file system and returns the EFSCORRUPTED
>  error.*This could be problematic for processing that takes several
> weeks.*2. Inode numbers
>
> The ext4 file system does not support more than 232 inodes.
>
> XFS dynamically allocates inodes. An XFS file system cannot run out of
> inodes as long as there is free space on the file system.
>
> Certain applications cannot properly handle inode numbers larger than 232 on
> an XFS file system. These applications might cause the failure of 32-bit
> stat calls with the EOVERFLOW return value. Inode number exceed 232 under
> the following conditions:
>
>- The file system is larger than 1 TiB with 256-byte inodes.
>- The file system is larger than 2 TiB with 512-byte inodes.
>
> If your application fails with large inode numbers, mount the XFS file
> system with the -o inode32 option to enforce inode numbers below 232.
> Note that using inode32 does not affect inodes that are already allocated
> with 64-bit numbers.
> *Has anyone encountered this issue? *3. The Red Hat 8 manual also warns
> that using xfs_repair -L might cause significant file system damage and
> data loss and should only be used as a last resort. The manual does not
> mention a similar warning about using e2fsck to repair an ext4 file system.
> Has anyone experienced issues repairing a corrupt XFS file system?
> Thanks,Eddie
>
> On Tue, Dec 5, 2023 at 8:46 PM Konstantin Olchanski 
> wrote:
>
>> On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote:
>> >
>> > We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
>> > have used ext4 for the past 10 years, however we are considering using
>> XFS
>> > for its better performance with larger files. Which file system do you
>> use
>> > for your lab?
>> >
>>
>> Historical background.
>>
>> XFS filesystem with the companion XLV logical volume manager (aka
>> "partitioning tool")
>> came to Linux from SGI IRIX, where it was developed circa late-1990-ies.
>> XFS was copied
>> to Linux verbatim (initially with shims and kludges, later, fully
>> integrated).
>> XLV was reimplemented as LVM.
>>
>> The EXT series of filesystems were developed together with the linux
>> kernel (first ext
>> filesystem may have originated with MINIX, look it up). As improvements
>> were made,
>> journaling, no need to fsck after crash, online grow/shrink, etc, they
>> were
>> renamed ext2/ext3/ext4 and they are still largely compatible between
>> themselves.
>>
>> For many purposes, both filesystems are obsoleted by ZFS, which added:
>>
>> - added metadata and data checksums - to detect silent bit rot on
>> current-generation HDDs and SSDs
>> - added online filesystem check - for broken data, gives you list of
>> filenames instead of inode numbers
>> - added "built-in" mirroring - together with checksums, online fsck (zfs
>> scrub) and monthly zfs scrub cron job, allows automatic healing of bit rot.
>> - added "built-in" raid-5 and raid-6 - again, together with checksums and
>> online fsck, allows automatic healing and robust operation in presence of
>> disk bad sectors, I/O errors, corruption and single-disk failure.
>> - other goodies like snapshots, large ram cache, dedup, online
>> compression, etc are taken for granted for current generation filesystems.
>>
>> On current generation HDDs and SSds, use of bare XFS and ext4 is
>> dangerous, SSD failure or "HDD grows bad sectors" will destroy your data
>> completely.
>>
>> On current generation HDDs, use of mirrored XFS and ext4 is dangerous
>> 

Re: XFS vs Ext4

[Edward Zuniga]

Thanks everyone for the feedback! I've learned so much from reading the
discussions.

For our application, we will have a LAN with a single server (1TB RAID1
array for OS, 200TB RAID5 array for data) and up to 16 workstations (1TB
RAID1 array for OS). Our IT department is more familiar with Rocky Linux 8,
which I assume will perform the same as AlmaLinux 8. Some of our MRI
processing can take weeks to finish, so we need a system that is very
reliable. We also work with individual files in the hundreds of gigabytes.

While reading the Red Hat 8 manual
,
I found a few possible issues regarding XFS. I'm curious to see if anyone
has experienced these as well.

1. Metadata error behaviorIn ext4, you can configure the behavior when the
file system encounters metadata errors. The default behavior is to simply
continue the operation. When XFS encounters an unrecoverable metadata
error, it shuts down the file system and returns the EFSCORRUPTED error.*This
could be problematic for processing that takes several weeks.*2. Inode
numbers

The ext4 file system does not support more than 232 inodes.

XFS dynamically allocates inodes. An XFS file system cannot run out of
inodes as long as there is free space on the file system.

Certain applications cannot properly handle inode numbers larger than 232 on
an XFS file system. These applications might cause the failure of 32-bit
stat calls with the EOVERFLOW return value. Inode number exceed 232 under
the following conditions:

   - The file system is larger than 1 TiB with 256-byte inodes.
   - The file system is larger than 2 TiB with 512-byte inodes.

If your application fails with large inode numbers, mount the XFS file
system with the -o inode32 option to enforce inode numbers below 232. Note
that using inode32 does not affect inodes that are already allocated with
64-bit numbers.
*Has anyone encountered this issue? *3. The Red Hat 8 manual also warns
that using xfs_repair -L might cause significant file system damage and
data loss and should only be used as a last resort. The manual does not
mention a similar warning about using e2fsck to repair an ext4 file system.
Has anyone experienced issues repairing a corrupt XFS file system?
Thanks,Eddie

On Tue, Dec 5, 2023 at 8:46 PM Konstantin Olchanski 
wrote:

> On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote:
> >
> > We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
> > have used ext4 for the past 10 years, however we are considering using
> XFS
> > for its better performance with larger files. Which file system do you
> use
> > for your lab?
> >
>
> Historical background.
>
> XFS filesystem with the companion XLV logical volume manager (aka
> "partitioning tool")
> came to Linux from SGI IRIX, where it was developed circa late-1990-ies.
> XFS was copied
> to Linux verbatim (initially with shims and kludges, later, fully
> integrated).
> XLV was reimplemented as LVM.
>
> The EXT series of filesystems were developed together with the linux
> kernel (first ext
> filesystem may have originated with MINIX, look it up). As improvements
> were made,
> journaling, no need to fsck after crash, online grow/shrink, etc, they were
> renamed ext2/ext3/ext4 and they are still largely compatible between
> themselves.
>
> For many purposes, both filesystems are obsoleted by ZFS, which added:
>
> - added metadata and data checksums - to detect silent bit rot on
> current-generation HDDs and SSDs
> - added online filesystem check - for broken data, gives you list of
> filenames instead of inode numbers
> - added "built-in" mirroring - together with checksums, online fsck (zfs
> scrub) and monthly zfs scrub cron job, allows automatic healing of bit rot.
> - added "built-in" raid-5 and raid-6 - again, together with checksums and
> online fsck, allows automatic healing and robust operation in presence of
> disk bad sectors, I/O errors, corruption and single-disk failure.
> - other goodies like snapshots, large ram cache, dedup, online
> compression, etc are taken for granted for current generation filesystems.
>
> On current generation HDDs and SSds, use of bare XFS and ext4 is
> dangerous, SSD failure or "HDD grows bad sectors" will destroy your data
> completely.
>
> On current generation HDDs, use of mirrored XFS and ext4 is dangerous
> (using mdadm or LVM mirroring), (a) bit rot inevitably causes differences
> between data between the two disks. Lacking checksums, mdadm and LVM
> mirroring cannot decide which of the two copies is the correct one. (b)
> after a crash, mirror 

Re: XFS vs Ext4

[Konstantin Olchanski]

On Mon, Dec 04, 2023 at 03:03:46PM -0500, Edward Zuniga wrote:
> 
> We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
> have used ext4 for the past 10 years, however we are considering using XFS
> for its better performance with larger files. Which file system do you use
> for your lab?
>

Historical background.

XFS filesystem with the companion XLV logical volume manager (aka "partitioning 
tool")
came to Linux from SGI IRIX, where it was developed circa late-1990-ies. XFS 
was copied
to Linux verbatim (initially with shims and kludges, later, fully integrated).
XLV was reimplemented as LVM.

The EXT series of filesystems were developed together with the linux kernel 
(first ext
filesystem may have originated with MINIX, look it up). As improvements were 
made,
journaling, no need to fsck after crash, online grow/shrink, etc, they were
renamed ext2/ext3/ext4 and they are still largely compatible between themselves.

For many purposes, both filesystems are obsoleted by ZFS, which added:

- added metadata and data checksums - to detect silent bit rot on 
current-generation HDDs and SSDs
- added online filesystem check - for broken data, gives you list of filenames 
instead of inode numbers
- added "built-in" mirroring - together with checksums, online fsck (zfs scrub) 
and monthly zfs scrub cron job, allows automatic healing of bit rot.
- added "built-in" raid-5 and raid-6 - again, together with checksums and 
online fsck, allows automatic healing and robust operation in presence of disk 
bad sectors, I/O errors, corruption and single-disk failure.
- other goodies like snapshots, large ram cache, dedup, online compression, etc 
are taken for granted for current generation filesystems.

On current generation HDDs and SSds, use of bare XFS and ext4 is dangerous, SSD 
failure or "HDD grows bad sectors" will destroy your data completely.

On current generation HDDs, use of mirrored XFS and ext4 is dangerous (using 
mdadm or LVM mirroring), (a) bit rot inevitably causes differences between data 
between the two disks. Lacking checksums, mdadm and LVM mirroring cannot decide 
which of the two copies is the correct one. (b) after a crash, mirror rebuild 
fill fail if both disks happen to have bad sectors (or throw random I/O errors).

Ditto for RAID5 and RAID6, probability of RAID rebuild failing because multiple 
disks have have sectors and I/O errors goes up with the number of disks.

ZFS was invented to resolve all these problems. (BTRFS was invented as a NIH 
erzatz ZFS, is still incomplete wrt RAID5/RAID6).

Bottom line, if you can, use ZFS. Current Ubuntu installer has a button 
"install on ZFS", use it!

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: XFS vs Ext4

[Patrick J. LoPresti]

On Tue, Dec 5, 2023 at 1:51 AM Paul Robert Marino 
wrote:
> With NFS always go with EXT4 because NFS isn't compatible with 64bit
inodes, so you need to disable a flag with XFS for "inode64" which means on
files over 2GB XFS' will need to create multiple inodes instead of 1 at the
beginning of the file which hurts it's performance.

Can you provide details on this, or a reference?

Even NFSv2 had fixed-size 32-byte/256-bit file handles. NFSv3 (1995) made
them variable size and up to 64 bytes / 512 bits. Linux NFS may have taken
a while to catch up, but I am pretty sure anything from the past decade
will be fine with 64 bit inodes.

We have been using XFS over NFS for at least 10 years now, with partitions
in the hundreds of terabytes and individual files in the hundreds of
gigabytes. No problem. I have no idea why anyone would consider ext4 for
any serious application.

 - Pat

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Yasha Karant]

Although CIS compliance (for "security" of assets) may be a requirement, 
it is not infallible and often gives a false sense of being secure from 
compromises.  Example: 
https://urldefense.proofpoint.com/v2/url?u=https-3A__techcrunch.com_2023_03_16_cisa-2Dnation-2Dstate-2Dhackers-2Dbreach-2Dfederal-2Dagency=DwIDaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=6PchYtJmUCDWyWETQdqlMWUVc480pT-DgabXWg_139Oa_NTleIYH_GXzCxcXwLCt=T_qwom_-eNAdMLSWz93PZyzR-ZfY2DhKqc79YZ4mCqE= 

and there are some discussions at Black Hat.  Whenever a compromise on a 
CIS compliant system is accomplished (sometimes by a nation-state 
clandestine service operation/agent), vulnerabilities are discovered 
that have to be corrected in the compliance testing applications.


Given how ancient SL is at this time, I am curious under which 
situations it is still used -- dedicated systems that cannot be changed 
to a more "current" "enterprise" Linux (such as Ubuntu LTS)?


-Original Message-
From: Nico Kadel-Garcia [mailto:nka...@gmail.com]
Sent: Tuesday, December 05, 2023 01:19 PM -08
To: Laura Hild
Cc: scientific-linux-users
Subject: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

On Tue, Dec 5, 2023 at 9:06 AM Laura Hild  wrote:



No! No. No LVM! Bad admin, no biscuit!
[...]
*Bad* admin. Where's my squirt bottle?


Yeah, I wrote "if you're a slice-and-dicer" for a reason.  One big root is a 
fine option, but it's not the situation I was imagining where one is concerned with 
shrinkability.  I think having hard limits on the growth of certain subtrees can be 
helpful, and sometimes different mount options.  I'm Kickstarting rather than imaging, so 
I don't have a problem including the hostname in the name of the Volume Group.  Everyone 
has different objectives (and I'm skeptical lack of LVM would have adequately protected 
you from your predecessors' and colleagues' :)).


Various options for various filesystems is considered a big deal CIS
compliance. I consider it a completely destabilizing waste of time
better spent elsewhere.

Avoiding LVM and its tendency to  use identical volume names in VM
images is a problem in cloud or VM environments where you may wish
to mount and access a previous snapshot of the same VM. Itt's much
easier to cut sections off a long rope than try to tie a bunch of
short pieces to make the rope you need, when trying to tie up loose
ends.

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Nico Kadel-Garcia]

On Tue, Dec 5, 2023 at 9:06 AM Laura Hild  wrote:
>
> > No! No. No LVM! Bad admin, no biscuit!
> > [...]
> > *Bad* admin. Where's my squirt bottle?
>
> Yeah, I wrote "if you're a slice-and-dicer" for a reason.  One big root is a 
> fine option, but it's not the situation I was imagining where one is 
> concerned with shrinkability.  I think having hard limits on the growth of 
> certain subtrees can be helpful, and sometimes different mount options.  I'm 
> Kickstarting rather than imaging, so I don't have a problem including the 
> hostname in the name of the Volume Group.  Everyone has different objectives 
> (and I'm skeptical lack of LVM would have adequately protected you from your 
> predecessors' and colleagues' :)).

Various options for various filesystems is considered a big deal CIS
compliance. I consider it a completely destabilizing waste of time
better spent elsewhere.

Avoiding LVM and its tendency to  use identical volume names in VM
images is a problem in cloud or VM environments where you may wish
to mount and access a previous snapshot of the same VM. Itt's much
easier to cut sections off a long rope than try to tie a bunch of
short pieces to make the rope you need, when trying to tie up loose
ends.

>

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Laura Hild]

> No! No. No LVM! Bad admin, no biscuit!
> [...]
> *Bad* admin. Where's my squirt bottle?

Yeah, I wrote "if you're a slice-and-dicer" for a reason.  One big root is a 
fine option, but it's not the situation I was imagining where one is concerned 
with shrinkability.  I think having hard limits on the growth of certain 
subtrees can be helpful, and sometimes different mount options.  I'm 
Kickstarting rather than imaging, so I don't have a problem including the 
hostname in the name of the Volume Group.  Everyone has different objectives 
(and I'm skeptical lack of LVM would have adequately protected you from your 
predecessors' and colleagues' :)).

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Jose Marques]

We use EXT4 mostly. We find it to be more robust for lab machines, and more 
efficient for VMs. XFS is limited to cases where its more advanced features are 
useful. We treat our user facing systems as being "disposable", if they develop 
a software/filesystem issue we usually wipe and reinstall rather than try to 
fix same. Important storage is mounted from separate storage servers.

Re: XFS vs Ext4

[Paul Robert Marino]

XFS is a from the ground up journaling filesystem, whereas EXT4 is still a
journal tacked on filesystem.
That's said EXT4 has caught up to XFS and in some specific
cases exceeded its performance but not all.

The short version is it depends on what you are doing both have pros and
cons,

Here are some short examples
If I'm doing a Gluster object store I will always use XFS because Gluster
is optimized for XFS,
If it's an NFS volume then EXT4 is usually better because of inode
compatibility issues which can be worked around in XFS but tank the
performance.
If I'm using it for temp space for compiles or an ETL, XFS will probably do
better because it will reduce the IOPS around the handling of inodes.
Extremely large volumes (100TB+ on a SAN or in a RAID) I always will use
XFS.
Desktop pick the one you know the best, but if undelete is important to you
use EXT4 (XFS has no mechanism for undeleting files).
if you are dealing with an embedded device that boots off of an SD card or
EMMC, XFS if possible because how it handles inodes puts less wear on it
over time.
High risk of file corruption due to unreliable power or hardware always XFS

The long version
There is a huge efficiency gain when deleting files in XFS. It's noticeably
faster when freeing up space from deleting files; this is because instead
of having an inode per block it only creates 1 inode at the beginning of
every file as needed for legacy application backward support.
Another positive side effect of his formatting an XFS filesystem is faster
because it never pre-creates inodes during the formatting process, infact
the only thing mkfs.xfs does is create a couple of redundant journals.
The 1 inode per file is also the reason XFS puts less wear on SSD's, This
difference wont make a noticeable impact on the life of a business class
SSD, and probably not in a decent consumed grade SSD, but for SD cards and
EMMC being used as a boot device this impact can be huge especially if that
device isn't using tmpfs for /tmp or is storing /var on the SSD and is
writing a lot of logs and temp files.
Risk of file system corruption due to unplanned power outages XFS,
because it keeps multiple copies of the journal, ignores the inodes for
recovery and generally is self healing. While they both do an amazing job
at recovering from this kind of situation, EXT4 still trusts the inodes
over its single copy of the journal and as a result is more likely to have
significant file corruption issues sooner. fsck.xfs normally only does a
comparison of the copies of the journal to fix journal corruption issues
and will usually roll back file changes that were incomplete at the time of
the power loss. By the way XFS does this automatically when the volumes are
mounted if the checksums of the copies of the journal don't match so
essentially it's normally a placebo command which you should never need to
run unless you are dealing with bad sectors in the physical media. EXT4
handles things way differently on mount and as a result you may need to
manually run fsck.ext4 after a power loss to recover and it may not
recover as well.
Essentially if you want to create and delete files quickly XFS is
definitely the way to go, the easiest way you can see this is by deleting a
large multi GB file and running the df command multiple times EXT4 will
take a few seconds to free the space and XFS will free it immediately.
XFS can also be tuned to optimize the performance in general on RAID
volumes if it is told some details about the RAID when the filesystem is
created.
https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_6_html_performance-5Ftuning-5Fguide_s-2Dstorage-2Dxfs=DwIFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=J0XUyOuCLK8Iw3r5uS2peX0cFvg-VzKQz417xJp_NKS1rBAJsaP7fmbiyaCWwCz3=anjmxPLaNFBFOW2Ma6LUp1D5fZRlbef28388r3x2kAA=
 

https://urldefense.proofpoint.com/v2/url?u=https-3A__xfs.org_index.php_XFS-5FFAQ-23Q-3A-5FHow-5Fto-5Fcalculate-5Fthe-5Fcorrect-5Fsunit.2Cswidth-5Fvalues-5Ffor-5Foptimal-5Fperformance=DwIFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=J0XUyOuCLK8Iw3r5uS2peX0cFvg-VzKQz417xJp_NKS1rBAJsaP7fmbiyaCWwCz3=NaUHHrZJZOKhn5EanoQKy78vFjwVakMN-Cm3kB3NdCw=
 
Also XFS has incremental backup  capabilities built in, it rudimentary but
its there
https://urldefense.proofpoint.com/v2/url?u=https-3A__access.redhat.com_documentation_en-2Dus_red-5Fhat-5Fenterprise-5Flinux_7_html_storage-5Fadministration-5Fguide_xfsbackuprestore=DwIFaQ=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=J0XUyOuCLK8Iw3r5uS2peX0cFvg-VzKQz417xJp_NKS1rBAJsaP7fmbiyaCWwCz3=JsxStjzjv_BahpqpOc2VZyEHPZIn73ex9tpkznid0_s=
 
.
With NFS always go with EXT4 because NFS isn't compatible with 64bit
inodes, so you need to disable a flag with XFS for "inode64" which means on
files over 2GB XFS' will need to create multiple 

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Nico Kadel-Garcia]

On Mon, Dec 4, 2023 at 5:07 PM Laura Hild  wrote:
>
> > But shrinking a partition is something that is far less likely to need
> > to be done in an enterprise environment (compared to me fighting with
> > too old hardware at home), so maybe that doesn't really matter.
>
> Especially if you use LVM and leave free space in the VG.  If you're a
> slice-and-dicer and you fully allocate and/or use fixed partitions, then
> using XFS you may come to regret, say, a 10 gig /var and a multi-hundred
> gig root, but if you left root closer to the size it needed to be, you
> just lvextend and xfs_growfs as needed.

No! No. No LVM! Bad admin, no biscuit!

LVM is not a file system, it's partition management scheme, and
encourages *extremely* bad disk management of people attempting to
pre-plot exactly how space every chunk of their OS will take. It
encourages micro-management that can be grotesquely painful to clean
up, especially when you decide to us docker, that run in
/var/lib/docker/, and mock, that lives both in /var/cache/mock/,  and
/var/lib/mock/, and some vendor shoves their database in "/opt/", and
some httpd "build-it-himself" advocate reads the Ubuntu directions and
puts it in "/home/httpd/", and now you have to play whack-a-mole
because some clever person you could make your disk space more
efficient by utting it into *perfectly* sized little bits with LVM.

*Bad* admin. Where's my squirt bottle?

LVM also suffers from the issue when you work from a base OS image and
you try to mount another image of the same reference OS for recovery
purposes, like for example an RHEL or Linux in VMware, the identical
names of the LVM partitions are a nightmare to resolve.

The "I  over-allocated disk" problem has almost nothing to do with
XFS, shrinking ext4 filesystems with resize2fs is usually not worth
the work. It's usually far simpler ot attach another disk and stash
content there for duplication and resizing. I've done this on a grand
scale, back when disks were *much* smaller with dual-disk drive
systems and I could copy to the second disk, about 20,000 systems in
one month.

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Laura Hild]

> But shrinking a partition is something that is far less likely to need
> to be done in an enterprise environment (compared to me fighting with
> too old hardware at home), so maybe that doesn't really matter.

Especially if you use LVM and leave free space in the VG.  If you're a
slice-and-dicer and you fully allocate and/or use fixed partitions, then
using XFS you may come to regret, say, a 10 gig /var and a multi-hundred
gig root, but if you left root closer to the size it needed to be, you
just lvextend and xfs_growfs as needed.

Re: XFS vs Ext4

[Laura Hild]

I've been letting it default as long as it's been EL's default, and the last 
time I had trouble with XFS was EL6.2.

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Alec Habig]

The one minor gotcha Pat pointed out to me recently: if you want to
resize a partition in xfs, you can grow it but not shrink it.

But shrinking a partition is something that is far less likely to need
to be done in an enterprise environment (compared to me fighting with
too old hardware at home), so maybe that doesn't really matter.

Current versions of most distros also default to xfs, so it's also
keeping up with the Joneses from a compatability standpoint.

-- 
   Alec Habig
 University of Minnesota Duluth
 Dept. of Physics and Astronomy
 aha...@umn.edu
   
https://urldefense.proofpoint.com/v2/url?u=http-3A__neutrino.d.umn.edu_-7Ehabig_=DwIBAw=gRgGjJ3BkIsb5y6s49QqsA=gd8BzeSQcySVxr0gDWSEbN-P-pgDXkdyCtaMqdCgPPdW1cyL5RIpaIYrCn8C5x2A=5XgYzVvh0gkxLnbJ-WwoovR9bTZ-LSsLduRvEyDqf-hGtY9gWCDXUk5ENzQcAE1e=3Nv-Bi2gDPdJFHbRqW13Sn1Qb3qopcRpuLSQtdOqs_U=
 

Re: [SCIENTIFIC-LINUX-USERS] XFS vs Ext4

[Patrick Riehecky]

Personally, I'm very fond of XFS.

Its been a while, but the last time I really looked into it, XFS had a
static memory allotment for each mount whereas EXT4 was able to shrink
the memory requirements based on the size of the mountpoints.  In
theory you may not feel the difference, but if you've got a VM host
running a few hundred VMs on top - each one using XFS that will
probably add up.

Pat

On Mon, 2023-12-04 at 15:03 -0500, Edward Zuniga wrote:
> Hello,
> 
> We are upgrading our MRI Lab servers and workstations to AlmaLinux 8.
> We have used ext4 for the past 10 years, however we are considering
> using XFS for its better performance with larger files. Which file
> system do you use for your lab?
> 
> Thanks,
> Eddie 
> 
> -- 
> Edward A. Zuniga
> Senior Research Analyst/CARI MRI Lab Manager
> Virginia Commonwealth University
> C. Kenneth and Dianne Wright Center for Clinical and Translational
> Research
> Institute for Drug and Alcohol Studies
> Collaborative Advanced Research Imaging (CARI) Facility
> 203 East Cary Street, Suite 202
> Richmond, Virginia 23219
> Phone: (804) 828-4184
> Fax: (804) 827-2565
> eazun...@vcu.edu 

Re: XFS vs Ext4

[Miles ONeal]

Sorry. We don't actually run clusters. They get called that, but they're really 
farms managed by LSF.

From: owner-scientific-linux-us...@listserv.fnal.gov 
 on behalf of Miles ONeal 
<0be99a30c213-dmarc-requ...@listserv.fnal.gov>
Sent: Monday, December 4, 2023 14:17
To: Edward Zuniga ; SCIENTIFIC-LINUX-USERS@fnal.gov 

Subject: Re: XFS vs Ext4

Cirrus isn't a lab, but we run clusters, have over a thousand workstations, and 
have many more infrastructure systems. We haven't used ext4 in a good while 
except on a few legacy hosts. xfs has been fast and stable. I don't recall the 
last
ZjQcmQRYFpfptBannerStart
This Message is from an Untrusted Sender
You have not previously corresponded with this sender.
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/DQ3KfwI!vKtvbqx0QU0aUhsCz18M0Qe4vOEoxPixc5q9xqyUa4NCgmtJfguXW1R1RwaKeFRSO9vnEoUNbHB21DtwkEANfYN1qkomE-at45PaBn58ZnRWwYZKT4qysprox4gRYZBL9eUISCZaiIjA$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd
Cirrus isn't a lab, but we run clusters, have over a thousand workstations, and 
have many more infrastructure systems. We haven't used ext4 in a good while 
except on a few legacy hosts. xfs has been fast and stable. I don't recall the 
last time I saw fsck run, but it was before moving to xfs. Also, xfs can create 
large (50TB+) filesystems very quickly, which makes configuring systems a lot 
quicker and easier - especially if you're testing and having to blow a 
filesystem away and recreate it multiple times.

From: owner-scientific-linux-us...@listserv.fnal.gov 
 on behalf of Edward Zuniga 

Sent: Monday, December 4, 2023 14:03
To: SCIENTIFIC-LINUX-USERS@fnal.gov 
Subject: XFS vs Ext4

Hello, We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We 
have used ext4 for the past 10 years, however we are considering using XFS for 
its better performance with larger files. Which file system do you use for your 
lab?Thanks,Eddie --
ZjQcmQRYFpfptBannerStart
This Message is from an Untrusted Sender
You have not previously corresponded with this sender.
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/DQ3KfwI!vitvTwL4hccUFPujRZGM3aCCuPzwZDRbZ85nXZ42gf_zAG7QlrXrJh_mum_GvdkHqAyCkRalHjNCJ0JiVpEBtaM5nKghrrxvSivoOeODZZMgfuTBWTQ3rx3tGR7L$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd
Hello,

We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We have 
used ext4 for the past 10 years, however we are considering using XFS for its 
better performance with larger files. Which file system do you use for your lab?

Thanks,
Eddie

--
Edward A. Zuniga
Senior Research Analyst/CARI MRI Lab Manager
Virginia Commonwealth University
C. Kenneth and Dianne Wright Center for Clinical and Translational Research
Institute for Drug and Alcohol Studies
Collaborative Advanced Research Imaging (CARI) Facility
203 East Cary Street, Suite 202
Richmond, Virginia 23219
Phone: (804) 828-4184
Fax: (804) 827-2565
eazun...@vcu.edu<mailto:eazun...@vcu.edu>

Re: XFS vs Ext4

[Miles ONeal]

Cirrus isn't a lab, but we run clusters, have over a thousand workstations, and 
have many more infrastructure systems. We haven't used ext4 in a good while 
except on a few legacy hosts. xfs has been fast and stable. I don't recall the 
last time I saw fsck run, but it was before moving to xfs. Also, xfs can create 
large (50TB+) filesystems very quickly, which makes configuring systems a lot 
quicker and easier - especially if you're testing and having to blow a 
filesystem away and recreate it multiple times.

From: owner-scientific-linux-us...@listserv.fnal.gov 
 on behalf of Edward Zuniga 

Sent: Monday, December 4, 2023 14:03
To: SCIENTIFIC-LINUX-USERS@fnal.gov 
Subject: XFS vs Ext4

Hello, We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We 
have used ext4 for the past 10 years, however we are considering using XFS for 
its better performance with larger files. Which file system do you use for your 
lab?Thanks,Eddie --
ZjQcmQRYFpfptBannerStart
This Message is from an Untrusted Sender
You have not previously corresponded with this sender.
<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/DQ3KfwI!vitvTwL4hccUFPujRZGM3aCCuPzwZDRbZ85nXZ42gf_zAG7QlrXrJh_mum_GvdkHqAyCkRalHjNCJ0JiVpEBtaM5nKghrrxvSivoOeODZZMgfuTBWTQ3rx3tGR7L$>
Report Suspicious

ZjQcmQRYFpfptBannerEnd
Hello,

We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We have 
used ext4 for the past 10 years, however we are considering using XFS for its 
better performance with larger files. Which file system do you use for your lab?

Thanks,
Eddie

--
Edward A. Zuniga
Senior Research Analyst/CARI MRI Lab Manager
Virginia Commonwealth University
C. Kenneth and Dianne Wright Center for Clinical and Translational Research
Institute for Drug and Alcohol Studies
Collaborative Advanced Research Imaging (CARI) Facility
203 East Cary Street, Suite 202
Richmond, Virginia 23219
Phone: (804) 828-4184
Fax: (804) 827-2565
eazun...@vcu.edu<mailto:eazun...@vcu.edu>

XFS vs Ext4

[Edward Zuniga]

Hello,

We are upgrading our MRI Lab servers and workstations to AlmaLinux 8. We
have used ext4 for the past 10 years, however we are considering using XFS
for its better performance with larger files. Which file system do you use
for your lab?

Thanks,
Eddie

-- 
Edward A. Zuniga
Senior Research Analyst/CARI MRI Lab Manager
Virginia Commonwealth University
C. Kenneth and Dianne Wright Center for Clinical and Translational Research
Institute for Drug and Alcohol Studies
Collaborative Advanced Research Imaging (CARI) Facility
203 East Cary Street, Suite 202
Richmond, Virginia 23219
Phone: (804) 828-4184
Fax: (804) 827-2565
eazun...@vcu.edu