Re: [gentoo-user] file system for new machine
Am Sun, Apr 30, 2023 at 10:03:01AM +0800 schrieb William Kenworthy: > > > That > > > said btrfs has its less than stellar moments. I still have systems that > > > use > > > ext4 and they "seem" reliable for light duty but I make sure I have > > > backups > > > and do not trust them with anything important - been bitten too many > > > times! > > In what kind of situations did you encounter these problems? > > Also a fixed number of nodes. on creation (annoying and sometimes > disastrous when it runs out - think lots of small files like mail > storage), That would be my biggest concern, especially back in the day when I had rather limited hardware resources. I was “haggling” with myself as to how many inodes I would really need. These days I’m more generous, but still modify the inodes count when formatting a partition. See Dale’s recent SSD thread. > power outages cause what seems like silent corruption that builds up. I > will admit ext4 does seem better these days but I am not a fan. OK, that I’ve never had. Maybe a few forced shutdowns because the machine hung up (e.g. memory full or a botched wake from suspend). > How do you find f2fs? - I lose (wear out I guess) SD cards on raspberry pi > and Odroid systems on a regular basis with any of the mainstream filesystems > - using them as a boot drive only extends their life, but that's not always > possible. Well, no problems so far. But I’m not stress-testing it, it just runs™. The Pi is just a simple pihole/radicale/nextcloud server with not much traffic and the data card in my surface just holds my music collection. The only “issue” I currently encounter is some warning messages on Arch when I do a system update. I can’t remember the exact error, but it’s just a warning about some feature. However: The Arch wiki says: “F2FS has a weak fsck that can lead to data loss in case of a sudden power loss [3][4]. If power losses are frequent, consider an alternative file system.“ OTOH, Google is now using f2fs in Android data partitions. Before that, it was ext4. :-) -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. After humans ceased to be apes, they became Egyptians. signature.asc Description: PGP signature
Re: [gentoo-user] file system for new machine
On 29/4/23 19:45, Frank Steinmetzger wrote: Am Sat, Apr 29, 2023 at 01:20:52PM +0800 schrieb William Kenworthy: Filesystem choice is very much to do with your particular use case. I am not a fan of ext4 - lost too much data too many times. I ve found btrfs and xfs much tougher, and the online tools much more convenient. I’ve been using ext4 possibly (don’t know for sure) since it was available in standard Gentoo land. I cannot remember ever having suffered data loss. These days I like to experiment with more flash-friendly systems like f2fs, which I use on the MicroSD card of my raspberry and the 400 GB data MicroSD in my Surface Go tablet. I also test-drive it on my mini desktop PC (all Arch linux) because, like all my machines, it has an SSD. That said btrfs has its less than stellar moments. I still have systems that use ext4 and they "seem" reliable for light duty but I make sure I have backups and do not trust them with anything important - been bitten too many times! In what kind of situations did you encounter these problems? It was particularly bad when used with Dirvish for backups (lose ALL your backups at once) :( - not a problem with btrfs. Also a fixed number of nodes on creation (annoying and sometimes disastrous when it runs out - think lots of small files like mail storage), power outages cause what seems like silent corruption that builds up. I will admit ext4 does seem better these days but I am not a fan. I am using btrfs on loopback container file systems (data for mail, web, dav servers etc.) and that's not always successful with crashes - is there a better one for this use case. How do you find f2fs? - I lose (wear out I guess) SD cards on raspberry pi and Odroid systems on a regular basis with any of the mainstream filesystems - using them as a boot drive only extends their life, but that's not always possible. BillK
Re: [gentoo-user] file system for new machine
Am Sat, Apr 29, 2023 at 02:04:52PM +0100 schrieb Michael: > On Saturday, 29 April 2023 12:45:31 BST Frank Steinmetzger wrote: > > Am Sat, Apr 29, 2023 at 01:20:52PM +0800 schrieb William Kenworthy: > > > > That > > > said btrfs has its less than stellar moments. I still have systems that > > > use ext4 and they "seem" reliable for light duty but I make sure I have > > > backups and do not trust them with anything important - been bitten too > > > many times! > > In what kind of situations did you encounter these problems? > > Can't speak for William, but it was a case where using older/early versions > of > btrfs tools from some live-USB you found at the bottom of your bin of spares > could cause worse damage and data loss on btrfs. I recall the devs > recommending to always use the latest version if you were attempting a > recovery of a damaged fs and seek advice if in doubt. I was asking about his data loss with ext4. ;-) -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. ATARI! We make top-notch toasters affordable! signature.asc Description: PGP signature
Re: [gentoo-user] file system for new machine
On Saturday, 29 April 2023 12:45:31 BST Frank Steinmetzger wrote: > Am Sat, Apr 29, 2023 at 01:20:52PM +0800 schrieb William Kenworthy: > > That > > said btrfs has its less than stellar moments. I still have systems that > > use ext4 and they "seem" reliable for light duty but I make sure I have > > backups and do not trust them with anything important - been bitten too > > many times! > In what kind of situations did you encounter these problems? Can't speak for William, but it was a case where using older/early versions of btrfs tools from some live-USB you found at the bottom of your bin of spares could cause worse damage and data loss on btrfs. I recall the devs recommending to always use the latest version if you were attempting a recovery of a damaged fs and seek advice if in doubt.
Re: [gentoo-user] file system for new machine
Am Sat, Apr 29, 2023 at 01:20:52PM +0800 schrieb William Kenworthy: > Filesystem choice is very much to do with your particular use case. > > I am not a fan of ext4 - lost too much data too many times. I ve found > btrfs and xfs much tougher, and the online tools much more convenient. I’ve been using ext4 possibly (don’t know for sure) since it was available in standard Gentoo land. I cannot remember ever having suffered data loss. These days I like to experiment with more flash-friendly systems like f2fs, which I use on the MicroSD card of my raspberry and the 400 GB data MicroSD in my Surface Go tablet. I also test-drive it on my mini desktop PC (all Arch linux) because, like all my machines, it has an SSD. > That > said btrfs has its less than stellar moments. I still have systems that use > ext4 and they "seem" reliable for light duty but I make sure I have backups > and do not trust them with anything important - been bitten too many times! In what kind of situations did you encounter these problems? -- Grüße | Greetings | Qapla’ Please do not share anything from, with or about me on any social network. Eating is the most important meal of the day. signature.asc Description: PGP signature
Re: [gentoo-user] file system for new machine
Am Thu, Apr 27, 2023 at 04:58:02PM +0200 schrieb tastytea: > > Does the transparent compression incur an overhead cost in processing, > > memory use, or disk writes? I feel like it certainly has to at least > > use more memory. Sorry if that's an RTFM question. > > it'll use more cpu and memory, but disk writes and reads will be lower, > because it compresses it on the fly. The lzo algorithm which is used by default incurs a negligible performance penalty. Give it a try: take some big file, e.g. a video and then: (with $FILE being the name of the file to compress) Compression-optimised algorithms: time gzip -k $FILE # will take long with medium benefit time xz -k $FILE# will take super long time bzip2 -k $FILE # will take also long-ish Runtime-optimised algorithms: time lz -k $FILE# will go very very fast, but compression is relat. low time zstd $FILE # will go fast with better compression (comp. effort 3) time zstd -6 $FILE # will go fast-ish with more compression > it should detect early if a file is not compressible and stop. AFAIK, zfs compresses the beginning of a file and only if that yields a certain benefit, the entire file will be compressed. -- Grüße | Greetings | Qapla’ Please do not share anything from, with or about me on any social network. The realist knows what he wants; the idealist wants what he knows. signature.asc Description: PGP signature
Re: [gentoo-user] file system for new machine
On 28/4/23 21:21, Michael wrote: On Friday, 28 April 2023 13:54:37 BST Peter Humphrey wrote: On Friday, 28 April 2023 10:08:01 BST Philip Webb wrote: 230428 Peter Humphrey wrote: On Thursday, 27 April 2023 13:23:01 BST Philip Webb wrote: I've built & tested the new machine I was planning in 2022 & am at the point of designing the partitions. For many years, I've used Reiserfs, but it is now obsolescent, so I need to choose an alternative. Reiserfs seemed appropriate for a system with a large number of small files. Ext4 seems to be used by well-known binary distros. What would others recommend ? It depends: is this a UEFI machine? No, it isn't. I await your recommendation with bated breath (smile). In that case I have nothing to add to others' suggestions; sorry. :) It used to be the case btrfs would suffer corruption if you ran out of space. I don't know if this is the same today. Anecdotally, I've run out of space and the fs did not become corrupt on that partition. It corrupted another time though, but thankfully no significant data loss happened after I ran btrfs scrub, followed by btrfs check. Now I'm getting this warning on dmesg, but I have no idea what it means: BTRFS warning (device sdb3): devid 1 physical 0 len 4194304 inside the reserved space and the same on 3 other partitions on the same disk. :-/ NOTE: I don't recall ever having problems with ext4, for many years now. Filesystem choice is very much to do with your particular use case. I am not a fan of ext4 - lost too much data too many times. I ve found btrfs and xfs much tougher, and the online tools much more convenient. That said btrfs has its less than stellar moments. I still have systems that use ext4 and they "seem" reliable for light duty but I make sure I have backups and do not trust them with anything important - been bitten too many times! BillK
Re: [gentoo-user] file system for new machine
On Friday, 28 April 2023 13:54:37 BST Peter Humphrey wrote: > On Friday, 28 April 2023 10:08:01 BST Philip Webb wrote: > > 230428 Peter Humphrey wrote: > > > On Thursday, 27 April 2023 13:23:01 BST Philip Webb wrote: > > >> I've built & tested the new machine I was planning in 2022 > > >> & am at the point of designing the partitions. > > >> For many years, I've used Reiserfs, but it is now obsolescent, > > >> so I need to choose an alternative. Reiserfs seemed appropriate > > >> for a system with a large number of small files. > > >> Ext4 seems to be used by well-known binary distros. > > >> What would others recommend ? > > > > > > It depends: is this a UEFI machine? > > > > No, it isn't. I await your recommendation with bated breath (smile). > > In that case I have nothing to add to others' suggestions; sorry. :) It used to be the case btrfs would suffer corruption if you ran out of space. I don't know if this is the same today. Anecdotally, I've run out of space and the fs did not become corrupt on that partition. It corrupted another time though, but thankfully no significant data loss happened after I ran btrfs scrub, followed by btrfs check. Now I'm getting this warning on dmesg, but I have no idea what it means: BTRFS warning (device sdb3): devid 1 physical 0 len 4194304 inside the reserved space and the same on 3 other partitions on the same disk. :-/ NOTE: I don't recall ever having problems with ext4, for many years now.
Re: [gentoo-user] file system for new machine
On Friday, 28 April 2023 10:08:01 BST Philip Webb wrote: > 230428 Peter Humphrey wrote: > > On Thursday, 27 April 2023 13:23:01 BST Philip Webb wrote: > >> I've built & tested the new machine I was planning in 2022 > >> & am at the point of designing the partitions. > >> For many years, I've used Reiserfs, but it is now obsolescent, > >> so I need to choose an alternative. Reiserfs seemed appropriate > >> for a system with a large number of small files. > >> Ext4 seems to be used by well-known binary distros. > >> What would others recommend ? > > > > It depends: is this a UEFI machine? > > No, it isn't. I await your recommendation with bated breath (smile). In that case I have nothing to add to others' suggestions; sorry. :) -- Regards, Peter.
Re: [gentoo-user] file system for new machine
230428 Peter Humphrey wrote: > On Thursday, 27 April 2023 13:23:01 BST Philip Webb wrote: >> I've built & tested the new machine I was planning in 2022 >> & am at the point of designing the partitions. >> For many years, I've used Reiserfs, but it is now obsolescent, >> so I need to choose an alternative. Reiserfs seemed appropriate >> for a system with a large number of small files. >> Ext4 seems to be used by well-known binary distros. >> What would others recommend ? > It depends: is this a UEFI machine? No, it isn't. I await your recommendation with bated breath (smile). Thanks to everyone else who has replied. -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca
Re: [gentoo-user] file system for new machine
On Thursday, 27 April 2023 13:23:01 BST Philip Webb wrote: > I've built & tested the new machine I was planning in 2022 > & am at the point of designing the partitions. > > For many years, I've used Reiserfs, but it is now obsolescent, > so I need to choose an alternative. Reiserfs seemed appropriate > for a system with a large number of small files. > Ext4 seems to be used by well-known binary distros. > > What would others recommend ? It depends: is this a UEFI machine? -- Regards, Peter.
Re: [gentoo-user] file system for new machine
On 27.04.23 14:23, Philip Webb wrote: I've built & tested the new machine I was planning in 2022 & am at the point of designing the partitions. For many years, I've used Reiserfs, but it is now obsolescent, so I need to choose an alternative. Reiserfs seemed appropriate for a system with a large number of small files. Ext4 seems to be used by well-known binary distros. What would others recommend ? I usually have a boot partition with ext2, a root partition containing everything that is necessary for booting on ext4 and everything else on zfs. Andreas
Re: [gentoo-user] file system for new machine
On 04/27/2023 02:23:01 PM, Philip Webb wrote: I've built & tested the new machine I was planning in 2022 & am at the point of designing the partitions. For many years, I've used Reiserfs, but it is now obsolescent, so I need to choose an alternative. Reiserfs seemed appropriate for a system with a large number of small files. Ext4 seems to be used by well-known binary distros. What would others recommend ? I have switched to BTRFS for all file systems except root (being cautious). The main reason is BTRFS' snapshot feature which allows me to keep many backup versions without consuming much space. I have never had any failure with BTRFS so far. Helmut.
Re: [gentoo-user] file system for new machine
On 2023-04-27 16:52+0100 Neil Bothwick wrote: > On Thu, 27 Apr 2023 15:54:34 +0200, tastytea wrote: > > > btrfs and zfs have some useful features for normal use cases. the > > transparent compression can save a lot of space and even increase > > speed in some cases, the checksumming guarantees that you will > > never get a corrupt file > > That's only true if you use RAID, when there is a good copy to use. If > you have a single disk, they can only let you know a file is corrupt > but not restore it. well, you need to have a backup, of course. with ext4 it can happen that a file is corrupted in a way that is not immediately obvious but causes further damage down the line. or you get artifacts in your music and only notice it after it is too late and all your backups have the damaged version.
Re: [gentoo-user] file system for new machine
On 27/04/2023 16:52, Neil Bothwick wrote: On Thu, 27 Apr 2023 15:54:34 +0200, tastytea wrote: btrfs and zfs have some useful features for normal use cases. the transparent compression can save a lot of space and even increase speed in some cases, the checksumming guarantees that you will never get a corrupt file That's only true if you use RAID, when there is a good copy to use. If you have a single disk, they can only let you know a file is corrupt but not restore it. I run ext4, over lvm, over raid, over dm-integrity, over spinning rust. Quite a lot ... dm-integrity in particular is (of necessity) lashed up a bit. But lvm over raid is normal. https://raid.wiki.kernel.org/index.php/Linux_Raid In particular, as a matter of course, I snapshot my root partition using lvm before I "emerge --update", though I've never had any trouble that warrants trying to recover. It'll be quite a shock if I do ... Cheers, Wol
Re: [gentoo-user] file system for new machine
On Thu, 27 Apr 2023 15:54:34 +0200, tastytea wrote: > btrfs and zfs have some useful features for normal use cases. the > transparent compression can save a lot of space and even increase speed > in some cases, the checksumming guarantees that you will never get a > corrupt file That's only true if you use RAID, when there is a good copy to use. If you have a single disk, they can only let you know a file is corrupt but not restore it. -- Neil Bothwick Tact is for people who don't understand sarcasm. pgpWHrwjmNWoJ.pgp Description: OpenPGP digital signature
Re: [gentoo-user] file system for new machine
On 2023-04-27 10:14-0400 Matt Connell wrote: > On Thu, 2023-04-27 at 15:54 +0200, tastytea wrote: > > btrfs and zfs have some useful features for normal use cases. the > > transparent compression can save a lot of space and even increase > > speed in some cases, the checksumming guarantees that you will > > never get a corrupt file and snapshots make backups and rollbacks > > easier. > > Does the transparent compression incur an overhead cost in processing, > memory use, or disk writes? I feel like it certainly has to at least > use more memory. Sorry if that's an RTFM question. it'll use more cpu and memory, but disk writes and reads will be lower, because it compresses it on the fly. it should detect early if a file is not compressible and stop. it's also possible (with btrfs at least) to enable it on a directory basis.
Re: [gentoo-user] file system for new machine
On Thu, 2023-04-27 at 15:54 +0200, tastytea wrote: > btrfs and zfs have some useful features for normal use cases. the > transparent compression can save a lot of space and even increase speed > in some cases, the checksumming guarantees that you will never get a > corrupt file and snapshots make backups and rollbacks easier. Does the transparent compression incur an overhead cost in processing, memory use, or disk writes? I feel like it certainly has to at least use more memory. Sorry if that's an RTFM question. > however, they do need a bit more maintenance (described in their > respective wiki articles). This is the part that is ultimately up to OP to decide. Personally I just want to read and write data without thinking about it or maintaining anything. I maintain enough other stuff as it is :) > > [2] I'm aware that zfs and others can do snapshots for recovery and > > "roll back" but there is no replacement for versioned hard copy > > backups > > you can send snapshots to other drives or computers, either as full or > incremental backups. i'd say it's pretty much the same. 😉 Okay, that's pretty clever, I have to admit. Incremental snapshots that can be off-sited, handled by the filesystem itself? I'll keep this in mind the next time I re-do a server machine.
Re: [gentoo-user] file system for new machine
On 2023-04-27 09:34-0400 Matt Connell wrote: > On Thu, 2023-04-27 at 08:23 -0400, Philip Webb wrote: > > Ext4 seems to be used by well-known binary distros. > > There's a reason for this. It can fulfill all but the most niche or > intensive roles, is robustly supported, well-tested both in > development and through wide use in the field, and generally "just > works". > > It is a great general purpose file system, for general purpose > computing. Standard LAMP stack, desktop, laptop, HTPC, etc. are all > satisfied by ext4 > > Since it is so broadly used and supported, you are guaranteed to find > documentation for whatever feature or issue you discover. i agree with all of the above. > > What would others recommend ? > > For general purpose computing/serving, in a non-scaling, non- > performance-critical, non-experimental scenario, ext4 > > Unless[1] you are specifically: > > * learning/exploring/experimenting > * storing billions of tiny files > * storing 1TB+ individual files > * not using any kind of backups[2] btrfs and zfs have some useful features for normal use cases. the transparent compression can save a lot of space and even increase speed in some cases, the checksumming guarantees that you will never get a corrupt file and snapshots make backups and rollbacks easier. however, they do need a bit more maintenance (described in their respective wiki articles). > -- > > [1] I'm certain that there are other use cases for which ext4 is not > an optimal choice, but I don't have first-hand experience with them. > > [2] I'm aware that zfs and others can do snapshots for recovery and > "roll back" but there is no replacement for versioned hard copy > backups you can send snapshots to other drives or computers, either as full or incremental backups. i'd say it's pretty much the same. 😉
Re: [gentoo-user] file system for new machine
On Thu, 2023-04-27 at 08:23 -0400, Philip Webb wrote: > Ext4 seems to be used by well-known binary distros. There's a reason for this. It can fulfill all but the most niche or intensive roles, is robustly supported, well-tested both in development and through wide use in the field, and generally "just works". It is a great general purpose file system, for general purpose computing. Standard LAMP stack, desktop, laptop, HTPC, etc. are all satisfied by ext4 Since it is so broadly used and supported, you are guaranteed to find documentation for whatever feature or issue you discover. > What would others recommend ? For general purpose computing/serving, in a non-scaling, non- performance-critical, non-experimental scenario, ext4 Unless[1] you are specifically: * learning/exploring/experimenting * storing billions of tiny files * storing 1TB+ individual files * not using any kind of backups[2] -- [1] I'm certain that there are other use cases for which ext4 is not an optimal choice, but I don't have first-hand experience with them. [2] I'm aware that zfs and others can do snapshots for recovery and "roll back" but there is no replacement for versioned hard copy backups
Re: [gentoo-user] file system for new machine
Philip Webb wrote: > I've built & tested the new machine I was planning in 2022 > & am at the point of designing the partitions. > > For many years, I've used Reiserfs, but it is now obsolescent, > so I need to choose an alternative. Reiserfs seemed appropriate > for a system with a large number of small files. > Ext4 seems to be used by well-known binary distros. > > What would others recommend ? > When I switched away from Reiserfs, I switched to ext4. I still use ext2 for /boot but everything else is ext4. So far, I haven't had any problems at all. I have some file systems that have a lot of small files, like /usr or /var, but I also have file systems with large files. It seems to work for them all. If you know you will only ever have large files, you may want to adjust the inodes. There's a recent discussion on one of my threads about that. I have not actually done it here yet tho. Depends on just how tight your space requirements are. Hope that helps. Dale :-) :-)
[gentoo-user] file system for new machine
I've built & tested the new machine I was planning in 2022 & am at the point of designing the partitions. For many years, I've used Reiserfs, but it is now obsolescent, so I need to choose an alternative. Reiserfs seemed appropriate for a system with a large number of small files. Ext4 seems to be used by well-known binary distros. What would others recommend ? -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca