Hi Chao, On Tue, Apr 11, 2023 at 12:44 AM Chao Yu <c...@kernel.org> wrote: > > Hi Juhyung, > > On 2023/4/4 15:36, Juhyung Park wrote: > > Hi everyone, > > > > I want to start a discussion on using f2fs for regular > > desktops/workstations. > > > > There are growing number of interests in using f2fs as the general > > root file-system: > > 2018: https://www.phoronix.com/news/GRUB-Now-Supports-F2FS > > 2020: https://www.phoronix.com/news/Clear-Linux-F2FS-Root-Option > > 2023: > > https://code.launchpad.net/~nexusprism/curtin/+git/curtin/+merge/439880 > > 2023: https://code.launchpad.net/~nexusprism/grub/+git/ubuntu/+merge/440193 > > > > I've been personally running f2fs on all of my x86 Linux boxes since > > 2015, and I have several concerns that I think we need to collectively > > address for regular non-Android normies to use f2fs: > > > > A. Bootloader and installer support > > B. Host-side GC > > C. Extended node bitmap > > > > I'll go through each one. > > > > === A. Bootloader and installer support === > > > > It seems that both GRUB and systemd-boot supports f2fs without the > > need for a separate ext4-formatted /boot partition. > > Some distros are seemingly disabling f2fs module for GRUB though for > > security reasons: > > https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1868664 > > > > It's ultimately up to the distro folks to enable this, and still in > > the worst-case scenario, they can specify a separate /boot partition > > and format it to ext4 upon installation. > > > > The installer itself to show f2fs and call mkfs.f2fs is being worked > > on currently on Ubuntu. See the 2023 links above. > > > > Nothing f2fs mainline developers should do here, imo. > > > > === B. Host-side GC === > > > > f2fs relieves most of the device-side GC but introduces a new > > host-side GC. This is extremely confusing for people who have no > > background in SSDs and flash storage to understand, let alone > > discard/trim/erase complications. > > > > In most consumer-grade blackbox SSDs, device-side GCs are handled > > automatically for various workloads. f2fs, however, leaves that > > responsibility to the userspace with conservative tuning on the > > We've proposed a f2fs feature named "space awared garbage collection" > and shipped it in huawei/honor's devices, but forgot to try upstreaming > it. :-P > > In this feature, we introduced three mode: > - performance mode: something like write-gc in ftl, it can trigger > background gc more frequently and tune its speed according to free > segs and reclaimable blks ratio. > - lifetime mode: slow down background gc to avoid high waf if there > is less free space. > - balance mode: behave as usual. > > I guess this may be helpful for Linux desktop distros since there is > no such storage service trigger gc_urgent. >
That indeed sounds interesting. If you need me to test something out, feel free to ask. I manually trigger gc_urgent from time to time on my 2TB SSD laptop (which, as a laptop, isn't left on 24/7 so f2fs have a bit of trouble finding enough idle time to trigger GC sufficiently). If I don't, I run out of free segments within a few weeks. > > kernel-side by default. Android handles this by init.rc tunings and a > > separate code running in vold to trigger gc_urgent. > > > > For regular Linux desktop distros, f2fs just runs on the default > > configuration set on the kernel and unless it’s running 24/7 with > > plentiful idle time, it quickly runs out of free segments and starts > > triggering foreground GC. This is giving people the wrong impression > > that f2fs slows down far drastically than other file-systems when > > that’s quite the contrary (i.e., less fragmentation overtime). > > > > This is almost the equivalent of re-living the nightmare of trim. On > > SSDs with very small to no over-provisioned space, running a > > file-system with no discard what-so-ever (sadly still a common case > > when an external SSD is used with no UAS) will also drastically slow > > What does UAS mean? > USB Attached SCSI. It's a protocol that sends SCSI commands over USB. Most SATA-to-USB and NVMe-to-USB chips support it. AFAIK, it's the only way of sending trim commands and query SMART data over USB. (Plus, it's faster.) If either the host or the chip doesn't support it, it's negotiated through "usb-storage" (aka mass-storage), which then prevents anyone from sending trim commands. The external SSD shenanigan is a whole another rant for another day.. > > the performance down. On file-systems with no asynchronous discard, > > There is no such performance issue in f2fs, right? as f2fs enables > discard mount option by default, and supports async discard feature. > Yup. It's one of my favorite f2fs feature :)) Though imo it might be a good idea to explicitly recommend people to NOT disable it as a lot of "how to improve SSD performance on Linux" guides online tell you to outright disable the "discard" mount option. Like you said, those concerns are invalid on f2fs. btrfs recently added discard=async and enabled it by default too, but I'm not sure if their implementation aims to do the same with what f2fs does. > > mounting a file-system with the discard option adds a non-negligible > > overhead on every remove/delete operations, so most distros now > > (thankfully) use a timer job registered to systemd to trigger fstrim: > > https://github.com/util-linux/util-linux/commits/master/sys-utils/fstrim.timer > > > > This is still far from ideal. The default file-system, ext4, slows > > down drastically almost to a halt when fstrim -a is called, especially > > on SATA. For some reason that is still a mystery for me, people seem > > to be happy with it. No one bothered to improve it for years > > ¯\_(ツ)_/¯. > > > > So here’s my proposal: > > As Linux distros don’t have a good mechanism for hinting when to > > trigger GC, introduce a new Kconfig, CONFIG_F2FS_GC_UPON_FSTRIM and > > enable it by default. > > This config will hook up ioctl(FITRIM), which is currently ignored on > > f2fs - > > https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs.git/commit/?h=master&id=e555da9f31210d2b62805cd7faf29228af7c3cfb > > , to perform discard and GC on all invalid segments. > > Userspace configuration with enough f2fs/GC knowledge such as Android > > should disable it. > > > > This will ensure that Linux distros that blindly call fstrim will at > > least avoid constant slowdowns when free segments are depleted with > > the occasional (once a week) slowdown, which *people are already > > living with on ext4*. I'll even go further and mention that since f2fs > > GC is a regular R/W workload, it doesn't cause an extreme slowdown > > comparable to a level of a full file-system trim operation. > > > > If this is acceptable, I’ll cook up a patch. > > > > In an ideal world, all Linux distros should have an explicit f2fs GC > > trigger mechanism (akin to > > https://github.com/kdave/btrfsmaintenance#distro-integration ), but > > it’s practically unrealistic to expect that, given the installer > > doesn’t even support f2fs for now. > > > > === C. Extended node bitmap === > > > > f2fs by default have a very limited number of allowed inodes compared > > to other file-systems. Just 2 AOSP syncs are enough to exhaust f2fs > > and result in -ENOSPC. > > > > Here are some of the stats collected from me and my colleague that we > > use daily as a regular desktop with GUI, web-browsing and everything: > > 1. Laptop > > Utilization: 68% (182914850 valid blocks, 462 discard blocks) > > - Node: 10234905 (Inode: 10106526, Other: 128379) > > - Data: 172679945 > > - Inline_xattr Inode: 2004827 > > - Inline_data Inode: 867204 > > - Inline_dentry Inode: 51456 > > > > 2. Desktop #1 > > Utilization: 55% (133310465 valid blocks, 0 discard blocks) > > - Node: 6389660 (Inode: 6289765, Other: 99895) > > - Data: 126920805 > > - Inline_xattr Inode: 2253838 > > - Inline_data Inode: 1119109 > > - Inline_dentry Inode: 187958 > > > > 3. Desktop #2 > > Utilization: 83% (202222003 valid blocks, 1 discard blocks) > > - Node: 21887836 (Inode: 21757139, Other: 130697) > > - Data: 180334167 > > - Inline_xattr Inode: 39292 > > - Inline_data Inode: 35213 > > - Inline_dentry Inode: 1127 > > > > 4. Colleague > > Utilization: 22% (108652929 valid blocks, 362420605 discard blocks) > > - Node: 5629348 (Inode: 5542909, Other: 86439) > > - Data: 103023581 > > - Inline_xattr Inode: 655752 > > - Inline_data Inode: 259900 > > - Inline_dentry Inode: 193000 > > > > 5. Android phone (for reference) > > Utilization: 78% (36505713 valid blocks, 1074 discard blocks) > > - Node: 704698 (Inode: 683337, Other: 21361) > > - Data: 35801015 > > - Inline_xattr Inode: 683333 > > - Inline_data Inode: 237470 > > - Inline_dentry Inode: 112177 > > > > Chao Yu added a functionality to expand this via the -i flag passed to > > mkfs.f2fs back in 2018 - > > https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/commit/?id=baaa076b4d576042913cfe34169442dfda651ca4 > > > > I occasionally find myself in a weird position of having to tell > > people "Oh you should use the -i option from mkfs.f2fs" when they > > encounter this issue only after they’ve migrated most of the data and > > ask back "Why isn’t this enabled by default?". > > > > While this might not be an issue for the foreseeable future in > > Android, I’d argue that this is a feature that needs to be enabled by > > default for desktop environments with preferably a robust testing > > Yes, I guess we need to add some testcases and do some robust tests for > large nat_bitmap feature first. Since I do remember once its design flaw > corrupted data. :( > And I'm glad to report everything's been rock solid ever since that fix :) I'm actively using it on many of my systems. One thing to note here is that my colleague (Alexander Koskovich) ran into an fsck issue with that feature enabled on a large SSD, preventing boot. I didn't encounter it as I never had f2fs-tools installed on my system (which also tells you how robust f2fs was for years on my setup with multiple sudden power-offs without any fsck runs). See here for the fix if you missed it: https://lore.kernel.org/all/cad14+f0fbtxfad_dm-ryfipbaong-b-6hqrms2m4riidx9y...@mail.gmail.com/ Btw, is there a downside (e.g., more disk usage, slower performance) from using large nat_bitmap except for legacy kernel compatibility? I was guessing not, but might as well ask to be sure. Thanks, regards > Thanks, > > > infrastructure. Guarding this with #ifndef __ANDROID__ doesn’t seem to > > make much sense as it introduces more complications to how > > fuzzing/testing should be done. > > > > I’ll also add that it’s a common practice for userspace mkfs tools to > > introduce breaking default changes to older kernels (with options to > > produce a legacy image, of course). > > > > This was a lengthy email, but I hope I was being reasonable. > > > > Jaegeuk and Chao, let me know what you think. > > And as always, thanks for your hard work :) > > > > Thanks, > > regards _______________________________________________ Linux-f2fs-devel mailing list Linux-f2fs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
