Re: F34 problems
On Thu, Apr 29, 2021 at 3:32 PM Chris Murphy wrote: > > On Thu, Apr 29, 2021 at 3:30 PM Chris Murphy wrote: > > > > On Thu, Apr 29, 2021 at 2:51 PM Matti Pulkkinen wrote: > > > > > > to, 2021-04-29 kello 23:38 +0300, Matti Pulkkinen kirjoitti: > > > > In the installer the keymap was definitely the Finnish one. It's only > > > > during startup where I have the US layout. Plymouth has an indicator > > > > below the password entry box to show what the layout is (or is > > > > supposed > > > > to be), and in this case it shows FI. I think what's happening is > > > > that > > > > whatever thing is in charge of setting the keymap during early boot > > > > is > > > > failing to do so, and then defaulting to the US keymap. I just don't > > > > know if that thing would be dracut, or systemd, or what. > > > > > > > > > > And after I got done sending this, I realized I hadn't searched with > > > the term "keymap" specifically because it's not a term familiar to me. > > > Having done that now, it's most likely that the "thing" in question is > > > the kbd package. I'll be filing a bug there; thanks once again for the > > > help! > > > > > > If you're not certain you've found a bug/regression in kbd, I'd just > > assign it to Anaconda. It's the highest level thing most responsible > > for making sure the resulting installation works, and installer folks > > know more about the potential trouble spots below the installer. > > > Looks like you found it? > https://bugzilla.redhat.com/show_bug.cgi?id=1955162 >but in fact I need to enter the password using the US layout I'm not certain this is the same bug, you might file a new one against Anaconda. Because the keyboard layout set in the installer should be the same one picked up by plymouth, otherwise you get a nasty mismatch and can't unlock the encrypted volume. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F34 problems
On Thu, Apr 29, 2021 at 3:30 PM Chris Murphy wrote: > > On Thu, Apr 29, 2021 at 2:51 PM Matti Pulkkinen wrote: > > > > to, 2021-04-29 kello 23:38 +0300, Matti Pulkkinen kirjoitti: > > > In the installer the keymap was definitely the Finnish one. It's only > > > during startup where I have the US layout. Plymouth has an indicator > > > below the password entry box to show what the layout is (or is > > > supposed > > > to be), and in this case it shows FI. I think what's happening is > > > that > > > whatever thing is in charge of setting the keymap during early boot > > > is > > > failing to do so, and then defaulting to the US keymap. I just don't > > > know if that thing would be dracut, or systemd, or what. > > > > > > > And after I got done sending this, I realized I hadn't searched with > > the term "keymap" specifically because it's not a term familiar to me. > > Having done that now, it's most likely that the "thing" in question is > > the kbd package. I'll be filing a bug there; thanks once again for the > > help! > > > If you're not certain you've found a bug/regression in kbd, I'd just > assign it to Anaconda. It's the highest level thing most responsible > for making sure the resulting installation works, and installer folks > know more about the potential trouble spots below the installer. Looks like you found it? https://bugzilla.redhat.com/show_bug.cgi?id=1955162 -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F34 problems
On Thu, Apr 29, 2021 at 2:51 PM Matti Pulkkinen wrote: > > to, 2021-04-29 kello 23:38 +0300, Matti Pulkkinen kirjoitti: > > In the installer the keymap was definitely the Finnish one. It's only > > during startup where I have the US layout. Plymouth has an indicator > > below the password entry box to show what the layout is (or is > > supposed > > to be), and in this case it shows FI. I think what's happening is > > that > > whatever thing is in charge of setting the keymap during early boot > > is > > failing to do so, and then defaulting to the US keymap. I just don't > > know if that thing would be dracut, or systemd, or what. > > > > And after I got done sending this, I realized I hadn't searched with > the term "keymap" specifically because it's not a term familiar to me. > Having done that now, it's most likely that the "thing" in question is > the kbd package. I'll be filing a bug there; thanks once again for the > help! If you're not certain you've found a bug/regression in kbd, I'd just assign it to Anaconda. It's the highest level thing most responsible for making sure the resulting installation works, and installer folks know more about the potential trouble spots below the installer. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F34 problems
On Thu, Apr 29, 2021 at 2:40 PM Matti Pulkkinen wrote: > > to, 2021-04-29 kello 13:54 -0600, Chris Murphy kirjoitti: > > I guess I'm confused about why US keymapping is needed to enter the > > LUKS passphrase during boot. The keymappings need to be the same for > > luksFormat (used by the installer) and luksOpen (used by plymouth > > during startup). I'm used to see a hint what that keymapping is > > supposed to be, but I haven't done a non-US English installation > > recently enough to remember the details :\ > > In the installer the keymap was definitely the Finnish one. It's only > during startup where I have the US layout. Yeah figure 2 and 8 https://docs.fedoraproject.org/en-US/fedora/f33/install-guide/install/Installing_Using_Anaconda/#sect-installation-gui-welcome https://docs.fedoraproject.org/en-US/fedora/f33/install-guide/install/Installing_Using_Anaconda/#sect-installation-gui-keyboard-layout I expect this selection to apply to the LUKS passphrase, and a hint is needed (either as a dracut dropin file or as a boot parameter) so that Plymouth has the correct keymap during startup. >Plymouth has an indicator > below the password entry box to show what the layout is (or is supposed > to be), and in this case it shows FI. I think what's happening is that > whatever thing is in charge of setting the keymap during early boot is > failing to do so, and then defaulting to the US keymap. I just don't > know if that thing would be dracut, or systemd, or what. If plymouth is showing FI, why do you think it's defaulting to US keymap? Is the passphrase you used during installation not working? In that case I would ding plymouth with the bug report at least to just get started. Maybe it turns out that it's a dracut bug for not picking up the proper hint. Or anaconda for not setting the proper hint. These are the five language/locale related test cases: https://fedoraproject.org/wiki/Test_Results:Fedora_35_Rawhide_20210428.n.1_Summary?rd=Test_Results:Current_Summary#Internationalization_and_Localization This one applies to your case I think: https://fedoraproject.org/wiki/QA:Testcase_Non-English_European_Language_Install And it sounds like item 5 in expected results is wrong. I'd think someone would have run into this because QA folks are pretty good about running all of them at one time or another. And also for final, coconut did this test. Coconut is a bot :) so it gets done many times. There may still be a bug, but if so we've got a bug in the testing too. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F34 problems
On Thu, Apr 29, 2021 at 6:41 AM Matti Pulkkinen wrote: > > ke, 2021-04-28 kello 21:49 -0600, Chris Murphy kirjoitti: > > Could you provide details about the installation choices made related > > to language and key mapping? Was it also a Finnish keyboard being > > used > > during the installation? And can you provide either the linux command > > line from grub.cfg or from /proc/cmdline from the booted system if > > you're able to get it booted? > > > > This could be Anaconda or dracut. > > > > Sure. The installer automatically detected my location over WiFi and > selected Finnish language and the Finnish keyboard layout accordingly. > I used a Finnish keyboard during the installation (the integrated one > on the laptop), and didn't select any additional languages or layouts. > I am able to get it booted, the only problem here is that the layout is > wrong during the encryption password prompt. Once I enter the password > (using the US layout) the machine boots just fine, and the keyboard > layout is Finnish as it should be. Here's the Linux command line: > > BOOT_IMAGE=(hd1,msdos2)/vmlinuz-5.11.15-300.fc34.x86_64 > root=UUID=f4bb44bd-f3cb-48eb-b0e5-1e0e06038f6c ro rootflags=subvol=root > rd.luks.uuid=luks-1273c19e-6876-47c5-aff4-e6c2dd781b3c rhgb quiet I guess I'm confused about why US keymapping is needed to enter the LUKS passphrase during boot. The keymappings need to be the same for luksFormat (used by the installer) and luksOpen (used by plymouth during startup). I'm used to see a hint what that keymapping is supposed to be, but I haven't done a non-US English installation recently enough to remember the details :\ But what I can say is neither LUKS1 or 2 have a specific way to indicate the keymapping. Hence dmcrypt/crypsetup folks recommend modhex or base16 or even just the lowercase printable ASCII plus numerals (no special characters). That way you don't end up getting stuck with a passphrase using characters that a keyboard doesn't support. Or alternatively you use UTF8 based passphrase with Finnish keyboard in one keyslot and add another password in a second keyslot that follows the recommendation, e.g. modhex based passphrase. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: F34 problems
On Wed, Apr 28, 2021 at 12:36 PM Matti Pulkkinen wrote: > > Hello! > > I installed Fedora 34 Workstation on my laptop with the latest ISO from > getfedora.org, and have now run into two problems: > > 1. Just before the computer prompts me for the disk encryption > password, the following flashes on the screen quickly: > > [FAILED] Failed to start Setup Virtual Console > [DEPEND] Dependency failed for dracũadditional cmdline parameters > > 2. The second thing is that even though the password prompt indicates a > Finnish keyboard layout (which is what I use), I actually need to enter > my password with a US layout. I suspect this is related to problem 1. > > Which component should I report this problem against? Would dracut do > it, since that's what the error message suggests? Could you provide details about the installation choices made related to language and key mapping? Was it also a Finnish keyboard being used during the installation? And can you provide either the linux command line from grub.cfg or from /proc/cmdline from the booted system if you're able to get it booted? This could be Anaconda or dracut. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: curious service files installed on fresh f34 system
On Wed, Apr 28, 2021 at 12:40 PM Tom Horsley wrote: > > Just been comparing my f33 and f34 partitions, and while every > other service file in the universe lives in /usr/lib/systemd/system > the two new services dbus-org.freedesktop.oom1.service and > dbus-org.freedesktop.resolve1.service are not installed there. > > They appear to be in > > /etc/systemd/system/dbus-org.freedesktop.oom1.service > /usr/share/dbus-1/system-services/org.freedesktop.oom1.service > /etc/systemd/system/dbus-org.freedesktop.resolve1.service > /usr/share/dbus-1/system-services/org.freedesktop.resolve1.service > > The /etc files are symlinks to other services in the "normal" > place, but with a different name (oom rather than oom1) > > Does anyone know what all this convolution is about? Overrides go in /etc, so I suspect that these are Fedora specific differences from upstream. Curiously I don't have /etc/systemd/system/dbus-org.freedesktop.oom1.service - it was clean installed prior to beta though and there were changes with both oomd and resolved during the pre-release period. > I thought the /etc directories were where system administrators were > supposed to install copies to override the system files, why are > things installed there by fedora? You could ask on devel@ list to find out for sure. I'm not certain, but I think there were were systemd 248 changes that were too big to take into Fedora, during pre-release period, and others weren't yet merged upstream, so I think all of this happened because of that. And it'll likely get cleaned up once there's a 248.1 or whatever. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: drive layout
On Wed, Apr 28, 2021 at 6:40 AM Richard Shaw wrote: > >> 1) How is your drive/system laid out regarding your paritions? > > > SSDs are getting cheaper, but not cheap enough for mass storage for me so > currently: > 500GB M.2 NVMe boot/system drive > * /boot > * /boot/efi > * / and /var subvolumes sharing remaining space > * /home 3TB spinning disk > > I like having /var as a separate volume because when it was EXT4 I could do a > reinstall and format / without losing data in /var. I setup using subvolumes > this time before I realized that was no longer possible. I guess it could > make snapshots easier though... > The installer definitely let's you reuse an existing subvolume for the /home mount point (unless it's a snapshot [1]). I'm not certain about reusing a subvolume or snapshot for /var because it contains the RPM database which is inextricably linked to most everything in '/' except for /home. There isn't a very elegant way of dealing with this in the general case, let alone the specifics of snapshotting and rollbacks. The installer will not let you reuse a file system for '/' however. There is an exception for Btrfs, which is it'll reuse an existing Btrfs, but it requires a new subvolume (created by the installer) for '/' mount point. That way it's possible to do a clean install while reusing a subvolume for /home. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1885102 -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: success Re: machine will not boot after moving root
On Sun, Apr 25, 2021, 10:36 AM Michael Hennebry < henne...@web.cs.ndsu.nodak.edu> wrote: > On Sat, 24 Apr 2021, Chris Murphy wrote: > > > You need to use -B to bind mount the pseudo filesystems, and probably > also > > need to include /sys > > > grub2-install /dev/sda > > mount /dev/sda3 /mnt/a3 # already done > mount -B /proc /mnt/a3/proc > mount -B /dev /mnt/a3/dev > mount -B /dev/pts /mnt/a3/dev/pts > mount -B /sys /mnt/a3/sys > > chroot /mnt/a3 > > grub2-install /dev/sda > mount /var # grub2-mkconfig wants it > grub2-mkconfig > /boot/grub2/grub.cfg > exit # out of chroot? > > reboot > > > Yes. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: machine will not boot after moving root
On Sat, Apr 24, 2021, 6:18 AM Michael Hennebry < henne...@web.cs.ndsu.nodak.edu> wrote: > > mount /dev/sda3 /mnt > mount /proc /mnt/proc > mount /dev /mnt/dev > mount /dev/pts /mnt/dev/pts > > You need to use -B to bind mount the pseudo filesystems, and probably also > need to include /sys > > > grub2-install /dev/sda3 > grub2-mkconfig > /boot/grub2/grub.cfg > exit # out of chroot? > > reboot > > Should work? grub2-install /dev/sda -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: machine will not boot after moving root
On Fri, Apr 23, 2021 at 11:17 PM Samuel Sieb wrote: > > On 4/23/21 3:37 PM, Michael Hennebry wrote: > >> With the hard disk only does it find grub and does it display a menu > >> or drop you to the grub> prompt? > > > > I get the grub prompt. > > And this is where you can only see (hd0)? That sounds like grub is > messed up now and can't find it's modules. BIOS GRUB has two kinds of prompts: grub> grub rescue> If it is the former, that means it did find modules (well, in particular it found normal.mod at least), but did not find grub.cfg. That's curious given the sequence we know so far. If it is the latter, then we're in core.img (stage 2) only, and it didn't find either grub.cfg or modules. This means core.img isn't pointing to the proper grub device and path, which is ordinarily baked into the core.img by grub2-mkimage which is part of what grub2-install does. So... yeah it's kinda confusing. But ordinarily a grub2-install /dev/sda will fix that. But I'm not sure in what environment the commands are being issued. grub2-install can be pointed at an alternate root, but grub2-mkconfig is kinda fussy, and doesn't really do the alternate root routine very well. I've only found it reliable to run in a chroot of a properly assembled system. Netinstall rescue option does this for us, but Live boots don't have that so you have to assemble manually: mount /dev/root /mnt/sysimage mount /dev/boot /mnt/sysimage/boot # in this case it's skipped because /boot is a dir on sysroot mount -B /dev /mnt/sysimage/dev mount -B /sys /mnt/sysimage/sys mount -B /proc /mnt/sysimage/proc mount -B /run /mnt/sysimage/run chroot then do grub2-mkconfig -o /boot/grub2/grub.cfg then reboot I don't know if all of those are needed. /run probably isn't. But... > At this point I suggest you use the live boot to backup any config files > you think you might need and just do a re-install. It will be a lot > easier. Also backup your /home files just in case. I agree. This is probably more grief than it's worth. Backup /home to some other drive entirely. And obliterate /dev/sda with automatic partitioning, using "reclaim space" to also Delete All. Or if allergic to both LVM and Btrfs, use "Custom-Automatic" UI, i.e. choose Custom, pick the Standard partition scheme popup, then the blue text to create partitions automatically. Install. Two independent backups for anything important. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: machine will not boot after moving root
On Fri, Apr 23, 2021 at 11:11 PM Samuel Sieb wrote: > His partition list has "extended" on sda4, so it looks like he's correct > about it being BIOS. Yep, I missed that. grub2-install --force /dev/sdXY can work, even though it's not recommended upstream. I think it's still some confusion over something in fstab listing something that isn't found during startup. It might be useful to see the /etc/fstab. Also, booting with rd.timeout=60 would at least get it to a (dracut/initramfs) prompt and from there mount any partition or USB stick, and save out a copy of the journal: journalctl -b > /path/to/rw/mount/journal.log And then post the journal.log and /etc/fstab somewhere. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: machine will not boot after moving root
On Fri, Apr 23, 2021 at 2:51 PM Michael Hennebry wrote: > I've read that I need --force to grub2-install onto a partition. That's sufficiently unreliable that upstream GRUB has recommended against it for almost a decade. First, what firmware type is this? [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS If this really is a computer with BIOS, it's something like: grub2-install /dev/sda And yes that steps on the Windows bootloader in the MBR. But grub2-mkconfig via os-prober should find the Windows bootloader and add a chainloader entry for it. I'm actually betting dollars to donuts that this is a UEFI system, and grub2-install was used, and now it's looking for grub.cfg in the wrong location. Fedora 33, UEFI: sudo dnf reinstall grub2-* shim-* sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg Fedora 34, UEFI: sudo dnf reinstall grub2-* shim-* sudo grub2-mkconfig -o /boot/grub2/grub.cfg > The UUID not found is used by the F33 live DVD. > Changing sda3's UUID to that seems a bad idea. It is. You should use 'sudo blkid' and find out what UUID is on sda3 and put that into fstab for / -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: machine will not boot after moving root
On Thu, Apr 22, 2021 at 1:41 PM Michael Hennebry wrote: > > I'm trying to move my root directory from sda5 to sda3. This is much harder to do without LVM or Btrfs. For LVM there's more steps but basically it depends on 'pvmove'. And for Btrfs it's 'btrfs replace' - both are live migrations and so long as /boot is separate, nothing else needs to be changed or updated. > After running grub2-install and grub2-mkconfig, If the computer has BIOS firmware, then grub2-install to a whole block device is indicated. If it's UEFI firmware, then grub2-install shouldn't be used. Instead use: sudo dnf reinstall grub-* shim-* Also note starting with Fedora 34 that UEFI and BIOS have /boot/grub2/grub.cfg now due to feature: https://fedoraproject.org/wiki/Changes/UnifyGrubConfig > my machine will not boot. > I performed the changes using a F33 live DVD which also fails to boot. > After plugging in a USB-connected SD card with Centos 7 on it, > the F33 live DVD decided to boot. > It is what I am running now. > The complaint is UUID 2b82edc2-4eb2-44a0-8b5b-c71da0de9b3a not found. > While running F33 live DVD I get > blkid | grep 2b82edc2-4eb2-44a0-8b5b-c71da0de9b3a > /dev/loop1: LABEL="Anaconda" UUID="2b82edc2-4eb2-44a0-8b5b-c71da0de9b3a" Sounds like /etc/fstab has the wrong UUID. > I backed up sda5 with rsync -auHAXUUx . > I copied from the backup with the same flags. > I edited the new etc/fstab to point at sda3, > actually LABEL=local3slash . OK but did you label the file system 'local3slash' at mkfs time? > > As root: > grupb2-install This is missing a block device. > grub2-mkconfig -o .../local3slash/boot/grub2/grub/cfg That's probably only going to work in a chroot where everything is properly assembled in advance. The easiest way to get it properly assembled for you is the rescue option in the troubleshooting menu on netinstall media. And since /boot is a dir in the copied sysroot... yeah dracut. A neat hint is to search a live install's program.log for 'rsync' to see what command is used for a live install. 08:18:29,672 INF program: Running... rsync -pogAXtlHrDx --exclude /dev/ --exclude /proc/ --exclude /tmp/* --exclude /sys/ --exclude /run/ --exclude /boot/*rescue* --exclude /boot/loader/ --exclude /boot/efi/loader/ --exclude /etc/machine-id /run/install/source/ /mnt/sysroot A lot of that does translate into -a option. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Abysmal VM performance even with btrfs+nodatacow
On Sat, Apr 3, 2021 at 1:32 PM Richard Shaw wrote: > > I just setup a Windows 10 to do some debugging of a mingw project. Gnome > Boxes made the process very simple but the performance is horrendous. > > Even after the following I still wouldn't consider it usable. > > # chattr +C ~/.local/share/gnome-boxes/images > # cd ~/.local/share/gnome-boxes/images > # mv win10 ../ > # cat ../win10 win10 > > $ lsattr > ---C ./win10 > > Here's a screenshot showing windows process manager with < 1MB transfer rates > but 100% disk busy while installing MinGW w64 in windows, which is not a huge > amount of data... > > https://www.dropbox.com/s/2m0p3lptbqwd7lp/Screenshot%20from%202021-04-03%2014-22-02.png > > Any tips? Or do I just need to buy a cheap SSD and format it EXT4 just for > virtual machine images? Pretty sure GNOME Boxes creates sparse qcow2 files, and virt-manager creates them with option preallocation=falloc, so this is worth a shot. I'm not sure it can account for all of the performance loss though. Check the configuration 'virsh dumpxml $vmname > vmname.xml' and see if it's using IDE or SATA for the drive. It is possible to use VirtIO for Windows after you've installed the VirtIO drivers for Windows and the performance is much better. I'm reluctant to outright recommend what I use, cache mode unsafe, but it performs a *lot* better. But if the host crashes, the VM image can be damaged beyond repair. If the guest crashes, it should be OK. (I force quit VMs all the time, mainly because I am trying to damage file systems.) I think the default cache mode for GNOME Boxes is none. It is possible to change it via virsh to writeback and see if that's better or worse. And if it's better then that maybe could become the default for Windows guests. I don't expect unsafe could ever be a default. https://documentation.suse.com/sles/11-SP4/html/SLES-kvm4zseries/cha-qemu-cachemodes.html -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mdadm -> BTRFS conversion
On Fri, Apr 2, 2021 at 4:23 AM Patrick O'Callaghan
wrote:
>
> On Thu, 2021-04-01 at 23:52 -0600, Chris Murphy wrote:
> > It's not an SMR concern, it's making sure the drive gives up on
> > errors
> > faster than the kernel tries to reset due to what it thinks is a
> > hanging drive.
> >
> > smartctl -l scterc /dev/sdX
> >
> > That'll tell you the default setting. I'm pretty sure Blues come with
> > SCT ERC disabled. Some support it. Some don't. If it's supported
> > you'll want to set it for something like 70-100 deciseconds (the
> > units
> > SATA drives use for this feature).
>
> One doesn´t and one does:
>
> # smartctl -l scterc /dev/sdd
> smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
> (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control command not supported
>
> # smartctl -l scterc /dev/sde
> smartctl 7.2 2021-01-17 r5171 [x86_64-linux-5.11.10-200.fc33.x86_64]
> (local build)
> Copyright (C) 2002-20, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control:
>Read: 85 (8.5 seconds)
> Write: 85 (8.5 seconds)
>
> So I guess the /dev/sde drive is set correctly, right? Or would you
> recommend disabling SCT ERC for this drive?
Leave /dev/sde alone, 85 deciseconds is fine.
Not much can be done with /dev/sdd itself directly. But it is possible
to increase the kernel's command timer for this drive. The usual way
of doing this is via sysfs. I think it can be done with a udev rule
as well, but I'm having a bit of a lapse how to do it. Udev needs to
identify the device by serial number or wwn, but changing the timeout
via sysfs requires knowing that the /dev node is - which of course can
change each time you boot or plug the device in. I don't know enough
about udev. But there should be examples on the internet or you can
just fudge it with the linux-raid wiki guide.
The alternatives? Change the timeout for all /dev/ nodes. That's how
things are by default on Windows and macOS, they just wait a long time
before resetting a drive, giving it enough time for it to give up on
its own. The negative side effect is you might get a long delay
without errors, should the device develop marginally bad sectors.
Another alternative is to just leave it alone, and periodically check
(manually or automate it somehow) for the telltale signs of bad
sectors masked by SATA link resets.
Looks like this:
kernel: ata7.00: status: { DRDY }
kernel: ata7.00: failed command: READ FPDMA QUEUED
kernel: ata7.00: cmd
60/40:f0:98:d2:2b/05:00:45:00:00/40 tag 30 ncq dma 688128 in
res
40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
With this interlaced occasionally
kernel: ata7: hard resetting link
If it happens *then* you can increase the timeout manually, and
initiate a scrub. As long as the timeout is set high enough (most
sources suggest 180 seconds which, yes, it's incredible) eventually
the drive will give up, spit out an error, and Btrfs will fix up that
sector by overwriting it with good data. It could be months, years, or
never, before it happens.
--
Chris Murphy
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: mdadm -> BTRFS conversion
On Thu, Apr 1, 2021 at 5:53 AM Patrick O'Callaghan wrote: > > On Wed, 2021-03-31 at 18:00 -0600, Chris Murphy wrote: > > Nothing to add but the usual caveats: > > https://raid.wiki.kernel.org/index.php/Timeout_Mismatch > > That´s pretty scary, though the drives I´m using are 1TB units > scavenged from my extinct NAS so are unlikely to be SMR. They´re both > WD model WD10EZEX drives. It's not an SMR concern, it's making sure the drive gives up on errors faster than the kernel tries to reset due to what it thinks is a hanging drive. smartctl -l scterc /dev/sdX That'll tell you the default setting. I'm pretty sure Blues come with SCT ERC disabled. Some support it. Some don't. If it's supported you'll want to set it for something like 70-100 deciseconds (the units SATA drives use for this feature). And yeah, linux-raid@ list is chock full of such misconfigurations. It filters out all the lucky people, and the unlucky people end up on the list with a big problem which generally looks like this: one dead drive, and one of the surviving drives with one bad sector that was never fixed up through normal raid bad sector recovery mechanism, because the kernel's default is to be impatient and do a link reset on consumer drives that overthink a simple problem. Upon link reset, the entire command queue in the drive is lost, and now there's no way to know what sector it was hanging on, and no way for raid to do a fixup. The fixup mechanism is, the drive reports an uncorrectable read error with a sector address *only once it gives up*. And then the md raid (and btrfs and zfs) can go lookup that sector, find out what data is on it, go find its mirror, read the good data, and overwrite the bad sector with good data. The overwrite is what fixes the problem. If the drive doesn't support SCT ERC, we have to get the kernel to be more patient. That's done via sysfs. > > > I use udev for that instead of init scripts. Concept is the same > > though, you want SCT ERC time to be shorter than kernel's command > > timer. > > I´ve been using MD for a while and haven´t seen any errors so far. And you may never see it. Or you may end up being an unlucky person with a raid who experiences complete loss of the array. When I say comes up all the time on linux-raid@ list, it's about once every couple of weeks. It's seen most often with raid5 because it has more drives, thus more failures, than raid1 setups. And tolerates only one failure *in a stripe*. Most everyone considers a failure a complete drive failure, but drives also partially fail. Two drives partially failing the sectors in the same stripe is pretty astronomical. But if one drive dies, and *any* of the remaining drives has a bad sector that can't be read, the entire stripe is lost. And depending on what's in that stripe, it can bring down the array. So, what you want is for the drives to report their errors, rather than the kernel doing link resets. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Suggestion to improve documentation of Changes/BtrfsTransparentCompression
On Wed, Mar 31, 2021 at 6:33 AM Richard Shaw wrote: > > On Wed, Mar 31, 2021 at 12:29 AM misterx42--- via users > wrote: >> >> Just a small thing, but as someone who never actually played around with >> filesystems so far, I was looking to try out the changes outlined in >> Changes/BtrfsTransparentCompression, but the command on the page: >> >> btrfs filesystem defrag -czstd -r > > > If you add the compression option to /etc/fstab, is the -czatd option needed > here? > I don't understand the question. Setting compression on /etc/fstab, and using compression option when defragmenting, can be used together or separately. It's legit to just change /etc/fstab, and reboot or remount, and now all new writes will be compressed. Old writes remain uncompressed. If you open a file, modify it, and save it, the degree to which it becomes compressed depends on application handling. Many applications open the entire file, and write out a whole new file (new temp name, new inode) and renameat() to replace the old file entirely. In that case the whole thing is subject to being compressed. If the application appends, inserts, overwrites specific blocks then only those blocks are subject to possible compression. The defragment method won't cause new writes to be compressed, it's a way of causing existing files to become compressed. It's completely reasonable to just change the fstab entry and allow files to become slowly compressed over time via attrition. Or you can choose to just compress your /home files and let system files get replaced with compressed versions as you normally updated the OS. Still another variation for new clean installs is to set 'btrfs property set /home compression none' which will set a "do not compress" flag on the (mostly) empty /home, and any new files will inherit this flag and not be compressed. In this way you get the advantage of the OS being compressed, without /home being compressed if for whatever reason you prefer to not compress. It's possible there are some workloads, likely heavy write workloads, on NVMe, that hypothetically are slower with compression. And that's mainly because NVMe is just so fast. I myself use compress=zstd:1 even on NVMe, I can't say I notice a difference with or without compression, other than the space savings. But then heavy write workloads are more likely to benefit from the reduced write amplification that comes with compression. Whereas most of us with typical write patterns, our SSD will last well past its warranty and might even be looking forward to the occasional failure just to have an excuse to upgrade. :D For sd cards, which are sad and remarkable at the same time, they definitely last longer with compression enabled. And often perform better just because they're slow and there's often plenty of CPU to spare to compress and end up doing less writes, less IO overall, improving performance a bit. If what I'm writing can be compressed 50%, and the device maxes out at 35 M/s writes, in effect I get 70 M/s writes. Latency per IO is still the same, but again less writes means less IO competition so there's still some gain even on the latency front. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: mdadm -> BTRFS conversion
Nothing to add but the usual caveats: https://raid.wiki.kernel.org/index.php/Timeout_Mismatch I use udev for that instead of init scripts. Concept is the same though, you want SCT ERC time to be shorter than kernel's command timer. The btrfsmaintenance package has a scrub.timer that you can configure and enable. It's sufficient to scrub once a month or two, but doesn't hurt to do it more often. There is a nagios btrfs plugin somewhere for monitoring. Otherwise something that parses 'btrfs device stats' for value changes would also work. A rather different thing about Btrfs compared to mdadm raid, is there's no concept of faulty drives, i.e. btrfs doesn't "kick" drives out. Since it can unambiguously know if any block is corrupt or not, it keeps a pesky failing drive in, and just complains (a lot) about the bad blocks while using the good blocks. Device replacement should use 'btrfs replace' rather than 'btrfs device add/remove'. More info in 'man mkfs.btrfs' and 'man 5 btrfs'. It's possible to simulate various problems with loop devices or in a VM. A fun one is setting up dm-flakey for one of the mirrors. A bit easier to chew off is to compile btrfs-corrupt-block.c from btrfs-progs source (it is not included in the btrfs-progs package in Fedora) which has various options for corrupting data and metadata, ability to choose which copy gets the damage, etc. It can be surprising how verbose btrfs is with just one data block is corrupt, when shared among multiple snapshots. It'll tell you about all of the instances of the files sharing that one bad block. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Suggestion to improve documentation of Changes/BtrfsTransparentCompression
On Wed, Mar 31, 2021 at 2:00 AM Francisco Tissera wrote: > > Hello there, > > > So, let's see if I got it right: if i put in / as a path, it should > compress the entire harddrive right? I think so, and it'll just complain about /boot because it's not Btrfs (at least not by default). If your /boot is Btrfs, it's ok because GRUB supports it. > If not, how should I go about doing that? I don't think it would be a > good idea to compress the /boot directory, cause the docs don't > recommend it, but how would I go about compressing the entire root > partition with all the file and such? > > The path /dev/sda gives me an error. That's a /dev node not a path to a directory or file. But it's a valid point in that there might be some noise whenever it touches the pseudo file systems like /dev, /sys, /run, /proc which are not on Btrfs either. Still, it should just skip those (with some noise) and continue on. If not then I've got an idea for a work around. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Suggestion to improve documentation of Changes/BtrfsTransparentCompression
On Tue, Mar 30, 2021 at 11:29 PM misterx42--- via users wrote: > > Just a small thing, but as someone who never actually played around with > filesystems so far, I was looking to try out the changes outlined in > Changes/BtrfsTransparentCompression, but the command on the page: > > btrfs filesystem defrag -czstd -r > > doesnt work without an additional argument. This is pretty obvious for people > who already worked with filesystems before, but I think it would be a good > idea to add a small, one sentence explanation of how to add the required path > to the end of the command (ie / in my case). It would help people like me > from getting confused and hopefully bring in more casual tinkerers trying the > new change out. > > Thanks. Thanks for the feedback, I've updated that section. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: BTRFS settings for media storage
On Mon, Mar 29, 2021 at 7:17 AM Richard Shaw wrote: > > So after about 12 forced power offs while copying data (via rsync) this is > the only output from dmesg: > > # dmesg | grep -i btrfs > [0.776375] Btrfs loaded, crc32c=crc32c-generic, zoned=yes > [5.497241] BTRFS: device fsid d9a2a011-77a2-43be-acd1-c9093d32125b devid > 1 transid 389 /dev/sdc scanned by systemd-udevd (732) > [5.521210] BTRFS: device fsid d9a2a011-77a2-43be-acd1-c9093d32125b devid > 2 transid 389 /dev/sdd scanned by systemd-udevd (743) > [6.743097] BTRFS info (device sdc): disk space caching is enabled > [6.743100] BTRFS info (device sdc): has skinny extents This is the usual case. Btrfs kernel code reads the super which is only pointing to a completely consistent set of btrees. A variation on this is if a process making use of fsync was interrupted by the crash, you might also see this line: BTRFS info (device vda3): start tree-log replay That also is normal and fine. No fsck required, nor scrub required. You don't need to do anything. If the hardware is doing the correct thing, you can hit it with power failures while writing all day long and the file system will never care about it. This is by design. But also Btrfs developers wrote dm-log-writes expressly for power failure testing, and is now used in xfstests for regression testing all the common file systems in the kernel. So there's quite a lot of certainty that write ordering is what it should be. > [ 61.730008] BTRFS info (device sdc): the free space cache file > (1833981444096) is invalid, skip it > > Should I worry about the last line? Nope. It's just gone stale, because it didn't get updated prior to the power fail. They rebuild quickly on their own. And it's not critical metadata, just an optimization. (This is a reference to the v1 space cache which exist as hidden files in data block groups. The stable and soon to be default v2 space cache tree moves this to metadata block groups and it's quite a lot more resilient and more performant for very busy and larger file systems. Anyone can switch to v2 by doing a one time mount with the mount option space_cache=v2. Currently it needs to be a new mount, there's a bug that causes it to claim it's using v2 but it's not actually setting up the v2 tree. Once this mount option is used, a feature flag is set, and it'll always be used from that point on. It's not something that goes in fstab. Use it one time and forget about it. Sorta like a file system upgrade if you will. Note that large file systems might see a long first time mount with this option being initially set. I've anecdotally heard of it taking hours for a 40T file system, because the whole tree must be created and written before the mount can complete. For me on a full 1T file system it took *maybe* 1 minute.) -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: How to debug grub2-efi -> kernel transition issues?
On Mon, Mar 29, 2021 at 1:55 AM Richard W.M. Jones wrote: > > > I have a weird problem on an old Asus Zenbook UX305C where new kernels > cannot be installed by grub. Specifically what happens is they appear > in the boot menu fine, but if you try to boot them then the machine > hangs hard with a completely black screen. > > Oddly the kernel installed by Anaconda can boot, but obviously this > makes upgrading the kernel RPM impossible. > > My real question is how on earth do I start debugging this? > > I edited the kernel command line, removed “rhgb quiet”, added > “loglevel=9 nomodeset” but still no output at all is visible before > the hang. The machine doesn't have a serial port. > > Any ideas? Does grub have debugging that can be enabled somehow? With rhgb and quiet removed, you should immediately get a boot scroll from the kernel and a panic if that's what's happening. If not then this is somewhere in between the hand off. When GRUB has a problem loading kernel or initramfs, it complains and goes back to GRUB. So this sounds like the kernel and initramfs did load, and boot command is executed and something fails at that point or right in the kernel at the earliest stage. I'd ask in #fedora-kernel to be sure of a better idea. But my idea is edit the failing grub menu entry (in GRUB) and after the initrd line add two more lines set pager=1 set debug=all Then F10 or Ctrl-X to boot that edited entry. You should get maybe just 4 lines of cryptic grub debug code (in my working case it doesn't mean anything to me) and then it goes black then i get kernel boot scroll. If you get a ton of errors, probably failure in grub. If you don't, probably already handed off to the kernel and it faceplanted during its very early bootstrapping sequence before we even usually see anything. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: BTRFS settings for media storage
On Sun, Mar 28, 2021 at 6:51 PM Tim via users wrote: > > I'm recently seeing info like this in logwatch emails: > > **Unmatched Entries** > Device: /dev/sda [SAT], CHECK POWER STATUS spins up disk (0x81 -> > 0xff) > > Which makes little sense to me. The system is a 24/7 server, not often > rebooted. It's a solid state drive, and I don't know what the hex that > means (pun intended). I've no idea if that's an error, or if it's just > telling me that drive has changed modes (idle/active). I'm not sure. The smartmontools folks probably know. They have a mailing list. Ask them and let us know? > Logically I'd expect that if SMART thought the drive might need > checking or chucking, it'd start to give me useful warnings ahead of > time, and I might be lucky enough to backup my files before disaster > struck. But the warnings ain't that useful. And, of course, it's > entirely possible for a drive to spontaneously fail before any > scheduled SMART test took place. The trend seems to be it can be semi-useful for HDD. The pending sectors and seek errors going up quickly is a pretty good warning something is going wrong. For SSDs, your first indication for prefail is often checksum errors, at least in Btrfs land. Either it returns garbage or zeros. It seems the drive itself is not likely to report an uncorrectable read or write error like you'd get on HDD. And the next level of failure for SSD is it goes read-only. Not the file system, the drive - and this too is totally silent in my handful of experiences with this failure mode, happily accepts all write command without error but none of them are persistent. Another mode of SSD failure that's common is, it just vanishes off the bus. Does not say bye! It just goes bye! -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: BTRFS settings for media storage
On Sun, Mar 28, 2021 at 4:31 PM George N. White III wrote: > > Your plan needs to consider backups and/or replication (to cloud or another > site).It is easy and cheap to lose data. Not losing data is not easy and > not cheap. Exactly this. If the data is important, it's backed up. If it's not backed up, in some sence both fair and unfair, the data isn't important to you. And I mean, it's an unfair penalty to find out the importance of backups via data loss. It's a disproportionate penalty. Raid1 is not a backup. If you accidentally delete a file, for sure raid1 will ensure both mirrors record the accident. Snapshots aren't a backup, because they aren't fully independent from either the file system or hardware. (They can make it easier to do a 'btrfs restore' in some cases - but I don't want folks thinking they can depend on restore, it's seriously a PITA. Very capable, but tedious and requires patience.) In aggregate, with large volume users with consumer hardware, we aren't seeing Btrfs fall over more often than other file systems. It's true that it's harder to fix if it gets broken. It's also true you have multiple chances of recovery if it gets broken but all of them are less convenient than having a backup. Nevertheless, if it breaks, we still want bug reports because, to the degree that it is possible, we don't want it breaking. Btrfs is quite noisy (in dmesg) when there are problems. It's not subtle about it, by design. Corruption of any kind is not normal, the source should be found. Persistent stats are also retained in the file system metadata, retrievable by: btrfs device stats /mntpoint or /dev/ node(s) -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: BTRFS settings for media storage
On Sun, Mar 28, 2021 at 11:47 AM Richard Shaw wrote: > > I appreciate all the detailed response, but at the same time the answers seem > often bi-polar... You can do all these great things with BTRFS! But even if > you test your raid array multiple times, a bad firmware may still eat all > your data :) It's uncommon even in the single device case. It takes a combination: a crash or power failure happening at the same time the firmware does something wrong. If you add raid1 to the mix, it's even less common all problems happen at the same time to two drives, in particular if the two drives don't share firmware bugs. But if there is a problem, there's a decent (not certain) chance of recovery by doing: mount -o rescue=usebackuproot There are three backup roots. If one is good, mount succeeds and nothing more need be done. But you might lose up to a couple of minutes of writes. Basically Btrfs does a rollback to a known good root and drops all the bad ones. Is this eating your data? Yes, as much as 2 minutes worth. If that doesn't work, what comes next depends on the kernel messages shown when the mount fails. And that unfortunately takes familiarity with Btrfs mount failure messages. My advice if anyone runs into a problem is to report a complete dmesg, and btrfs check --readonly. And in the meantime they can try: mount -o rescue=all That is a read-only mount that can skip over many kinds of significant root tree damage, and you can still get your data out using normal tools like rsync, cp, or any other backup program. And freshen up the backups just in case the problem can't be fixed. If that doesn't work there's 'btrfs restore' which is an offline scrape utility. Ugly, but again data not eaten. > I know you can't give absolute answers sometimes, but it feels like you're > often a btrfs cheerleader and critic at the same time :) Yes. > It sounds like there needs to be an easy to find list of known good and known > bad drives to use (be it btrfs or other similar filesystem). Yes but that takes a lot of data and work to find the trends and then publish it and maintain it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Identifying what is accessing a USB device.
On Fri, Mar 26, 2021 at 5:21 PM Patrick O'Callaghan wrote: > > I have an external USB-3 2-disk docking station, and a script which can > power up and down the drives as needed. > > I have a systemd automount unit that correctly powers up the dock when > accessed, then mounts the drives (thanks Ed). > > After an idle time, automount unmounts the drives. A script detects > when this happens and powers them down ... *at which point they > immediately power up again, and remain up until I intervene manually, > even though they are unmounted*. > > This never happens if I run the script directly from the command line > (i.e. the drives power down and stay down). > > Clearly the docking unit isn't just doing this flakily on its own. > Something is making it happen, and I've no idea how to discover what it > is except that it seems to be correlated with systemd in some way. > > All of the above is 100% reproducible. > > I'm open to suggestions if anyone has any ideas. smartd? I use this line in smartd.conf to keep it from waking up the drive all the time. /dev/disk/by-id/wwn-0x5000c500a93cae8a -l selftest -s L/../15/./22 -n standby,250 Since it's unmounted, fatrace won't work, but blktrace will.. blktrace -d /dev/sdb -o - | blkparse -i - It will generate a lot of lines but it'll also report the process that's sending commands to the drive. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
rent devices, they have their own pool of inodes. You can't create hardlinks across subvolumes (you can create reflinks across subvolumes, but there is a VFS limitation that enforces no reflinks the cross mount points.) > > In summary: > - if a process calls fsync and then complains about having to wait > to get unblocked, it is just creating its own problem (are you doing fsync > of big things in your UI thread?) > - if a process gets heavily delayed because another process is doing > fsync the kernel is not doing its job in terms of fairness > It very much depends on the workload. And there's also cgroup io.latency to consider as well. The desktop is in a better position to make decisions on what's more important: UI/UX responsiveness is often more important than performance. > I can agree that reality may not be ideal, but I don't like this > attitude of "dropping the ball" by disabling caching here and there > because (provocative paradox) web browser authors are using fsync > for bookmarks and cookies DB in the UI thread. No one has suggested disabling caching. Reducing the *time* to start write back is what was suggested and we don't even know if that matters anymore, and I wasn't even the one who first suggested it, goes back to Linus saying the defaults are crazy 10 years ago. > NOTE: > I know about eatmydata, I've used it sometimes. > There is also this nice trick for programs stupidly doing too many fsync: >system-nspawn --system-call-filter='~sync:0 fsync:0' That is awesome! Way easier to deal with than eatmydata. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Thu, Mar 25, 2021 at 7:26 PM Chris Murphy wrote: > > The problem is well understood for some time. > https://lwn.net/Articles/572911/ This is an update on that 8 year old story. Last year writebehind patches were proposed, and a discussion ensued. https://lore.kernel.org/linux-mm/CAHk-=whf2bq8xqvbf8yuxrznbyrp-otgchsy9dgdnrftxps...@mail.gmail.com/ I don't think an fsync/fdatasync centric workload is going to be affected by these knobs. And it'd take testing to find out if they affect anything we care about, but it's interesting that changing them could improve real world performance while negatively impacting synthetic benchmarking. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Can't mount XFS filesystems on Fedora 33
What kernel version? i.e. uname -r At the time of the mount failure, what do you get for: $ lsmod | grep xfs xfs 1892352 0 $ dmesg | grep -i xfs [ 717.658384] SGI XFS with ACLs, security attributes, scrub, quota, no debug enabled I suspect you get neither of the above results. Try: $ sudo modprobe xfs Now try mount. Does that work? -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Thu, Mar 25, 2021 at 7:26 PM Chris Murphy wrote: > > The defaults are crazy. > https://lwn.net/Articles/572921/ > > Does this really make a difference though outside the slow USB stick > example? I don't know. Seems like it won't for fsync heavy handedness > because that'll take precedence. There's more about this in that same 8 year old thread. https://lore.kernel.org/linux-mm/2013032211.GT6188@dastard/ I wonder what the state of the implied work is now, and whether the applications should be making better use of fadvise to inform the kernel of its intentions/expectations with respect to the files it's writing. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Thu, Mar 25, 2021 at 6:39 AM Richard Shaw wrote: > > On Wed, Mar 24, 2021 at 11:05 PM Chris Murphy wrote: >> Append writes are the same on overwriting and cow file systems. You >> might get slightly higher iowait because datacow means datasum which >> means more metadata to write. But that's it. There's no data to COW if >> it's just appending to a file. And metadata writes are always COW. > > > Hmm... While still annoying (chrome locking up because it can't read/write to > it's cache in my /home) my desk chair benchmarking says that it was > definitely better as nodatacow. Now that I think about it, initial syncing > I'm likely getting the blocks out of order which would explain things a bit > more. I'm not too worried about nodatasum for this file as the nature of the > blockchain is to be able to detect errors (intentional or accidental) already > and should be self correcting. Is this information in a database? What kind? There are cow friendly databases (e.g. rocksdb, sqlite with WAL enabled), and comparatively cow unfriendly ones - so it may be that setting the file to nodatacow helps. If there's also multiple sources of frequent syncing, this can also exacerbate things. You can attach strace to both processes and see if either or both of them are doing sync() of any kind, and what interval. I'm not certain whether bcc-tools biosnoop shows all kinds of sync. It'd probably be useful to know both what ioctl is being used by the two programs (chrome and whatever is writing to the large file), as well as their concurrent effect on bios using biosnoop. > >> You could install bcc-tools and run btrfsslower with the same >> (exclusive) workload with datacow and nodatacow to see if latency is >> meaningfully higher with datacow but I don't expect that this is a >> factor. > > > That's an interesting tool. So I don't want to post all of it here as it > could have some private info in it but I'd be willing to share it privately. TIME(s) COMM PIDDISKT SECTOR BYTES LAT(ms) There's no content displaced in any case. > > One interesting output now is the blockchain file is almost constantly > getting written to but since it's synced, it's only getting appended to (my > guess) and I'm not noticing any "chair benchmark" issues but one of the > writes did take 1.8s while most of them were a few hundred ms or less. 1.8s is quite a lot of latency. It could be the result of a flush delay due to a lot of dirty data, and while that flush is happening it's not going to be easily or quickly preempted by some other process demanding its data be written right now. Btrfs is quite adept at taking multiple write streams from many processes and merging the writes into sequential writes. Even when the writes are random, Btrfs tends to make them sequential. This is thwarted by sync() which is a demand to write a specific file's outstanding data and metadata right now. It sets up all kinds of seek behavior as the data must be written, then the metadata, then the super block. > > I'm pretty sure that's exactly what's happening. But is there a better I/O > scheduler for traditional hard disks, currently I have: > > $ cat /sys/block/sda/queue/scheduler > mq-deadline kyber [bfq] none I don't know anything about the workload still so I'm only able to speculate. Bfq is biased toward reads and is targeted at the desktop use case. mq-deadline is biased toward writes and is targeted at server use case. This is perhaps more server-like in that the chrome writes, like firefox, are sqlite databases. Firefox enables WAL, but I don't see that Chrome does (not sure). You could try mq-deadline. > $ ls -sh data.mdb > 101G data.mdb > > A large bittorrent download should also be similar since you don't get the > parts in order, but perhaps it's smart enough to allocate all the space on > the front end? That's up to the application that owns the file and is writing to it. There's going to be a seek hit no matter what because they're both written and read out of order. And while they might be database files, they aren't active databases - different write pattern. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: BTRFS settings for media storage
On Thu, Mar 25, 2021 at 6:00 AM Richard Shaw wrote:
>
> So how long do you wait until you consider the drive "good"? :)
>
> I'm not in a hurry so I could setup two of the drives in a RAID1 mirror and
> copy my media over and just let it run for a while before I add disks 3 & 4.
We don't have much control over when and how bugs manifest. It could
be true that drive firmware is improperly reordering 1 in 100 commits,
or only a particular sequence of flushing, or always. Such behavior
not a problem by itself, it takes a power fail or a crash to expose
the problem at just the right time. The whole point of write ordering
is to make sure the file system is consistent.
In the case of btrfs, the write order is simplistically:
data->metadata->flush/fua->superblock->flush/fua
What makes this safe with copy on write is no data or metadata is
overwritten, so there's no in between state. All the data writes in a
commit are represented by metadata (the btrees) in that commit, and
those trees aren't pointed to as current and valid until they are on
stable media. That's the point of the first flush. Then comes the
superblock which is what points to the new trees.
The problem happens if the super is written pointing to new trees
before all the metadata (or data) has arrived on stable media. And
then you get a power fail or crash. Now the super block is pointing to
locations that don't have consistent tree state and you get some
variation on mount failure.
>> Still, I'd probably opt to set a udev rule for all the drives and
>> disable the write cache. It isn't worth the trouble.
>
>
> I know you mentioned this on another thread but google is failing me as to
> how to do it. Do you have a useful link?
hdparm -W0 but double check that, there's w and W and one of them
controls write cache, the other is dangerous.
$ cat /etc/udev/rules.d/69-hdparm.rules
ACTION=="add", SUBSYSTEM=="block", \
KERNEL=="sd*[!0-9]", \
ENV{ID_SERIAL_SHORT}=="WDZ47F0A", \
RUN+="/usr/sbin/hdparm -B 100 -S 252 /dev/disk/by-id/wwn-0x5000c500a93cae8a"
$
--
Chris Murphy
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Thu, Mar 25, 2021 at 8:59 AM Roberto Ragusa wrote: > > On 3/25/21 4:25 AM, Chris Murphy wrote: > > > It might be appropriate to set dirty_bytes to 500M across the board, > > desktop and server. And dirty_background to 1/4 that. But all of these > > are kinda rudimentary guides. What we really want is something that > > knows what the throughput of the storage is, and is making sure there > > isn't more than a few seconds of writeback needed at any given time. > > > > The default, dirty_ratio 20%, is high by today's memory standards. But > > upstream will not change it. All kernel knobs are distro > > responsibility to change from the defaults. > > I don't agree with the base reasoning. > There is nothing wrong in having many gigabytes of dirty data in memory, > if the machine has enough RAM to do it. It is one of the things that make > the difference between Linux and toy systems. The problem is well understood for some time. https://lwn.net/Articles/572911/ > "500M will be enough" sound like the historical "640k will be enough", > because 500M could be flushed in a fraction of a second on modern SSDs. Which would be an Ok combination. What you don't want is 1G of dirty data accumulating before your 80 M/s drive starts writeback. If the process fsync's a file, now you've got a blocked task that must write out all dirty data right now, and the whole storage stack down to the drive is not going to easily stop doing that just because some other program fsync's a 100K cache file. There will be a delay. Now, if this delay causes that program to stall from the user's perspective, is that well behaved? I mean come on, we all know web browser cache files are 100% throw away garbage, they're there to make things faster, not to cause problems and yet here we are. There's enough misuse of fsync by applications that there's a utility to cause fsync to be dropped. https://www.flamingspork.com/projects/libeatmydata/ In fact some folks run their web browser in a qemu-kvm with cache mode "unsafe" to drop all the fsyncs. > What you really want is that if there are 40GB of outstanding data going > to the disk, processes are still: > 1) able to write to the disks without heavy latency (and delaying it in > memory is exactly achieving that) > 2) able to read the disks without heavy latency, which is something the > disk scheduling code will care to provide (reads have priority over writes). If you have 40G of dirty data and your program says "fsync it" you've got 40G of data that has been ordered flushed to stable media. Everything else wanting access is going to come close to stopping. That's the way it works. You don't get to "fsync this very important thing..but oh yeah wait I wanna read a b.s. chrome cache file hold on a sec. Ok thanks, now please continue syncing." That's in effect something that multiqueue NVMe can do. So there's a work around. > The kernel has even got per-device queues to avoid a slow USB drive to stall > the I/O for the other devices. > > If the filesystem is not able to read 50kB because there are 10GB of > dirty data in memory, the problem is in the filesystem code. The defaults are crazy. https://lwn.net/Articles/572921/ Does this really make a difference though outside the slow USB stick example? I don't know. Seems like it won't for fsync heavy handedness because that'll take precedence. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: BTRFS settings for media storage
On Wed, Mar 24, 2021 at 3:47 PM Richard Shaw wrote: > > I've been running MythTV for about 10 years now and I've finally outgrown my > media storage, currently a single 4TB disk drive. I have purchased 3 > additional drives of the same model and plan to put them into a BTRFS RAID1 > array. > > Setting nodatacow on the media directories is a no-brainer, but what other > optimizations can I do? nodatacow means nodatasum and no compression. If the file becomes corrupt, it can't be detected or corrected. Due to all the firmware bugs, I tend to mix make/model drives rather than get the same ones that are all going to have the same bug at the time time if something goes wrong like a power fail or crash right after going a bunch of writes. Whereas separate bugs, btrfs can always do fix ups from the good drive whenever the bad one misbehaves. If they've proven their reliability in case of crash or power fail, as in start doing a big file(s) copy, and yank power on all the drives at once: Reboot. Reattach the drives. Remount. Any errors? Does it mount? If you can do this 10x without errors, it might be OK. Still, I'd probably opt to set a udev rule for all the drives and disable the write cache. It isn't worth the trouble. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Wed, Mar 24, 2021 at 6:09 AM Richard Shaw wrote: > > I was syncing a 100GB blockchain, which means it was frequently getting > appended to, so COW was really killing my I/O (iowait > 50%) but I had hoped > that marking as nodatacow would be a 100% fix, however iowait would be quite > low but jump up on a regular basis to 25%-50% occasionally locking up the GUI > briefly. It was worst when the blockchain was syncing and I was rm the old > COW version even after rm returned. I assume there was quite a bit of > background tasks that were still updating. > I assume for a blockchain, starts small and just grows / appended to. Append writes are the same on overwriting and cow file systems. You might get slightly higher iowait because datacow means datasum which means more metadata to write. But that's it. There's no data to COW if it's just appending to a file. And metadata writes are always COW. You could install bcc-tools and run btrfsslower with the same (exclusive) workload with datacow and nodatacow to see if latency is meaningfully higher with datacow but I don't expect that this is a factor. iowait just means the CPU is idle waiting for IO to complete. It could do other things, even IO, if that IO can be preempted by proper scheduling. So the GUI freezes are probably because there's some other file on /home, along with this 100G file, that needs to be accessed and between the kernel scheduler, the file system, the IO scheduler, and the drive, it's just reluctant to go do that IO. Again, bcc-tools can help here in the form of fileslower, which will show latency spikes regardless of the file system (it's at the VFS layer and thus closer to the application layer which is where the GUI stalls will happen). Any way this workload can be described in sufficient detail that anyone can reproduce the setup, can help make it possible for multiple other people trying to collect the information we'd need to track down what's going on. And that also includes A/B testing, such as the exact same setup but merely running the 100G (presumably it is not actually the exact size but the workload as the sync is happening) Also the more we can take this from the specific case to the general case, including using generic tools like xfs_io instead of a blockchain program, the more attention we can give it because people don't have to learn app specific things. And we can apply the fix to all similar workloads. >> > >> > On a tangent, it took about 30 minutes to delete the old file... My system >> > is a Ryzen 5 3600 w/ 16GB or memory but it is a spinning disk. I use an >> > NVME for the system and the spinning disk for /home. >> >> filefrag 100G.file >> What's the path to the file? > > > $ filefrag /home/richard/.bitmonero/lmdb/data.mdb > /home/richard/.bitmonero/lmdb/data.mdb: 1424 extents found Just today I deleted a 100G Windows 10 raw file with over 6000 extents and it deleted in 3 seconds. So I'm not sure why the delay in your case. So more information is needed, I'm not sure what to use in this case, maybe btrfsslower while also stracing the rm. There is only one ioctl, unlinkat(), and it does need to exit before rm will return to a prompt. But unlinkat() does not imply sync, so it's not necessary for btrfs to write the metadata change unless something else has issued fsync on the enclosing directory, maybe. In that case the command would hang until all the dirty metadata as a result of the delete is updated. And btrfsslower will show this. > However, I let a rebalance run overnight. It shouldn't be necessary to run balance. If you've hit ENOSPC, it's a bug and needs to be reported. And a separate thread can be started on balance if folks want more info on balance, maintenance, ENOSPC things. I don't ever worry about them anymore. Not since ticketed ENOSPC infrastructure landed circa 2016 in kernel ~4.8. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Wed, Mar 24, 2021 at 4:29 AM John Mellor wrote: > > With Fedora being intended as a desktop platform, why are these settings > not the default? > > The highest priority for a desktop system is to keep the user experience > flowing smoothly, not to maximize disk i/o rates. > > Can this be fixed in time for F34? Do we need a bug report? Post to devel@ or desktop@ lists, is my advice. If you convince Workstation edition folks, it'll certainly get more attention on devel@ once it's a feature/change request. It might be appropriate to set dirty_bytes to 500M across the board, desktop and server. And dirty_background to 1/4 that. But all of these are kinda rudimentary guides. What we really want is something that knows what the throughput of the storage is, and is making sure there isn't more than a few seconds of writeback needed at any given time. The default, dirty_ratio 20%, is high by today's memory standards. But upstream will not change it. All kernel knobs are distro responsibility to change from the defaults. I'm the first person to poo poo benchmarks. But if you run all of them, it should be pretty easy to show whether it's generally useful, or generally not useful (i.e. either bad or ambiguous) to change it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: btrfs system slow down with 100GB file
On Tue, Mar 23, 2021 at 8:39 AM Richard Shaw wrote: > > I'm getting significant iowait while writing to a 100GB file. High iowait means the system is under load and not CPU bound but IO bound. It sounds like the drive is writing as fast as it can. What's the workload? Reproduce the GUI stalls and capture all of the following: sudo iostat -x -d -m 5 This is part of sysstat package (you can disable the service and timer units it installs). Probably best to copy/paste into a plaint text file and put it up in a file share service, most anything else is going to wrap it, making it hard to read. A minute of capture while the workload is proceeding is enough. Also capture a few grep -R . /proc/pressure And each of these (workload doesn't need to be running) lsblk -o NAME,FSTYPE,SIZE,FSUSE%,MOUNTPOINT,UUID,MIN-IO,SCHED,DISC-GRAN,MODEL uname -r mount | grep btrfs >I have already made it nocow by copying it to another directory, marking the >director nocow (+C) and using cat to re-create it from >scratch. > > I was under the impression that this should fix the problem. It depends on the workload for this file. Was the 100G file fallocated or created as a sparse file? File format? > > On a tangent, it took about 30 minutes to delete the old file... My system is > a Ryzen 5 3600 w/ 16GB or memory but it is a spinning disk. I use an NVME for > the system and the spinning disk for /home. filefrag 100G.file What's the path to the file? > > Currently I'm getting random GUI freezes due to the iowait problem and my HDD > indicator light basically stays on solid for over an hour now. > Have sysrq+t ready in a shell but don't issue it. Reproduce this problem (the GUI freezes) and then issue the sysrq+t. Depending on how many processes, this could exceed both the kernel message buffer and the journald rate limiter. Either use log-buf-len=8M boot parameter, and then dmesg will have the whole sysrq+t. The other option is temporarily turn off journald rate limiting in journald.conf #RateLimitIntervalSec=30s #RateLimitBurst=1 Add a 0 should work. Restart journald. Issue sysrq+t. Output to a file by 'journalctl -k -o short-monotonic --no-hostname > journal.log' I suggest opening a bug against the kernel, and post the URL here so I can tag it. Attach the iostat output and dmesg/journalctl output as files to that bug, and everything else can just go in the description. Also note any other customizations to /proc or /sys that differ from Fedora defaults. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: boot hang on F33 this morning
The bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1941335 Fix is nearly ready: https://github.com/systemd/systemd/pull/19075 -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: boot hang on F33 this morning
On Sun, Mar 21, 2021 at 4:37 PM Patrick O'Callaghan wrote: > > On Mon, 2021-03-22 at 06:12 +0800, Ed Greshko wrote: > > On 22/03/2021 02:27, Chris Murphy wrote: > > > Some folks are reporting a hang when booting Fedora 33 this morning > > > (or also possibly wake from sleep). We're still digging into it but > > > looks like a work around right now is to boot with parameter: > > > systemd.mask=raid-check.timer > > > > > > You can subsequently 'systemctl disable raid-check.timer'. Folks > > > with > > > md raid setups might want to set a reminder to enable the timer > > > again > > > once the issue is sorted out. > > > > > > > It has been noted on the devel list that it only seems to fail if the > > time zone is set > > to Europe/Dublin. > > > > Also, FWIW, when I used timedatectl to set my time zone to > > Europe/Dublin my > > system hung when doing an "init 6". > > Interesting. Does this happen with Europe/London (which is the exact > same timezone)? Possibly. And perhaps not because of the timezone but when the change is scheduled to happen: 2021-03-28 01:00:00 That's the same datetime as this particular timer: OnCalendar=Sun *-*-* 01:00:00 That translates as Sun 2021-03-28 01:00:00 as of today. Yesterday that was Sun 2021-03-21 01:00:00 which is probably why the issue didn't show up last week. Further, it might be true that other Europe timezones aren't affected because their official time change happens at 02:00:00. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
boot hang on F33 this morning
Some folks are reporting a hang when booting Fedora 33 this morning (or also possibly wake from sleep). We're still digging into it but looks like a work around right now is to boot with parameter: systemd.mask=raid-check.timer You can subsequently 'systemctl disable raid-check.timer'. Folks with md raid setups might want to set a reminder to enable the timer again once the issue is sorted out. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Can't burn a CentOS 7.9 DVD
On Fri, Mar 12, 2021 at 3:49 PM Robert G. (Doc) Savage via users wrote: > > The checksums are perfect. What I am probably looking for is anyone who CAN > burn that iso to a blank DVD-R or DVD+R. If Brasero won't do it, is it > because the iso image is just every-so-slightly too large? There's a bunch of bugs. https://gitlab.gnome.org/GNOME/brasero/-/issues Pretty sure last time I burned a DVD-RW I used wodim from the command line. > > In this particular situation, dd-ing the image to an 8GB thumb drive is not a > player because the target machine is a 2011 vintage Dell PowerEdge that does > not support booting from thumb drives. Hmm, I've got a vintage 2006 Dell laptop that boots from a USB stick. There might have been a BIOS setting to enable it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Formatting second hard disk
On Sun, Feb 28, 2021 at 5:40 AM Paul Smith wrote: > > Dear All, > > I have got a second hard disk on my desktop computer, which I would > like to format. My machine runs Fedora 33, and I would like to ask you > what format to adopt: the new Btrfs is advisable? > > Moreover, could you please direct me to some documentation on how to > format the hard disk? Using GParted? If you're already familiar with gparted, use that. Workstation edition also comes with GNOME Disks, and KDE comes with Partition Manager, which can also do this task. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Hooks for automount
On Sun, Feb 28, 2021 at 10:13 AM Patrick O'Callaghan wrote: > > Is there a way to invoke scripts before auto-mounting and after auto- > unmounting? I want to be able to power an external drive up and down as > needed. If you don't need much sophistication, you can add to fstab with options: noauto,x-systemd.automount Since it's not a startup time dependency, startup won't hang waiting for it if it's not present. If present, it won't be automatically mounted until the mountpoint is accessed. If you want it to automatically unmount when not being used, also include: x-systemd.idle-timeout="10m" https://www.freedesktop.org/software/systemd/man/systemd.mount.html https://www.freedesktop.org/software/systemd/man/systemd.automount.html# -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Btrfs performance
Upstream discussion. https://lore.kernel.org/linux-btrfs/b311ab52-86f7-1e7d-c0b6-15b2d050e...@suse.com/T/#mb16ec76e76ef6b7259c09c9221b1593ed097d617 -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Windows 10 + F33 + SSD + RAID
On Fri, Jan 29, 2021 at 8:45 PM Greg Woods wrote: > Now if I can only figure out why this damn machine won't resume from > hibernation properly. Windows can do it, so I'll bet it's a configuration > issue somewhere. Hibernation is disabled when UEFI Secure Boot is enabled. And even if disabled, which I don't recommend, there are ACPI bugs galore. I don't know how Windows works around them. But it also does seem Microsoft, Apple and hardware vendors are focusing on S0 low power idle (and variations on that theme). Another idea is for the desktop to have some way of saving state but there's no single API that apps can use to do that. https://pagure.io/fedora-workstation/blob/master/f/hibernationstatus.md -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: F32 not booting
First problem is: [8.090730] shure kernel: EXT4-fs (dm-0): unable to read superblock [8.090784] shure mount[471]: mount: /sysroot: can't read superblock on /dev/mapper/fedora_shure-root. Therefore sysroot can't mount. The reason it can't read it is really bad luck: [ 10.481731] shure kernel: sd 0:0:0:0: [sda] tag#14 Add. Sense: Unrecovered read error - auto reallocate failed [ 10.481735] shure kernel: sd 0:0:0:0: [sda] tag#14 CDB: Read(10) 28 00 1d 6a 30 06 00 00 01 00 An unrecoverable read error due to a bad sector, and (drive firmware) ECC can't reconstruct it. [ 10.481737] shure kernel: print_req_error: 9 callbacks suppressed [ 10.481740] shure kernel: blk_update_request: I/O error, dev sda, sector 493498374 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 [ 10.481757] shure kernel: ata1: EH complete And that is the bad sector LBA as reported by the drive itself to the kernel, which coincides with this superblock, apparently. It's vaguely possible an fsck of the root file system will fix this, more likely it fails due to the I/O error. ext4 has more than one super block so it should be able to reconstruct from the others, but perhaps only if you can get rid of the I/O error first. My highly speculative and cautious suggestion is you could write zeros to just this LBA, thereby getting rid of the I/O error (the drive firmware will overwrite the sector and it'll either succeed or fail, if it fails it will remap the LBA to a reserve sector). How do do this depends on whether the drive has 512 byte or 4096 byte physical sectors. If it's 4KiB physical, you have to write 4KiB worth of zeros. But the LBA is based on 512 byte addresses so you have to do an adjustment. If you write just a 512 byte sector, the drive firmware actually *reads* first in order to do read-modify-write of that 4096 byte physical sector, and you'll just get a read error again. Super hilarious that you can do a write and get a read error returned, but that's why. Once that LBA is zeros instead of an IO error, then e2fsck can almost certainly fix it. But I have to couch in in a warning because... I don't know with 100% certainty. Someone else on this list might have a better idea or you could review the strategy on the ext4 list. Now, if this were Btrfs it too would fail to mount if the primary super block is bad or returns an IO error. I'm likewise not certain if any of the repair tools will easily deal with this with just a repair request, due to the I/O error. That might just stop the repair and has to be dealt with first. In the Btrfs case the number of superblocks depends on the size of the file system, 2 if it's at least 256MB, and 3 if it's at least 256G. There is a super recover specific command that can fix this problem, but again probably only once the uncorrectable read error is fixed by overwriting the bad sector. And yes that should sound a bit scary because people make mistakes with dd all the time. I have for sure. The way I remember not to screw up is skip has an i, therefore it applies to if= so when you're doing your sector read test based on 4KiB block size, you use skip, but when you reverse it to write to that sector you have to use seek= otherwise you're gonna overwrite the wrong thing and ... oopsie! And there are more ways to screw it up that this... I've forgotten to set count= before and end up writing megabytes of zeros before I CTRL-C. That was pretty funny. Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: dnf error , rpm database damaged ???
On Wed, Jan 20, 2021 at 4:27 PM Kostas Sfakiotakis wrote: > > Or a brain -damaged SELINUX is screwing up the /var/ partition because > of an imaginary threat > ( a threat that it only exists in it's own twisted mind or whatever > equivalent it has ) and in so > doing it fills up the entire /var partition . That shouldn't happen, or it's a bug. Things need to clean up after themselves, and not just assume they can use all free space. For example systemd out of the box is capped to 4G (it could be a good deal less depending on free space). [ 25.988281] systemd-journald[555]: System Journal (/var/log/journal/6a8c936a3d3048dfb12c1e99bd3a2ad5) is 48.0M, max 4.0G, 3.9G free. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: [F31] Cannot boot from time to time due to systemd-cryptsetup
What kernel version? I suggest updating it. You can use a Fedora 32 kernel on Fedora 31. Even better, upgrade to 33. :D There have been recent changes in the kernel related to random entropy generation during boot. Touching the keyboard helps produce more entropy. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Fri, Jan 15, 2021 at 1:17 AM Tim via users wrote: > > On Thu, 2021-01-14 at 19:56 -0700, Chris Murphy wrote: > > Clearly those lifetime numbers are bogus, as well as sometimes > > reporting a lower lifetime value than the test before it. It's > > getting younger!! My SSD found the fountain of youth! > > Or is it counting down to its death, instead? It's done this since day 1. Otherwise reliable. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Thu, Jan 14, 2021 at 6:58 PM Gordon Messmer wrote: > > On 1/13/21 11:21 PM, Chris Murphy wrote: > > 548 hours is not an old drive. It shouldn't have any write errors. But > > as a drive ages there might be some and they should be handled > > transparently and not affect any other operation. Something is > > definitely wrong that there are write errors followed by read errors > > followed by this IDNF error which suggests the drive is having > > problems read its own data written to sectors reserved for its own > > use > > > Might "its own use" include SMART data? Sreyan indicated that this > laptop was purchased in 2016, and the HGST Travelstar 5K1000 would have > been a new drive at that time. It's odd, then, that the drive reports > only 1671 Power_On_Hours. Yeah. Not sure. Here's a pretty benign firmware bug in a Samsung SSD 840 EVO 250GB. No newer firmware is available for it than what it has. ID# ATTRIBUTE_NAME FLAGSVALUE WORST THRESH FAIL RAW_VALUE 9 Power_On_Hours -O--CK 096 096 000-16513 However... SMART Extended Self-test Log Version: 1 (1 sectors) Num Test_DescriptionStatus Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offlineCompleted without error 00% 3 - # 2 Short offline Completed without error 00% 2 - # 3 Extended offlineCompleted without error 00% 8 - # 4 Extended offlineCompleted without error 00% 2 - # 5 Offline Completed without error 50% 2 - # 6 Extended offlineCompleted without error 00% 2 - # 7 Short offline Completed without error 00% 8 - # 8 Extended offlineCompleted without error 00% 6 - # 9 Extended offlineCompleted without error 00% 5 - #10 Short offline Completed without error 00% 6 - #11 Extended offlineCompleted without error 00% 5 - #12 Short offline Completed without error 00%19 - #13 Extended offlineCompleted without error 00%11 - #14 Short offline Completed without error 00% 0 - Clearly those lifetime numbers are bogus, as well as sometimes reporting a lower lifetime value than the test before it. It's getting younger!! My SSD found the fountain of youth! The #1 entry was just done a week ago. Is it possible the test itself is bogus? I've done ~150 "pull the cord" poweroffs while doing writes. No Btrfs complaints on the next boot, and no errors when scrubbing. So it's probably decently well behaved. But firmware defects can be pernicious so if it suddenly goes kaboom I won't be shocked. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: errors during update.
Hmmm https://bugzilla.redhat.com/show_bug.cgi?id=1908005 https://bugzilla.redhat.com/show_bug.cgi?id=1911038 ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: errors during update.
On Thu, Jan 14, 2021 at 3:26 PM Frank McCormick wrote: > > Saw a couple of errors in the middle of an update today on Fedora 33. > > Couldn't figure out how to copy/paste so I have a small screenshot. > > The system **seems** to work fine after this. Is it something > I should be concerned about? Anything in dmesg or journalctl, at the time of this message? -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Wed, Jan 13, 2021 at 1:09 PM Sreyan Chakravarty wrote: > > On Tue, Jan 12, 2021 at 9:04 AM Chris Murphy wrote: > > > > Because of the --init-csum-tree failure i'm a bit concerned that it's > > possible the --repair step didn't completely repair things. So... just > > keep important things backed up. Top priority. If you told me you hate > > your data then I wouldn't worry about it. > > > > Just letting you know, your suspicions were right. The fsck was not > fully successful. > > I have lost all my previous snapshots, one of them is an empty > directory and one of them, while I can see files in it, can't be used > for restoration as the system won't boot. > > My system is now a ticking time bomb for another crash. At this point I suggest not booting from it. Boot the live USB, mount the file system, and copy the important files from the "home" subvolume (or any of its snapshots) onto some other drive. Using the file manager is sufficient. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Wed, Jan 13, 2021 at 6:02 AM George N. White III wrote: > Your drive has been used for 1671 hours, and > 1491 power-on cycles. Mechanical device wear is often highest at startup, > so this is probably getting close to the design lifetime of a consumer laptop > HDD. I missed that this is a 2.5 inch drive. Probably time for it to have a dirt nap. Or use it for torture testing file systems! -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Wed, Jan 13, 2021 at 2:41 AM Sreyan Chakravarty wrote: > > On Tue, Jan 12, 2021 at 9:16 AM Chris Murphy wrote: > > > > > > -x has more information that might be relevant including firmware > > revision and some additional logs for recent drive reported errors > > which usually are benign. But might be clues. > > > > These two attributes I'm not familiar with > > 187 Reported_Uncorrect 0x0032 100 096 000Old_age > > Always - 4294967301 > > 188 Command_Timeout 0x0032 100 100 000Old_age > > Always - 98785820672 > > > > But the value is well above threshold for both so I'm not worried about it. > > > > > > Here is the output of: > > # smartctl -Ax /dev/sda > > https://pastebin.com/raw/GrgrQrSf > > I have no idea what it means. 9 Power_On_Hours -O--CK 097 097 000-1671 There's a bunch of write errors at 548 hours. And more recently read errors followed by: 10 -- 41 05 40 00 00 60 d9 6e 08 00 00 Error: IDNF at LBA = 0x60d96e08 = 1624862216 548 hours is not an old drive. It shouldn't have any write errors. But as a drive ages there might be some and they should be handled transparently and not affect any other operation. Something is definitely wrong that there are write errors followed by read errors followed by this IDNF error which suggests the drive is having problems read its own data written to sectors reserved for its own use, for example its firmware is often on the drive for some make/models, and so is the bad blocks map. If this sector isn't reading correctly *who knows* what weird things that could trigger. I know who, the person who writes drive firmware. I don't, so I can only speculate that this looks pretty much like a drive that needs to be used for some other purpose or disposed of. And also? Maybe it's getting worse. Maybe it wasn't exhibiting any of these problems yet when you were using NTFS and ext4. Or wasn't ever detected. The first thing that did detect it was LVM thin's metadata checksumming. And the most recent is btrfs. If it were in warranty I'd say make it the manufacturers problem. If it's not, well you could take a chance with it in a Btrfs raid1 with some other cheap drive. Whatever problems they have will be different, and Btrfs will automatically detect and repair them as long as they don't happen at the same time in the same place (astronomically unlikely). So that'd be a good use for it. I personally wouldn't be any other file system on it except maybe ZFS, because with this behavior, you don't want to trust any data to it without full data checksumming. Let alone important data. > This is the problem with SMART tests, they are so esoteric that it is > difficult for a common user to make sense of it. All low level stuff is like this, yeah. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
t ignore all claims of reliability and I don't even care about 5+ year warranties. No 90 day warranties. 1 year if it's dirt cheap. Otherwise 3 years. And never buy an extended warranty. But I gotta say for sysroot, a small inexpensive SSD is pretty awesome. It's a major upgrade. And yeah we probably see more firmware bugs with SSD than HDD, but at least Facebook is using the cheapest consumer drives possible, with Btrfs. And it's fine. Until it's not. So again, all things come back to the backup. Don't worry. Just backup. And if you backup, you won't worry. Or at least, you'll worry less. > > 4) My manufacturer HP, does not make firmware updates for Linux, only > for Windows. So is there a way to update the firmware(if available) > without being on Windows ? Any ideas? Would a Windows PXE help ? I don't think this is the problem. But also, https://www.microsoft.com/en-us/software-download/windows10 Free download. If it only can update the logic board firmware with Windows. It'll even work without a product key, just say you don't have a product key at the part where it asks for one. It'll still work, with some extra limitations that won't matter. > 5) When you say "checksum errors in the month's old report" - which > report are you referring to ? The thin-LVM crash or the smartctl crash > ? LVM thin. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs: What to do with large growing files?
On Tue, Jan 12, 2021 at 6:21 AM Richard Shaw wrote: > > On Mon, Jan 11, 2021 at 11:49 PM Chris Murphy wrote: >> >> On Mon, Jan 11, 2021 at 12:57 PM Richard Shaw wrote: >> >> > I'm playing around with cryptocurrency and the current one downloads the >> > whole blockchain and it's at 65GB and growing. I think the growing part is >> > the problem. That and it's being stored on /home which is a spinning disk >> > not an SSD like the system disk. >> > >> > I'm seeing a lot of IO wait in glances and the system is sluggish even >> > though CPU and memory usage is relatively low. >> >> It'd be useful to know if this is some kind of database, if it's one >> big file or many, if it's doing insertions (typical with database) or >> append. It might just be busy and will settle itself out once it's >> done downloading. And even maybe know what the program is. And I can >> ask folks on #btrfs. > > > I would think a blockchain is roughly equivalent to a database but I have no > idea if the blocks were downloaded in order (appended) or randomly > (inserted). The single file (data.mdb) ended up being 96GB in total. It seems > to have settled down as I expected but it took about 2 days to download so it > was a bit frustrating using my computer at that time. Anything that tried to > access /home (my spinning btrfs drive) would basically stall including > youtube videos that ran out of cache, browser tabs, etc. Sounds like that's MariaDB, mysql. Can you post cat /proc/mounts for this file system? > > The program/cryptocurrency is Monero and was downloaded via Flatpack (I > think, could be an Appimage). There are two things to try if anyone wants to reproduce. Change scheduler from bfq to mq-deadline. Swap out the containing directory for a subvolume. Of course, don't do both at the same time. A/B testing ext4 vs Btrfs is also useful. I'll put it on my todo, but it might be a little while before I get to it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 11, 2021 at 12:24 AM Sreyan Chakravarty wrote: > > On Sun, Jan 10, 2021 at 3:10 AM Chris Murphy wrote: > > Sreyan, is the drive that this file system is on the same drive as in > > this reported thread? > > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/thread/DJWIOXUOSAAHAAXSUTKREV452QDCXV3D/#6UDYNNWOC23NJKSLMIABSAU4QFYANSRB > > > > > > Yes it is the same drive. > > Does that mean something is wrong with the drive ? Short version: Josef (Btrfs dev) and I agree there's probably something wrong with the drive. The advice is to replace it, just because trying to investigate further isn't worth your time, and also necessarily puts your data at risk. Longer version: LVM thinp uses device-mapper as its backend to do all the work, and we see checksum errors in the months old report. Where LVM thick has a simpler on-disk format, so it's not as likely to discover such a problem. And LUKS/dm-crypt is also using device-mapper as its backend to do all work. So the two problems have two things in common: the drive, and device-mapper. It's more probable there's an issue with the drive, just because if there was a problem with device-mapper, we'd have dozens of reports of it at the rate you're seeing this problem once every couple of months (if that trend holds). Is it possible LVM+ext4 on this same drive is more reliable? I think that's specious. The problem can be masked due to much less stringent integrity guarantees, i.e. there are no data checksums. Since the data is overwhelmingly the largest target for corruption, just being a much bigger volume compared to file system metadata. All things being equal, there's a greater chance the problem affects data. On the other hand, if it adversely impacts metadata, it could be true that e2fsck has a better chance of fixing the damage than btrfsck right now. Of course no fsck fixes your data. So if you keep using the drive, you're in a catch-22. Btrfs is more sensitive because everything is checksummed, so the good news is you'll be informed about it fairly quickly, the bad news is that it's harder to repair in this case. If you revert to LVM+ext4 the automatic fsck at startup might fix up these problems as they occur, but it's possible undetected data corruption shows up later or replicates into your backups. Regardless of what you decide to do, definitely keep frequent backups of anything important. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs: What to do with large growing files?
On Mon, Jan 11, 2021 at 12:57 PM Richard Shaw wrote: > > So from what I understand once a file is >0 that you can't set nodatacow. Correct. > I'm playing around with cryptocurrency and the current one downloads the > whole blockchain and it's at 65GB and growing. I think the growing part is > the problem. That and it's being stored on /home which is a spinning disk not > an SSD like the system disk. > > I'm seeing a lot of IO wait in glances and the system is sluggish even though > CPU and memory usage is relatively low. > > Ideas? It'd be useful to know if this is some kind of database, if it's one big file or many, if it's doing insertions (typical with database) or append. It might just be busy and will settle itself out once it's done downloading. And even maybe know what the program is. And I can ask folks on #btrfs. Btrfs does quite a lot of delayed allocation so you might see higher IO wait than you're used to, but as a side effect it turns random writes into sequential writes which write and read faster and have less fragmentation. So it's overall better. But the system being sluggish makes me wonder if something else is competing for IO with this task and what that process's write pattern is too. It could be lock contention, the easy way to alleviate that is give one of the tasks its own subvolume. I'm assuming this problem is all over by now but if you can reproduce it the first thing to look at is sysrq+t https://fedoraproject.org/wiki/QA/Sysrq that likely fills the dmesg buffer, but 'journalctl -k --no-hostname -o short-monotonic > dmesg.txt` will do it and then you can upload that file somewhere and I'll bug someone about it :P if no one on here knows how to read sysrq+t (i'm not great at it unless it's really obvious). Ideally have the sysrq+t ready to just hit enter. When the sluggishness happens, hit return. If that's all normal we might have to get into bcc-tools or something. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 11, 2021 at 12:02 PM Sreyan Chakravarty wrote: > > On Tue, Jan 12, 2021 at 12:02 AM Matthew Miller > wrote: > > > > > > Well, another way of saying that is "I wasn't having problems but now I have > > them", which isn't super-surprising with a hard drive. What does smartctl > > tell you about the drive's self-tests? > > > > Well I provided this info at the beginning of this thread. > > I am providing here again: > > SmartCTL output (smartctl -A) : > https://pastebin.com/raw/B6AdLZXt -x has more information that might be relevant including firmware revision and some additional logs for recent drive reported errors which usually are benign. But might be clues. These two attributes I'm not familiar with 187 Reported_Uncorrect 0x0032 100 096 000Old_age Always - 4294967301 188 Command_Timeout 0x0032 100 100 000Old_age Always - 98785820672 But the value is well above threshold for both so I'm not worried about it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 11, 2021 at 11:55 AM Sreyan Chakravarty wrote: > > Actually it is me who should be thanking you for not giving up. > > For me giving up was not an option since that would involve me losing > all my data. No problem! We should talk about a backup plan in another thread :D In my own case, I never need them, but I have 1/2 dozen completely independent backups, because I hate my data and I'm constantly trying to destroy it. (This is turning into a really bad joke but hopefully it's weirdly amusing. If you like your data you *definitely* should be backing up! Doesn't matter what the file system is, or if it's on raid, or "safe" in the cloud, etc.) > > Full output of repair command over here: > > https://drive.google.com/file/d/17EQ5TYJE5a4iwRGTXXj9Hh38btxFBLAJ/view?usp=sharing > > 2) btrfs check --init-csum-tree /dev/mapper/dm_crypt > >This is where the minor bad news happens. > >This command exited with a "Segmentation Fault(Core Dumped)" > >Full output here: > > https://drive.google.com/file/d/1IYnLcw-vkZ2UeH0rOZje4kbhzB_tYaKy/view?usp=sharing Forwarded this to Btrfs dev, so we'll just have to be patient. I strongly suggest some kind of backup of the data, and confirm it's good. Since you were stressed about maybe losing it? Even two copies! Not anything because it's Btrfs but because it's important to you. Bad things happen all the time that have nothing to do with the file system. Because of the --init-csum-tree failure i'm a bit concerned that it's possible the --repair step didn't completely repair things. So... just keep important things backed up. Top priority. If you told me you hate your data then I wouldn't worry about it. > Let me know the next steps ie. Do you need BTRFS logs to diagnose the > root cause of the problem ? Not sure. I've asked. If there's a request to, we'll run it through gdb and see what happens. Can you tell me what you get for: free -m btrfs fi us / It'll give an idea what the memory requirement is for the init-csum-tree. It really should not be a lot, it's less complicated than the repair by a lot. Glad that worked though, phew! -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 11, 2021 at 11:30 AM Sreyan Chakravarty wrote: > > On Mon, Jan 11, 2021 at 10:55 PM Matthew Miller > wrote: > > > > On Mon, Jan 11, 2021 at 12:54:07PM +0530, Sreyan Chakravarty wrote: > > > > Sreyan, is the drive that this file system is on the same drive as in > > > > this reported thread? > > > > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/thread/DJWIOXUOSAAHAAXSUTKREV452QDCXV3D/#6UDYNNWOC23NJKSLMIABSAU4QFYANSRB > > > > > > > > > > Yes it is the same drive. > > > Does that mean something is wrong with the drive ? > > > > It seems very likely -- either with the drive, or a cable, or the > > controller, or some other aspect of your system. You're getting serious > > errors regardless of what you're actually putting on the drive. > > > > I beg to differ. > > I never faced problems like this when I was on "thick/older" LVM with EXT4. > > It is only when I switched to thin-pools and then to BTRFS that I > started getting these problems. This is an interesting data point. I will incorporate it. In the meantime, we really need to stay focused on facts and not jump to conclusions. We can evaluate whether you stick with Btrfs, pros/cons. Bugs like this are really slippery. The #1 reason they don't get caught and set free, is tester exhaustion. This is a reasonable outcome. People have better things to do than debug stuff. But it's sometimes a necessary effort because we really are at significant confidence in Btrfs stability equivalent to other file systems, even though we're aware of its fragility in certain bad failure cases like this one. The only way it gets better is to persevere. Thanks, -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 11, 2021, 12:22 AM Sreyan Chakravarty wrote: > On Sun, Jan 10, 2021 at 2:11 AM Matthew Miller > wrote: > > > > On Sun, Jan 10, 2021 at 02:01:56AM +0530, Sreyan Chakravarty wrote: > > > If I don't have ECC does that mean I shouldn't use BTRFS ? > > > > The chance of data corruption due to memory errors is rare, but it does > > happen. If it happened and you were using ext4, the result wouldn't be > that > > everything is fine, it's that you'd have corrupted data -- hopefully not > > important, but... you won't know. > > > > Now, I don't think btrfs's current state of "now your system won't boot > and > > you need to be an expert to figure out what's going on" is ideal either, > but > > it's a rare situation and recovery tools will improve. > > > > With the amount of time that Chris and everyone else put into this, I > just hope this bug is fixed for good. > No bug has been identified. Have you gone through the numbered list and tried to repair? It is important you install specifically the btrfs-progs in that email. It has an enhancement for this particular kind of damage. https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/message/4G52PRZ4ETFSXCSOJYFBGP6H64FRSRZT/ -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Compression on Btrfs
On Sun, Jan 10, 2021 at 4:39 PM Matthew Miller wrote: > > On Sun, Jan 10, 2021 at 11:09:31PM +, Patrick O'Callaghan wrote: > > > It's not possible to set this attribute once a file has data in it, it's > > > not retroactive. You'll need to duplicate the file, in that same > > > directory. > > > Because the directory has the attribute now, the duplicate will inherit > > > the > > > attribute. > > > > That's somewhat painful as the file is over 900GB and will need to be > > copied to another drive and then back again, but thanks anyway. > > Can you use `cp --reflink=always`? No, it will fail. A reflink copy creates shared extents. It can't change them between datacow and nodatacow. Whatever they are in the source, datacow or nodatacow, they must be in the copy. If the +C attribute is set on the enclosing directory, and the file being copied is datacow - there's a mismatch. A cp operation requires a conventional copy in this case, to create the nodatacow extents. But with --reflink=always, the conversion is disallowed, so the cp will fail. This applies to both directions, so it's datacow<-->nodatacow, is a conventional copy operation. It also means as long as you aren't switching between them, you automatically get a reflink copy because the default on Fedora 33 is cp --reflink=auto, i.e. it'll try to reflink copy if it can, and if it can't it'll fall back to a conventional copy. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Compression on Btrfs
On Sun, Jan 10, 2021 at 4:09 PM Patrick O'Callaghan wrote: > > > If you're using libvirt, creating a new pool on Btrfs will automatically > > set +C attribute. > > It wasn't Btrfs at the time the image was created. Ahh gotcha. A proper conversion guide should point out this gotcha. The work around might be chattr +C before the conversion, but I haven't tested it. > > > It's not possible to set this attribute once a file has data in it, it's > > not retroactive. You'll need to duplicate the file, in that same directory. > > Because the directory has the attribute now, the duplicate will inherit the > > attribute. > > That's somewhat painful as the file is over 900GB and will need to be > copied to another drive and then back again, but thanks anyway. As long as the cache mode is writeback or none, it'll be OK. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Compression on Btrfs
On Sun, Jan 10, 2021, 3:52 AM Patrick O'Callaghan wrote: On Sat, 2021-01-09 at 18:01 -0700, Chris Murphy wrote: > > > Does this mean that my VM image subvolume is being included in the > > > compression? If that's the case I'll cease and desist. > > > > Did you set 'chattr +C' for nodatacow on the enclosing > > directory/subvolume for the VM images? Nodatacow implies no > > compression and no checksums. > > Probably but I don't remember. I've now set it just in case, but lsattr > doesn't show it: > > $ sudo chattr +C /home/Windows > $ lsattr -l /home/Windows > /home/Windows/wimage --- > > > Is this a bug or a feature? > If you're using libvirt, creating a new pool on Btrfs will automatically set +C attribute. It's not possible to set this attribute once a file has data in it, it's not retroactive. You'll need to duplicate the file, in that same directory. Because the directory has the attribute now, the duplicate will inherit the attribute. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Limit size of a btrfs subvolume?
On Sat, Jan 9, 2021 at 7:25 PM Richard Shaw wrote: > > On Sat, Jan 9, 2021 at 6:25 PM Chris Murphy wrote: >> >> On Sat, Jan 9, 2021 at 6:11 AM Richard Shaw wrote: >> > >> > On Fri, Jan 8, 2021 at 10:12 AM Qiyu Yan wrote: >> >> >> >> I think https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-qgroup and >> >> https://btrfs.wiki.kernel.org/index.php/Quota_support are what you are >> >> looking for. >> > >> >> While the quota tracks space usage for a subvolume, once it's reached, >> writes fail. I'm not sure what interface exists for applications to >> learn whether a quota is set and what value. I've noticed that ext4, >> xfs, btrfs each have their own quota implementation and they're all >> kinda confusing in different ways. > > > Any way to achieve what I'm trying to do then? Maybe I should ask on the > MythTV mailing list. Yeah I'm not sure how its threshold works, how configurable it is. I think it's functionally the same thing as a separate partition if its threshold triggers once a used amount value has been reached, i.e. in its own directory. Or alternatively % used or % remaining. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Compression on Btrfs
On Sat, Jan 9, 2021 at 4:15 AM Patrick O'Callaghan wrote: > > On Fri, 2021-01-08 at 00:03 -0700, Chris Murphy wrote: > > On Thu, Jan 7, 2021 at 5:52 AM Patrick O'Callaghan > > wrote: > > > > > > I did the following: > > > > > > # btrfs filesystem defragment -czstd -r -v /home > > > > > > and changed the fstab entry: > > > > > > # grep /home /etc/fstab > > > UUID=8e1f7af4-c0bf-434e-b1c4-a9af2c810d56 /home btrfs > > > subvol=home,discard=async,compress-force=zstd 0 0 > > > > > > (this is an SSD, hence the discard-async) > > > > > > I then rebooted, but find: > > > > > > # btrfs prop get -t i /home compression > > > # > > > (i.e. no output) > > > > The 'btrfs property' method of setting compression per file, > > directory, or subvolume, sets an xattr. The mount option method does > > not. > > > > Also, the mount option is file system wide, it's not per subvolume. It > > just seems like it could be this way due to fstab and the subvol mount > > option (which is really just a bind mount behind the scenes). > > Does this mean that my VM image subvolume is being included in the > compression? If that's the case I'll cease and desist. Did you set 'chattr +C' for nodatacow on the enclosing directory/subvolume for the VM images? Nodatacow implies no compression and no checksums. There's some advantages to compressing VM images, so it's a valid workflow. But there's enough tradeoffs that it's just way simpler to recommend nodatacow because it's the most compatible option. If you want to experiment with VM images with cow and compression enabled, the main two things: long standing O_DIRECT interaction bug/RFE, just avoid using any cache options that use O_DIRECT, e.g. use writeback or unsafe. [1] Learning a bit about fragmentation mitigation, which can be scheduled with a timer. [1] Unsafe means the guest file system is at significant risk of corruption if the host has a crash or power failure. Of course, I'm using Btrfs in the guest and on the host, and I'm always force quitting the guest with impunity. That's OK. But yanking the powercord on the host while the guest is writing is just asking for big trouble. Naturally, I do that often. And I don't recommend it unless, like me, you hate your data. But if you take precautions to avoid the host falling over, you can use unsafe. I'm pretty sure Fedora infrastructure is now using unsafe on some portion of the compose process to significantly boost performance. But also they have reliable hosts. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Limit size of a btrfs subvolume?
On Sat, Jan 9, 2021 at 6:11 AM Richard Shaw wrote: > > On Fri, Jan 8, 2021 at 10:12 AM Qiyu Yan wrote: >> >> I think https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-qgroup and >> https://btrfs.wiki.kernel.org/index.php/Quota_support are what you are >> looking for. > > > Thanks. That should do it! > While the quota tracks space usage for a subvolume, once it's reached, writes fail. I'm not sure what interface exists for applications to learn whether a quota is set and what value. I've noticed that ext4, xfs, btrfs each have their own quota implementation and they're all kinda confusing in different ways. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Sat, Jan 9, 2021 at 1:32 PM Sreyan Chakravarty wrote: > > If I don't have ECC does that mean I shouldn't use BTRFS ? No. And also, this problem is not related to a memory error. In this case multiple writes from one commit are missing. They just didn't make it to stable media. There isn't enough information to know why. We might get lucky and find some clues in the systemd journal, before the file system went read only. What I can tell you is Btrfs doesn't have any known or suspected defects related to this kind of problem. It writes a superblock only once it has reason to believe the prior metadata writes were on stable media. There is also a write time tree checker designed to complain and go read only the moment there's an inconsistency when writing. This file system has a longer period of time of writes following the missing writes, which is also why the 'usebackup' root option didn't work. And the dropped writes also affect both copies of metadata. It's very unusual. Sreyan, is the drive that this file system is on the same drive as in this reported thread? https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org/thread/DJWIOXUOSAAHAAXSUTKREV452QDCXV3D/#6UDYNNWOC23NJKSLMIABSAU4QFYANSRB -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Sat, Jan 9, 2021, 10:11 AM Sreyan Chakravarty wrote: > On Sat, Jan 9, 2021 at 11:55 AM Chris Murphy > wrote: > > > > 1. If you don't have a backup of important data, you should first use > > 'btrfs restore' as I've described in previous emails. The fsck should > > fix the problem, but there is a chance it makes things worse. No fsck > > is guaranteed safe, because it's making changes to the file system. > > > > Let me know if it makes more sense to use ddrescue instead of `btrfs > restore`. > It's up to you. Both methods make a copy. Thats whats important. ddrescue will take longer and needs more space than just pulling out files with btrfs restore. But if you're already familiar with ddrescue, that's an advantage. -- Chris Murphy > > ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
1. If you don't have a backup of important data, you should first use 'btrfs restore' as I've described in previous emails. The fsck should fix the problem, but there is a chance it makes things worse. No fsck is guaranteed safe, because it's making changes to the file system. 2. Install the following updated btrfs-progs branch that has a tested fix for this problem. And by the way your test image has helped improve the Btrfs fsck. Pretty much all fsck improvements for all file systems depend on getting good user reports and information. So thanks for not giving up. sudo dnf install https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robustimg/fedora-33-x86_64/01873195-btrfs-progs/btrfs-progs-5.9+git20210108.3783313-0.fc33.1.x86_64.rpm 3. If it's a laptop, make sure it has power connected, and that you can let it run until it finishes. Because it's going to take a long time. I don't have a time estimate for what that means exactly, it depends mostly on memory. 4. This is the command to run, good idea to save all the terminal output because if it doesn't work we need to see what went wrong. sudo btrfs check --repair /dev/sdXY Near the end I'm told there will be complaints about missing csum errors. Ignore that for now. And try to mount normally. You should be able to use it normally at this point. And there should be no errors in dmesg. If there are let me know before the next step. 5. There are some checksums that didn't make it to the drive when the original problem happened. That has a separate repair: sudo btrfs check --init-csum-tree /dev/sdXY That's it, everything should be OK. If something goes wrong, best to not try again, just ask here or on IRC. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Fri, Jan 8, 2021 at 4:42 AM Sreyan Chakravarty wrote: > > Is there an fsck fix available for my problem ? Yes it just got done. > > Can my filesystem be fixed? Or do I have to erase everything and do a > reinstall ? Strong likelihood that it can be fixed. But if the user data is important we should do a btrfs restore first. That's a safe read-only operation, does not change the file system like fsck which always has risk. > Also is the fix for my particular problem being worked on by the devs > at the particular moment ? Yes. > Are we living that close to the edge ? It's actually an unusual failure, so it is an edge case. I'm happy to discuss this part later. Priority is to get your data backed up, and fix the file system. > > You could add -x -m to this if you want. > > > > Wait didn't you just say -x and -m will give errors. I don't > understand what you mean by "-m and -x require files to be restored > first or you get additional errors." -m and -x only get applied after files are restored. The files are extracted and saved. And then the metadata/xattr is applied to the files. -D prevents the files from being restored. It's a dry run. Since there's no files, -m -x results in errors. > If you want I am available on US time but only if you let me know in advance. I saw your message today on IRC and replied about 15 seconds later but didn't get a reply. Anyway, next email I'll write up a procedure. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Compression on Btrfs
On Thu, Jan 7, 2021 at 7:52 AM Patrick O'Callaghan wrote: > > On Thu, 2021-01-07 at 21:33 +0800, Qiyu Yan wrote: > > > and the space usage doesn't seem to have changed. > > Try using compsize to get space usage? > > # compsize /home > Processed 373221 files, 936531 regular extents (1058479 refs), 155981 > inline. > Type Perc Disk Usage Uncompressed Referenced > TOTAL 98% 1.1T 1.1T 1.0T > none 100% 1.1T 1.1T1013G > zstd46% 17G 38G 40G And are there any new subvolumes you've created below /home? Those are not subject to -r, the defragment will stop at subvolume boundaries. Any VM images? If they're nodatacow, they won't compress. And quite a lot of media files will not compress because they're already compressed. Based on the above numbers I don't think this is as likely the issue but mention it anyway: Are there any snapshots? 'btrfs fi defragment' is not snapshot aware and will split up shared extents (makes them exclusive again, increasing space consumption). Lower priority, more experimentation, purely anecdotal: Using zstd:1 might result in faster writes to SSD; where zstd:3 (same as zstd) might result in faster overall writes to HDD. For Fedora 34 the proposal is 'compress=zstd:1' and that's a bit conservative but also is low hanging fruit with the biggest gain for the effort. Folks who don't care about performance or want to use this for archives can experiment with even higher values. I tend to use zstd:7 on backups. It does get quite slow eventually, around 11 or higher. But zstd is fairly invariant to compression level on reads, as in I doubt anyone can tell a difference on read between a file that was compressed with 1 vs 9, and maybe not notice 15 unless they were paying attention or timing it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Compression on Btrfs
On Thu, Jan 7, 2021 at 5:52 AM Patrick O'Callaghan wrote: > > I did the following: > > # btrfs filesystem defragment -czstd -r -v /home > > and changed the fstab entry: > > # grep /home /etc/fstab > UUID=8e1f7af4-c0bf-434e-b1c4-a9af2c810d56 /home btrfs > subvol=home,discard=async,compress-force=zstd 0 0 > > (this is an SSD, hence the discard-async) > > I then rebooted, but find: > > # btrfs prop get -t i /home compression > # > (i.e. no output) The 'btrfs property' method of setting compression per file, directory, or subvolume, sets an xattr. The mount option method does not. Also, the mount option is file system wide, it's not per subvolume. It just seems like it could be this way due to fstab and the subvol mount option (which is really just a bind mount behind the scenes). > and the space usage doesn't seem to have changed. 'du' doesn't know about Btrfs compression, it's too transparent and so it sees everything as uncompressed. 'df' doesn't directly know about Btrfs compression, but it does report on free space which is affected by the fact that less space is being used due to compression. So it reports correctly. And the 'btrfs' command (btrfs-progs) doesn't yet have a compression specific reporting option. So yeah, compsize is what you're after. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Thu, Jan 7, 2021 at 1:40 AM Sreyan Chakravarty wrote: > > Can you tell me how you are planning to use the metadata image ? > > Are you going to mount it with losetup and then run btrfs-check on it ? > > I am asking because I can perform the checks from my side. Sorry I missed you on IRC. The Btrfs dev has been doing this with a devel branch, it's not in the released version of btrfs-progs yet. And it needed further enhancement for this case, because a whole bunch of writes in a commit just didn't get to stable media. This fsck enhancement is still being tested, but should be ready soon, and when it is I'll get it built in copr. In the meantime, about your btrfs restore command: # btrfs restore -sxmS -D /dev/mapper/dm_crypt /run/media/liveuser/Backup\ Plus/restore/ Try adding -i and remove -D. The -D means files aren't restored. And -m and -x require files to be restored first or you get additional errors. However, note that the -s option restores the contents of all snapshots, and puts them into directories as independent (not deduped) files in the restore destination. So if you have a file in 50 snapshots, that file gets replicated 50 times. So while you get all your files out of the snapshots, it's not really putting Humpty Dumpty back together again in terms of the original organization. You might prefer just getting all the files out of a particular snapshot. That's the -r option. But *sigh* it only takes subvolids. Since you can't mount, you can't use 'btrfs sub list' to get a listing of names to ids. There is another way to do it: sudo btrfs inspect-internal dump-tree -t 1 /dev/sdXY | grep -A1 'ROOT_REF' That'll produce something like this, maybe many lines if you have many snapshots. item 4 key (FS_TREE ROOT_REF 268) itemoff 14921 itemsize 28 root ref key dirid 256 sequence 5 name home item 5 key (FS_TREE ROOT_REF 525) itemoff 14899 itemsize 22 root ref key dirid 256 sequence 33 name root Above, the first item is the subvolume "home" and its ID is 268. The second is the subvolume "root" and its ID is 525. Snapshots appear in this same list with the same scheme. If I want to pull out just the files in "home" I do this: # btrfs restore -v -i -r 268 /dev/sdXY /path/to/restore/to You could add -x -m to this if you want. There is more here, in case I'm not around again, but hopefully we'll figure out an overlap time on IRC. https://btrfs.wiki.kernel.org/index.php/Restore -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs RAID 5?
On Tue, Jan 5, 2021 at 11:45 AM John Mellor wrote: > > Is there a Fedora doc coming out with all this btrfs minutia? Its great > stuff, but without the doc its also a college-level course in how to do > things the right way. In the heat of the moment, this missing doc would > be ultra-handy. There's quite a lot of information in 'man 5 btrfs' and in each subcommand's man page, e.g. 'man btrfs subvolume' where you can do 'man btrfs' to get a listing if the subcommand man pages. There's also quite a bit of information on btrfs.wiki.kernel.org with some degrees of outdatedness among the material. I haven't looked yet but I imagine there must be a chunk of (open)SUSE docs. I'm reluctant to just duplicate that work and put Fedora in a position where a lot of material also starts going stale, as we already have that issue with docs generally and not just in Fedora. If there are specific topics that need single source documentation, including how to, with examples, possibly also with references - maybe that'd be more useful and maintainable. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs RAID 5?
On Tue, Jan 5, 2021 at 11:49 AM Richard Shaw wrote: > > On Tue, Jan 5, 2021 at 12:31 PM Chris Murphy wrote: >> >> On Tue, Jan 5, 2021 at 6:24 AM Richard Shaw wrote: >> > Ok, so not so bad. The main reason I'm considering raid5 is that I have >> > one 4TB drive right now, if I add 2 more with raid one, I'm only going to >> > get 2TB. I know it's kind of a duh, you're mirroring and right now I have >> > no redundancy, but this is for home use and $$$/TB is important and I >> > can't fit any more drives in this case :) >> >> Adding 2 drives at a time to Btrfs raid1 is trivial, you just add each: >> btrfs dev add /dev/3 /mnt >> btrfs dev add /dev/4 /mnt >> >> Implies both mkfs and resize. You don't need to balance it. > > > Ok, but my current drive is ext4 formatted. I could convert it in place but I > just did that with my home drive on my desktop and it does take a while to > create and remove the ext2_saved image. Wouldn't it be better to create a 2 > drive raid1 array, copy the files over, update fstab, and then add the > original drive to the raid1 array? Yes that. I just meant when you add the original, you don't mkfs.btrfs on it. You just add it, mkfs is implied when adding. Convert is stable and does get attention if bugs are found and seems to be reliable. But there could still be edge cases, mainly because there's not tons of aged ext4 file systems being converted. > > As far as balancing, I wasn't sure it was helpful but was thinking of just > spreading the data around :) For the odd drive case, and where you're adding one drive to Btrfs raid1, best to do a balance. It won't equally spread the data around, it favors the drive(s) with the most free space. Once they all have the same amount of free space, the allocator takes turns among them to try to maintain equal free space. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs RAID 5?
d:100 (10.0 seconds) > Write:100 (10.0 seconds) yeah if that's the default it's fine. The kernel's command timer is 30s, so the drive will give up on a read/write error before the kernel will think it's MIA. > > The drive is a Seagate Terascale which is supposedly designed for cloud > computing / datacenters. > > So raid1 or raid5... You can change it later either way with -dconvert. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: systemd-resolved floods the logs
On Tue, Jan 5, 2021 at 10:32 AM Chris Murphy wrote: > > On Tue, Jan 5, 2021 at 9:10 AM Jerome Lille wrote: > > > > Hi > > > > I've just updated a desktop from Fedora 32 to 33 and after that the > > logs are flooded with the following message > > > > systemd-resolved[]: Using degraded feature set TCP instead of UDP for > > DNS server 127.0.0.1. > > systemd-resolved[]: Using degraded feature set UDP instead of TCP for > > DNS server 127.0.0.1. > > > > This machine uses a VPN service that is always on. The file > > /etc/resolver.conf has just one line with the nameserver from the VPN > > provider. There's no problem in name resolution. Just the constant > > flooding of the logs. > > > > What can be done? > > Maybe this bug: https://github.com/systemd/systemd/issues/13432 -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: systemd-resolved floods the logs
On Tue, Jan 5, 2021 at 9:10 AM Jerome Lille wrote: > > Hi > > I've just updated a desktop from Fedora 32 to 33 and after that the > logs are flooded with the following message > > systemd-resolved[]: Using degraded feature set TCP instead of UDP for > DNS server 127.0.0.1. > systemd-resolved[]: Using degraded feature set UDP instead of TCP for > DNS server 127.0.0.1. > > This machine uses a VPN service that is always on. The file > /etc/resolver.conf has just one line with the nameserver from the VPN > provider. There's no problem in name resolution. Just the constant > flooding of the logs. > > What can be done? Maybe this bug: -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs RAID 5?
On Sun, Jan 3, 2021 at 3:09 PM Richard Shaw wrote: > > On Sun, Jan 3, 2021 at 3:34 PM Chris Murphy wrote: >> >> >> >> On Sun, Jan 3, 2021, 6:26 AM Richard Shaw wrote: >>> >>> Chris, >>> >>> Thanks for the detailed explanation. Too much to quote for my follow up >>> question :) >>> >>> So for 3 drives and my desire to have more capacity and redundancy (for >>> drive failure) would I be better off with RAID1 or RAID5 w/ btrfs? >> >> >> Depends on what you mean by better. :D In terms of survivability of your >> data? You're better off with more independent copies/backups than you are >> with any kind of raid. Raid improves availability, i.e. instead of 0% >> working it has a degraded mode where it's mostly working, but will require >> specific actions to make it healthy again, before it also becomes 0% working. > > > Yeah, the RAID1 seems a lot easier with the caveat that the free space > reporting is bogus, which may be important for a media drive. :) The RAID5 > caveats don't scare me too much. The odd number device raid1 free space reporting issue is 'df' specific. If you try it out and fallocate a bunch of files 10G at a time (no writes for fallocate, it's fast) you can see the goofy thing that happens in the bug report. It isn't ever 100% wrong, but it is confusing. The btrfs specific commands tell the truth always: btrf fi df is short and sweet; btrfs fi us is very information dense. metadata raid1 + data raid5 is fine if you're ok with caveats email from Zygo; people have run it for years and it's saved their butt in other ways. The Btrfs write hole thing is not as bad as other write holes, because while btrfs raid5/6 does not checksum parity it will still spit out csum error upon reconstructing from bad parity. So while it's a form of partial data loss, which would be the case with any raid5 in the same situation, you at least get warned about it and it doesn't propagate into your backups or user space, etc. Toss up on xxhash64, it's as fast or faster than crc32c to compute, collision resistance, but csum algo is a mkfs time only option - only reason why I mention it. I can write more upon request. If these are consumer drives: (a) timeout mismatch (b) disable each drive's write cache. This is not btrfs specific advice, applies to mdadm and LVM raid as well. Maybe someone has udev rules for this somewhere and if not we ought to get them into Fedora somehow. hdparm -W is the command, -w is dangerous (!). It is ok to use the write cache if it's determined that the drive firmware honors flush/fua, which they usually do, but the penalty is so bad if they don't and you get a crash that it's maybe not worth taking the risk. Btrfs raid1 metadata helps here, as will using different drive make/models, because if one drive does the wrong thing, btrfs self heals from another drive even passively - but for parity raid you really ought to scrub following a crash. Or hey, just avoid crashes :) https://raid.wiki.kernel.org/index.php/Timeout_Mismatch >> Striped parity raids perform well for sequential workloads. They're not good >> for metadata heavy workloads. Btrfs alters this calculation because metadata >> (the fs itself) can have a different profile than data. i.e. the >> recommendation is to use raid1 metadata when using raid5 data; and raid1c3 >> metadata when using raid6 data. > > > For a media drive (smallest file is a few MB ogg to 30GB movie) I don't think > things will be very metadata heavy. Yeah it's fine. The strip size (mdadm calls it chunk, which is an entirely different thing on btrfs - of course) is 64KiB so even smaller files will not have bad performance. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: multiboot
On Mon, Jan 4, 2021 at 2:19 PM Patrick Dupre wrote: > > Hello, > > I can compare 2 machines (fedora 32). > One of them (laptop) never boot improperly, > grub displays the boot options and the elapsed time, and finally it boots > automatically on the right system. > The other one, occasionally does not boot automatically at all. > The elapsed time is not displayed, and it does not boot automatically. > The only option is to force the boot manually. > > Would you see a reason for this behavior? Different upgrade histories? Compare these files on both systems: /etc/default/grub grub2-editenv list 2nd one is for grubenv file but this is the proper way to show it My guess is the computer that does nothing is missing GRUB_TIMEOUT=5 in /etc/default/grub so you'll change that and then do the grub2-mkconfig -o /path/to/grub.cfg hocus pocus. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 4, 2021 at 3:00 PM Chris Murphy wrote: > > Ignore the above. New plan. > > Can you clone and build this? And then do 'btrfs-image -c9 -t4 -w > /dev/sdXY /mnt/pathtoimagefile' New new plan, ngompa built it for us in Fedora copr. sudo dnf install https://download.copr.fedorainfracloud.org/results/ngompa/btrfsprogs-robustimg/fedora-33-x86_64/01858610-btrfs-progs/btrfs-progs-5.9+git20210104.5fa05dd-0.fc33.1.x86_64.rpm That will replace your btrfs-progs with josef's special repo with the better -w. Later, you can revert back to the original: sudo dnf install https://kojipkgs.fedoraproject.org//packages/btrfs-progs/5.9/1.fc33/x86_64/btrfs-progs-5.9-1.fc33.x86_64.rpm But there's no urgency to revert, it's the same thing just with this btrfs-image enhancement. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 4, 2021 at 1:56 PM Chris Murphy wrote: > > On Mon, Jan 4, 2021 at 11:32 AM Sreyan Chakravarty wrote: > > What do I do now ? > > Rats. Can you retry by adding -w option? In the meantime I'll report > back to upstream and see what they recommend next. Ignore the above. New plan. Can you clone and build this? And then do 'btrfs-image -c9 -t4 -w /dev/sdXY /mnt/pathtoimagefile' https://github.com/josefbacik/btrfs-progs/tree/more-robust-image The Fedora btrfs-progs spec file says you'll need: BuildRequires: gcc, autoconf, automake BuildRequires: e2fsprogs-devel, libuuid-devel, zlib-devel, libzstd-devel BuildRequires: libacl-devel, libblkid-devel, lzo-devel BuildRequires: asciidoc, xmlto BuildRequires: systemd BuildRequires: python3-devel >= 3.4 BuildRequires: make I'm not sure if it's easier to build the whole thing or if you can just build btrfs-image using the stub makefile in image/ directory. When I build it doesn't take long anyway, the longest is docs :P If you can't get it to build, lemme know. If it does build and now you get an image file; if it's bigger than 20M it won't attach to the bug I filed; so you'll have to upload it to e.g. google drive and the post the URL in the bug or ping me on #fedora. Then we'll test if current fsck can fix it, and if not then enhance it. So we're maybe half way there. Obviously the first goal is fix the fs, but then a second goal is to see if the systemd journal on this file system has any hints about what went wrong because that's the *actual* important thing to get to, is the finish line. That way it hopefully doesn't happen again to you or anyone else. It's bad luck and tedious but sometimes the way it goes. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
I've filed a bug for tracking. https://bugzilla.redhat.com/show_bug.cgi?id=1912598 ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Sun, Jan 3, 2021 at 9:11 AM Sreyan Chakravarty wrote: > > I don't know what happened, but I just logged into my system and > deleted some unused files in my home directory. Just some directories. > > Suddenly everything on my system became read only. While rebooting I > think I saw messages like: > > "BTRFS Error" What kernel version was running when this happened? (I want to know the kernel version running at the time of the first instance of a problem.) Thanks, -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 4, 2021 at 1:56 PM Chris Murphy wrote: > > An alternative is matrix. We have a matrix-irc bridge in #fedora and > pretty soon I think the plan is to switch mainly to matrix. So if you > know about matrix then you can join #fedora - but I don't know how to > explain it very well since I don't use matrix yet. I think it keeps > the history for you, unlike IRC (I use a bouncer so I will see your > messages later). I keep weird hours so it might overlap at some point. OK one way to do this is get an account at element.io and use their webapp (or android or ios). I still don't know how you find #fedora channel once you're on matrix but maybe that's obvious - probably something like find irc.freenode.net and then find #fedora or #fedora-qa (I'm on both channels). -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 4, 2021 at 11:32 AM Sreyan Chakravarty wrote: > > On Mon, Jan 4, 2021 at 10:14 PM Chris Murphy wrote: > > transid errors like this indicate out of order writes due to drive > > firmware not honoring file system write ordering and then getting a > > badly timed crash/powerfail/shutdown. > > First of all thanks for your quick response. > > So would I be correct assuming that the problem is in my firmware ? Or > is it too early to say anything like that ? Too early. The usual case of transid errors is drive firmware bugs *and* ill timed shutdown. Since you don't have an ill timed shutdown, it's less likely this is a drive firmware bug, but can't be ruled out. i.e. I'm proposing there might be a software bug here and we just need to figure it out. Bad memory usually shows up as bit flips and doesn't result in damage like this - but it has to be considered whether a bitflip can affect code. It can also be a kernel bug - the storage stack has many layers, not just Btrfs and dm-crypt. But no one wants to go blaming other people's work without understanding the problem. > Is my firmware so outdated that it can't handle BTRFS ? No. It's a bit complicated. Buggy drive firmware is common. But normally it doesn't matter mainly due to good luck. More than one thing has to go wrong to cause a problem like (a) firmware bug exists (b) firmware bug is triggered (c) crash/powerfail. If one of those is not true, then it's not a problem. There is also the transient hardware defect problem that can act like a bug but it's just rotting the metadata or data. It's not obvious but it is possible to piece together what's happened when we have enough information. > # btrfs-image -c9 -t4 /dev/mapper/dm_crypt /run/media/liveuser/Backup\ > Plus/btrfs_meta.img > > parent transid verify failed on 55640064 wanted 44146 found 44438 > parent transid verify failed on 55640064 wanted 44146 found 44438 > parent transid verify failed on 55640064 wanted 44146 found 44438 > Ignoring transid failure > parent transid verify failed on 55902208 wanted 44170 found 44438 > Ignoring transid failure > parent transid verify failed on 56410112 wanted 44170 found 44439 > Ignoring transid failure > parent transid verify failed on 58621952 wanted 44170 found 44439 > Ignoring transid failure > ERROR: child eb corrupted: parent bytenr=178081497088 item=246 parent > level=1 child level=2 > ERROR: cannot go to next leaf -5 > ERROR: create failed: -5 > > What do I do now ? Rats. Can you retry by adding -w option? In the meantime I'll report back to upstream and see what they recommend next. > > I'm on irc.freenode.net as cmurf that's usually the easier way to get > > help, on #fedora channel. > > > > Do I need to have a bouncer ? I am in India, and I believe you are in > the US, so when you are active, I am usually sleeping. An alternative is matrix. We have a matrix-irc bridge in #fedora and pretty soon I think the plan is to switch mainly to matrix. So if you know about matrix then you can join #fedora - but I don't know how to explain it very well since I don't use matrix yet. I think it keeps the history for you, unlike IRC (I use a bouncer so I will see your messages later). I keep weird hours so it might overlap at some point. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Mon, Jan 4, 2021 at 6:59 AM Sreyan Chakravarty wrote: > > On Mon, Jan 4, 2021 at 1:16 AM Chris Murphy wrote: > > > > Try to mount normally, then: > > I am unable to mount normally : > > # mount -t btrfs /dev/mapper/dm_crypt /mnt/ > mount: /mnt: wrong fs type, bad option, bad superblock on > /dev/mapper/dm_crypt, missing codepage or helper program, or other > error. > > > > > dmesg > > This is what I get in dmesg: > > [29867.234062] BTRFS info (device dm-4): disk space caching is enabled > [29867.234067] BTRFS info (device dm-4): has skinny extents > [29867.317955] BTRFS error (device dm-4): parent transid verify failed > on 55640064 wanted 44146 found 44438 > [29867.326701] BTRFS error (device dm-4): parent transid verify failed > on 55640064 wanted 44146 found 44438 > [29867.326727] BTRFS warning (device dm-4): failed to read root (objectid=9): > -5 > [29867.333668] BTRFS error (device dm-4): open_ctree failed transid errors like this indicate out of order writes due to drive firmware not honoring file system write ordering and then getting a badly timed crash/powerfail/shutdown. However... You report that the file system went read only while using it. This suggests a dropped write and the file system went read-only to limit the damage. Ideally we'd get the log, if it made it to disk, to see what lead up to this so we can determine what the problem is and get it fixed. What I can tell you is this is not user error but that's not much comfort. > > > btrfs check --readonly > > A lot of errors, could not even upload to pastebin. > > This is in my Google Drive: > https://drive.google.com/file/d/1dpW7aftB3FuD8i1J7d4nRrzZHaGF4vuN/view?usp=sharing Yeah that's bad. I think it's fixable. We need to get a metadata dump of the file system to see if fsck will fix it. btrfs-image -c9 -t4 /dev/sdXY /mnt/path/to/file That will include filenames but not any data. If you need to mask filenames, add -ss option to the above. (-s won't help here). And the path to file if you're on a live USB stick can just be something like ~/sreyan-btrfs.img and then put it up on the google drive. By the way I'm on irc.freenode.net as cmurf that's usually the easier way to get help, on #fedora channel. Also, have you ever done a balance on this file system? (That is not a suggestion that you should or shouldn't have. Just a yes or no question to try and piece together some other data points.) -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
By the way, fpaste is installed by default in Fedora, and it has a new feature: --btrfsinfo fpaste --btrfsinfo Then paste the URL here. It'll likely format way better than whatever MUA you're using. --- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs RAID 5?
On Sun, Jan 3, 2021, 6:26 AM Richard Shaw wrote: > Chris, > > Thanks for the detailed explanation. Too much to quote for my follow up > question :) > > So for 3 drives and my desire to have more capacity and redundancy (for > drive failure) would I be better off with RAID1 or RAID5 w/ btrfs? > Depends on what you mean by better. :D In terms of survivability of your data? You're better off with more independent copies/backups than you are with any kind of raid. Raid improves availability, i.e. instead of 0% working it has a degraded mode where it's mostly working, but will require specific actions to make it healthy again, before it also becomes 0% working. Would RAID1 still allow for any single drive failure? > Yes. Same for raid10. > I don't really need the faster IO RAID5 might provide. > Striped parity raids perform well for sequential workloads. They're not good for metadata heavy workloads. Btrfs alters this calculation because metadata (the fs itself) can have a different profile than data. i.e. the recommendation is to use raid1 metadata when using raid5 data; and raid1c3 metadata when using raid6 data. By the way that does introduce quite an odd duck. You can have raid1 metadata, and single data. I'll let that hang out to dry until someone wants to start a thread about that thing :D Funny enough it's now the default mkfs for multiple device Btrfs arrays, first in Fedora. And it also behaves the way pretty much everyone who knows *nothing* about file systems and concat raid arrays expects it should behave. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: BTRFS partition corrupted after deleting files in /home
On Sun, Jan 3, 2021, 9:11 AM Sreyan Chakravarty wrote: > Hi, > > I don't know what happened, but I just logged into my system and > deleted some unused files in my home directory. Just some directories. > > Suddenly everything on my system became read only. While rebooting I > think I saw messages like: > > "BTRFS Error" > > I booted into a live environment to restore my snapshots, then while > mounting the BTRFS root partition I got: > > mount: /mnt: wrong fs type, bad option, bad superblock on > /dev/mapper/dm_crypt, missing codepage or helper program, or other > error. > Try to mount normally, then: dmesg btrfs check --readonly Post the results. You can then try: mount -o ro,usebackuproot -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: btrfs RAID 5?
On Sat, Jan 2, 2021 at 11:55 AM Richard Shaw wrote: > > From what I've been able to find online it doesn't look like RAID 5 is well > supported by btrfs currently. Anyone have any data to the contrary? There are caveats. If you can live with them, then it's OK. But obligatory reading: https://lore.kernel.org/linux-btrfs/20200627032414.gx10...@hungrycats.org/ You definitely want metadata raid1, and use free space tree (space_cache v2) for this setup. A top use case and expectation is the uptime in the face of a device failure. If you're able to quickly do a 'btrfs replace' (another requirement, don't use 'dev add' + 'dev remove') then it might be OK. > I have a 4TB media drive that's filling up and I'd like to add two more disks > to both increase capacity and add some redundancy. My understanding is that > if I did a 3 drive btrfs RAID 1 that I would just have 3 copies of the same > data and zero increased capacity. No. Btrfs raid1 is exactly two copies no matter how many drives are in the pool. There's raid1c3 for three copies, and raid1c4 for four copies. A three device btrfs raid1 is valid. Btrfs does not do raid at a block level like md raid. It's at the block group level. A block group is a ~1G contiguous region of extents. You could consider the block group a super extent, similar to how an extent is a collection of blocks. Like extent size, the block group size isn't fixed, but generally it appears in 1G amounts. The block group "profile" is what determines the number and behavior of chunks that get allocated to various devices. A chunk is the physical manifestation of a block group on actual devices. The single profile means 1:1 block group to chunk. The dup profile means 1:2 block group to chunk *on the same device*. The raid1 profile also is 1:2 block group to chunk but on different devices. And how this gets allocated depends mainly on free space remaining. So it is possible for a block group with raid1 profile to be allocated with mirror chunks on drive A and B, and for the very next block group allocated will be chunks on drives B and C. This is why free space reporting on Btrfs can be confusing. Even number of same sized drives with raid1 is OK, but then odd number of drives or mix and match drive sizes and it gets screwy, even though it's valid and Btrfs will use Tetris technique to fill in the free space wherever it is. https://github.com/kdave/btrfs-progs/issues/277 https://carfax.org.uk/btrfs-usage/ -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Migration to btrfs: Is a reinstall the best way?
On Sat, Jan 2, 2021 at 9:04 AM Richard Shaw wrote: > > My plan is to upgrade from my current 500GB Samsung EVO to a 1TB version and > copy all my data over. > > Creating the subvolumes and updating fstab is easy enough, but what else > needs to be done? grub2-mkconfig. The root fs UUID has changed as well as needs a hint for the subvolume to use as sysroot, e.g. rootflags=subvol=root. These are picked up by grub2-mkconfig. initramfs probably does not need to be done, because btrfs is now built-in to the kernel instead of as a module. > Would a reinstall do any "magic" that's difficult to do by hand? Not really. When in doubt use defaults. And the installer uses upstream defaults. Two features will soon be the default, are stable for a long time, and I use them both. Since you're doing your own mkfs you could opt in now: '-R free-space-tree -O no-holes' - both can be enabled after mfks time though too. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: Backing up Btrfs (was: btrfs or ext4)
On Sat, Jan 2, 2021 at 6:27 AM Patrick O'Callaghan wrote: > > Brief recap: I'm currently using BorgBackup, which among other things > does compression and deduplication. I'm looking at the pros and cons of > changing to Btrfs for backup (I already use it as my main fs). The > backup drives are currently ext4 so would need to be converted. > > Now read on. > > Following Chris Murphy's suggestion, I looked at: > > https://fedoramagazine.org/btrfs-snapshots-backup-incremental/ > > My question is whether I can use compression and deduplication on the > backup even if these have not been enabled on the original fs. My > instinct is to say no, but I could be wrong. You can use compression and/or dedup on the backup even if they are not used on the original file system (whether ext4 or Btrfs). Compression is just a mount option on Btrfs. [1] The current version of the Btrfs send stream is not compressed. [2] Btrfs dedup is "out-of-band" in that the file system is mounted, but the dedup is not live. Writes, including duplicates, get written to disk, and are later submitted for deduplication by a utility like duperemove (in Fedora repo) or Bees. [3] The way reflinks (efficient copies), snapshots, and dedup relate to each other, is they all result in shared extents. The actions for arriving at shared extents differ quite a bit. What makes btrfs different from other file systems, and how it relates to send/receive, is it has the ability to cheaply compute differences between snapshots, without requiring deep traversal on either the send or receive file systems. The less your /home changes, the more convenient it is to keep the backup up to date, because the cost of figuring out what to back up is so chea. If your ~/ contains many things that change that you don't care about backing up, then most any other backup solution provides "excludes" to filter out such things. Btrfs send/receive doesn't have such a thing. The closest approximation is you can create a nested subvolume in place of a directory to create a snapshot boundary, because btrfs snapshots are not recursive into subvolumes. e.g. you could create a new ~/.cache subvolume to substitute the directory. It could be a ~1G of savings just for the Firefox cache alone. Note: subvolumes have their own pool of inodes, they're separate file trees not separate file systems, but can sometimes behave like a separate file system. Programs like rsync and find will include subvolumes by default unless you use -x or -xdev respectively. There's an equivalent option in borgbackup which is enabled by default. So it excludes nested subvolumes and snapshots by default. Part of the reasoning for this is that folks with many snapshots were finding that their backup program sees each snapshot as a completely separate and massive pile of files subject to being backed up. Sooo yeah... something to watch out for. If you ask borg to backup /, therefore it will not automatically backup /home because it's considered a separate "file system". Whew... -- Chris Murphy [1] You can change the compression option while mounted by using the remount option along with the new compression values you want. Mix and match is valid. Whatever is set at the time of writing, is what's being used. If you forget to mount with compression, then those writes aren't compressed. Etc. [2] There are patches upstream that will persist compression in the stream; it's preliminary work to support native encryption which will likewise preserve encryption in the send stream. This means we'll be able to replicate snapshots even if they aren't unlocked, and the replication uses the same key as the original. Stay turned. [3] https://github.com/Zygo/bees -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: usb stick problem
Slight tangent, but once getting new flash, test it. I recommend using f3 on any new USB stick. It's in Fedora repo. The project is originally intended to identify fake flash. It'll report a larger capacity than it has flash memory for, and has firmware that does a loop. Super convenient that if you never hit the loop, you don't find out; and if you hit the loop, it overwrites and corrupts earlier data - sad panda. But it also will test for any kind of corruptions or read errors, not just bogus flash. It's also reasonable to use it for "burning in" hard drives - it's just writing a bunch of pattern tests and reading them back in. It doesn't matter what file system you use to format the device. Format, mount, f3write /mnt then f3read /mnt. That's it. My opinion is anything brand new that has even one error, send it back. It's not like it's gonna get better with age! Details: https://github.com/AltraMayor/f3 https://fight-flash-fraud.readthedocs.io/en/latest/ -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: ERROR: installing 'ext4', when installing kernel-core-5.9.16-200.fc33.x86_64
On Sat, Jan 2, 2021, 1:23 AM Sreyan Chakravarty wrote: > Hi, > > I am trying to create a livecd by using the livecd-creator. > > But whenever during the build process I get the error message: > > dracut: No '/dev/log' or 'logger' included for syslog logging > dracut-install: ERROR: installing 'sr_mod' > dracut-install: ERROR: installing 'ext4' > sort: fflush failed: 'standard output': Broken pipe > sort: write error > > gzip: stdout: Broken pipe > > gzip: stdout: Broken pipe > sort: write failed: 'standard output': Broken pipe > sort: write error > > I have tried to debug this via adding selinux-policy-targeted to my > kickstart, but nothing changed. > > I have tried unpacking the scripts from > kernel-core-5.9.16-200.fc33.x86_64.rpm to see what was going wrong, > but I could see what was going wrong. > > rpm -q --scripts kernel-core-5.9.16-200.fc33.x86_64.rpm > > postinstall scriptlet (using /bin/sh): > if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ] && >[ -f /etc/sysconfig/kernel ]; then > /bin/sed -r -i -e > 's/^DEFAULTKERNEL=kernel-smp$/DEFAULTKERNEL=kernel/' > /etc/sysconfig/kernel || exit $? > fi > > preuninstall scriptlet (using /bin/sh): > /bin/kernel-install remove 5.9.16-200.fc33.x86_64 > /lib/modules/5.9.16-200.fc33.x86_64/vmlinuz || exit $? > posttrans scriptlet (using /bin/sh): > /bin/kernel-install add 5.9.16-200.fc33.x86_64 > /lib/modules/5.9.16-200.fc33.x86_64/vmlinuz || exit $? > > > As you can see there is no dracut_install. > > Can anyone tell me where dracut_install is being called from ? What is > going wrong here ? > ext4 is built-in so there is no kernel module for dracut to find. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
Re: entropy generation
On Fri, Jan 1, 2021, 7:14 PM Tom Horsley wrote: > On Fri, 1 Jan 2021 19:06:44 -0700 > Chris Murphy wrote: > > > You can remove the rng-tools package if you want. It's being removed > > in Fedora 34. > > So where does random data come from in f34? Kernel handles it. -- Chris Murphy ___ users mailing list -- users@lists.fedoraproject.org To unsubscribe send an email to users-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
