FS gives kernel UPS on attempt to create snapshot and after running balance it's unmountable.
Hi all, So my main storage filesystem got some sort of veird corruption (that I can gather). Everything seems to work OK, but when I try to create a snapshot or run balance (no filters) it will get remounted read only. Fun part is that balance seems to be running even on read only FS, and I continuously get kernel traces in /var/log/messages so it might as well in the back ground silently eat my data away :/ UPDATE: Yeah, after rebooting the system it does not even mount the FS, mount.btrfs sits in some sort of spinlock and consumes 100% of singe core. UPDATE 2: System is completelly cooked :/ [root@server ~]# btrfs fi show Label: 'rockstor_server' uuid: 5581a647-40ef-4a7a-9d73-847bf35a142b Total devices 1 FS bytes used 5.72GiB devid1 size 53.17GiB used 7.03GiB path /dev/sda2 Label: 'broken_pool' uuid: 26095277-a234-455b-8c97-8dac8ad934c8 Total devices 2 FS bytes used 193.52GiB devid1 size 1.82TiB used 196.03GiB path /dev/sdb devid2 size 1.82TiB used 196.03GiB path /dev/sdi Label: 'main_pool' uuid: 0576d577-8954-4a60-a02b-9492b3c29318 Total devices 8 FS bytes used 5.83TiB devid1 size 1.82TiB used 1.50TiB path /dev/sdc devid2 size 1.82TiB used 1.50TiB path /dev/sdd devid3 size 1.82TiB used 1.50TiB path /dev/sde devid4 size 1.82TiB used 1.50TiB path /dev/sdf devid5 size 1.82TiB used 1.50TiB path /dev/sdg devid6 size 1.82TiB used 1.50TiB path /dev/sdh devid7 size 1.82TiB used 1.50TiB path /dev/sdj devid8 size 1.82TiB used 1.50TiB path /dev/sdk [root@server ~]# mount /dev/sdc /mnt2/main_pool/ mount: wrong fs type, bad option, bad superblock on /dev/sdc, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. [root@server ~]# mount /dev/sdd /mnt2/main_pool/ mount: wrong fs type, bad option, bad superblock on /dev/sdd, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. [root@server ~]# mount /dev/sde /mnt2/main_pool/ mount: wrong fs type, bad option, bad superblock on /dev/sde, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. dmesg tail retuns: [ 9507.835629] systemd-udevd[1873]: Validate module index [ 9507.835656] systemd-udevd[1873]: Check if link configuration needs reloading. [ 9507.835690] systemd-udevd[1873]: seq 3698 queued, 'add' 'bdi' [ 9507.835873] systemd-udevd[1873]: seq 3698 forked new worker [13858] [ 9507.836202] BTRFS info (device sdd): disk space caching is enabled [ 9507.836204] BTRFS info (device sdd): has skinny extents [ 9507.836322] systemd-udevd[13858]: seq 3698 running [ 9507.836443] systemd-udevd[13858]: no db file to read /run/udev/data/+bdi:btrfs-4: No such file or directory [ 9507.836474] systemd-udevd[13858]: RUN '/bin/mknod /dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1 [ 9507.837366] systemd-udevd[13861]: starting '/bin/mknod /dev/btrfs-control c 10 234' [ 9507.837833] BTRFS error (device sdd): failed to read the system array: -5 [ 9507.838231] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c 10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists' [ 9507.838262] systemd-udevd[13858]: '/bin/mknod /dev/btrfs-control c 10 234' [13861] exit with return code 1 [ 9507.854757] BTRFS: open_ctree failed [ 9511.370878] BTRFS info (device sdd): disk space caching is enabled [ 9511.370881] BTRFS info (device sdd): has skinny extents [ 9511.375097] BTRFS error (device sdd): failed to read the system array: -5 [ 9511.392792] BTRFS: open_ctree failed [ 9514.233627] BTRFS: device label main_pool devid 3 transid 150680 /dev/sde [ 9514.234399] systemd-udevd[1873]: Validate module index [ 9514.234431] systemd-udevd[1873]: Check if link configuration needs reloading. [ 9514.234465] systemd-udevd[1873]: seq 3702 queued, 'add' 'bdi' [ 9514.234522] systemd-udevd[1873]: passed 142 bytes to netlink monitor 0x5628f65d40d0 [ 9514.234554] systemd-udevd[13882]: seq 3702 running [ 9514.234780] systemd-udevd[13882]: no db file to read /run/udev/data/+bdi:btrfs-6: No such file or directory [ 9514.234790] BTRFS info (device sde): disk space caching is enabled [ 9514.234792] BTRFS info (device sde): has skinny extents [ 9514.234798] systemd-udevd[13882]: RUN '/bin/mknod /dev/btrfs-control c 10 234' /etc/udev/rules.d/64-btrfs.rules:1 [ 9514.235181] systemd-udevd[13906]: starting '/bin/mknod /dev/btrfs-control c 10 234' [ 9514.236448] systemd-udevd[13882]: '/bin/mknod /dev/btrfs-control c 10 234'(err) '/bin/mknod: '/dev/btrfs-control': File exists' [ 9514.236514] systemd-udevd[13882]: '/bin/mknod /dev/btrfs-control c 10 234' [13906] exit with return code 1 [ 9514.238726] BTRFS error (device sde): failed to read the system array: -5 [ 9514.255472] BTRFS: open_ctree failed -- To unsubscribe from thi
Re: proper syslinux settings for ext4 boot, dm-crypt + btrfs subvol root?
On Fri, Feb 10, 2017 at 11:40 AM, John Hendy wrote: > Greetings, > > > I'm trying to utilize btrfs subvols to allow me to boot separate > distros without having to create so many partitions. I'm on Arch Linux > and not finding the wiki super helpful with specifics on dm-crypt/luks > and btrfs. [snip] > Here's the setup I'm looking for once booted, with the /mnt/btf becoming / > > $ mount > /dev/mapper/btr on /mnt/btr type btrfs > (rw,relatime,ssd,space_cache,subvolid=259,subvol=/arch/root) > /dev/mapper/btr on /mnt/btr/home type btrfs > (rw,relatime,ssd,space_cache,subvolid=260,subvol=/arch/home) > /dev/sdb1 on /mnt/btr/boot type ext4 (rw,relatime,data=ordered) > > My functioning syslinux.cfg: > > LABEL arch-ssd > MENU LABEL arch-ssd-uuid > LINUX ../vmlinuz-linux > APPEND luks.uuid=7101e83b-31c0-4cdf-bc07-678e00e19c32 > root=UUID=eb20c219-0df8-4051-bad2-39d57aed7b59 luks.allow-discards rw > INITRD ../intel-ucode.img,../initramfs-linux.img > > I tried this: > > LABEL arch-btrfs > MENU LABEL arch-btrfs-uuid > LINUX ../vmlinuz-linux > APPEND luks.uuid=bcacb6d5-2874-4652-a25d-88b0bf3bbce8 > root=UUID=e1284231-c264-4944-807d-5fcb1832ce47 > rootflags=subvol=/arch/root luks.allow-discards rw > INITRD ../intel-ucode.img,../initramfs-linux.img > > Current behavior is I successfully get a syslinux boot menu (at least > after I disabled the 64bit default option), but selecting the above > entry just refreshes the menu and I'm stuck there with an automatic > countdown from 5sec and then back to 5sec when it hits 0. It's like it > just keeps pointing at the /dev/sdb MBR or can't find something and > thus tries again? Closing the loop on this. It took a while but after a couple of install option tries, I got ubuntu planing nicely with syslinux so I figured the arch process would be piece of cake. I had everything setup and then went to boot and got the same behavior... I tabbed to get out the loop behavior and thought "ah! I wonder if I installed the intel-ucode package?" So, previously, I thought it was specifying the compress=lzo option, but now I think that was just one of many steps I did when fiddling with the syslinux entry above. I'm guessing another was removing the loading of intel-ucode. I don't want to go through all the steps again to be sure, but at least wanted this updated with my experience in case someone else stumbles on the issue. Thus far: - I have a single /boot partition (sdb1) with syslinux installed - I can boot either arch or ubuntu from separate btrfs subvols on the same encrypted partition (sdb2) - thus far I just have an arch and ubuntu top level subvol with nested home subvol under each - I *don't* think the issue was loading a nested subvol for root [1] - this is awesome and so much cooler than having to guess what size to make everything ahead of time Thanks for the quick responses all and like most issues the problem was almost certainly me! John [1] this post suggests loading __active/rootvol - https://www.brunoparmentier.be/blog/how-to-install-arch-linux-on-an-encrypted-btrfs-partition.html [snip] > Best regards, > John -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Subvolume corruption after restart on Raid1 array
Hello all, I have been running a Rockstor 3.8.16-8 on an older Dell Optiplex for about a month. The system has four drives separated into two Raid1 filesystems (“pools” in Rockstor terminology). A few days ago I restarted it and noticed that the services (NFS, Samba, etc) weren’t working. Looking at dmesg, I saw: kernel: BTRFS error (device sdb): parent transid verify failed on 1721409388544 wanted 19188 found 83121 and sure enough, one of the subvolumes on my main filesystem is corrupted. By corrupted I mean it can’t be accessed, deleted, or even looked at: ls -l kernel: BTRFS error (device sdb): parent transid verify failed on 1721409388544 wanted 19188 found 83121 kernel: BTRFS error (device sdb): parent transid verify failed on 1721409388544 wanted 19188 found 83121 ls: cannot access /mnt2/Primary/Movies: Input/output error total 16 drwxr-xr-x 1 root root 100 Dec 29 02:00 . drwxr-xr-x 1 root root 208 Jan 3 12:05 .. drwxr-x--- 1 kbogert root 698 Feb 6 08:49 Documents drwxr-xrwx 1 root root 916 Jan 3 12:54 Games drwxr-xrwx 1 xenserver xenserver 2904 Jan 3 12:54 ISO d? ? ? ? ?? Movies drwxr-xrwx 1 root root 139430 Jan 3 12:53 Music drwxr-xrwx 1 root root 82470 Jan 3 12:53 RawPhotos drwxr-xr-x 1 root root 80 Jan 1 04:00 .snapshots drwxr-xrwx 1 root root 72 Jan 3 13:07 VMs The input/output error is given for any operation on Movies. Luckily there has been no data loss that I am aware of. As it turns out I have a snapshot of the Movies subvolume taken a few days before the incident. I was able to simply cp -a all files off of the entire filesystem, with no reported errors, and verified a handful of them. Note that the transid error in dmesg alternates between sdb and sda5 after each startup. SETUP DETAILS uname -a Linux ironmountain 4.8.7-1.el7.elrepo.x86_64 #1 SMP Thu Nov 10 20:47:24 EST 2016 x86_64 x86_64 x86_64 GNU/Linux btrfs —version btrfs-progs v4.8.3 btrfs dev scan kernel: BTRFS: device label Primary devid 1 transid 83461 /dev/sdb kernel: BTRFS: device label Primary devid 2 transid 83461 /dev/sda5 btrfs fi show /mnt2/Primary Label: 'Primary' uuid: 21e09dd8-a54d-49ec-95cb-93fdd94f0c17 Total devices 2 FS bytes used 943.67GiB devid1 size 2.73TiB used 947.06GiB path /dev/sdb devid2 size 2.70TiB used 947.06GiB path /dev/sda5 btrfs dev usage /mnt2/Primary /dev/sda5, ID: 2 Device size: 2.70TiB Device slack: 0.00B Data,RAID1:944.00GiB Metadata,RAID1: 3.00GiB System,RAID1: 64.00MiB Unallocated: 1.77TiB /dev/sdb, ID: 1 Device size: 2.73TiB Device slack: 0.00B Data,RAID1:944.00GiB Metadata,RAID1: 3.00GiB System,RAID1: 64.00MiB Unallocated: 1.80TiB btrfs fi df /mnt2/Primary Data, RAID1: total=944.00GiB, used=942.60GiB System, RAID1: total=64.00MiB, used=176.00KiB Metadata, RAID1: total=3.00GiB, used=1.07GiB GlobalReserve, single: total=512.00MiB, used=0.00B This server is very light use, however, I do have a number of VMs in the VMs filesystem, exported over NFS, that are used by a Xenserver. These are not marked nocow, though I probably should have. At the time of restart no VMs were running. I have deviated from Rockstor’s default setup a bit. They take an “appliance” view and try to enforce btrfs partitions that cover entire disks. I installed Rockstor onto /dev/sda4, created the Primary partition on /dev/sdb using Rockstor’s gui, then on the command line added /dev/sda5 to it and converted to raid1. As far as I can tell Rockstor is just CentOS 7 with a few updated utilities and a bunch of python scripts for providing a web interface to btrfs-progs. I have it setup to take monthly snapshots and do monthly scrubs, with the exception of the Documents subvolume which takes daily snapshots. These are all readonly and go in the .snapshots directory. Rockstor automatically deletes old snapshots once a limit is reached (7 daily snapshots, for instance). Side note, btrfs-progs 4.8.3 apparently has problems with CentOS 7’s glibc: https://github.com/rockstor/rockstor-core/issues/1608 . I have confirmed that bug in my own compiled version of 4.8.3, and that 4.9.1 does not have it. WHAT I’VE TRIED AND RESULTS First off, I have created an image with btrfs-image that I can make available (though large, I believe it was a few Gbs and the filesystem is 3 TB) * btrfs-zero-log had no discernible effect. * At this point, I compiled btrfs-progs 4.9.1. The following commands were run with this version: * btrfs check This exits in an assert fairly quickly: checking extents cmds-check.c:5406: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1 /mnt/usb/btrfs-progs-bin/bin/btrfs[0x42139b] /
[GIT PULL] Btrfs
Hi Linus, My for-linus-4.10 branch: git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git for-linus-4.10 Has two last minute fixes. The highest priority here is a regression fix for the decompression code, but we also fixed up a problem with the 32 bit compat ioctls. The decompression bug could hand back the wrong data on big reads when zlib was used. I have a larger cleanup to make the math here less error prone, but at this stage in the release Omar's patch is the best choice. Omar Sandoval (1) commits (+24/-15): Btrfs: fix btrfs_decompress_buf2page() Jeff Mahoney (1) commits (+4/-2): btrfs: fix btrfs_compat_ioctl failures on non-compat ioctls Total: (2) commits (+28/-17) fs/btrfs/compression.c | 39 --- fs/btrfs/ioctl.c | 6 -- 2 files changed, 28 insertions(+), 17 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] Btrfs: fix btrfs_decompress_buf2page()
On Fri, Feb 10, 2017 at 08:12:35PM -0800, Pat Erley wrote: On 02/10/17 15:03, Omar Sandoval wrote: From: Omar Sandoval If btrfs_decompress_buf2page() is handed a bio with its page in the middle of the working buffer, then we adjust the offset into the working buffer. After we copy into the bio, we advance the iterator by the number of bytes we copied. Then, we have some logic to handle the case of discontiguous pages and adjust the offset into the working buffer again. However, if we didn't advance the bio to a new page, we may enter this case in error, essentially repeating the adjustment that we already made when we entered the function. The end result is bogus data in the bio. Previously, we only checked for this case when we advanced to a new page, but the conversion to bio iterators changed that. This restores the old, correct behavior. I can confirm this fixes the corruption I was seeing Feel free to add: Tested-by: Pat Erley Thanks again Pat for bisecting this down. It passed overnight so I'm sending in right now. -chris -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
On Saturday 11 February 2017 03:01:39 Kai Krakow wrote: > Am Fri, 10 Feb 2017 23:15:03 +0100 > > schrieb Marc Joliet : > > # btrfs filesystem df /media/MARCEC_BACKUP/ > > Data, single: total=851.00GiB, used=831.36GiB > > System, DUP: total=64.00MiB, used=120.00KiB > > Metadata, DUP: total=13.00GiB, used=10.38GiB > > Metadata, single: total=1.12GiB, used=0.00B > > GlobalReserve, single: total=512.00MiB, used=0.00B > > > > Hmm, I take it that the single metadata is a leftover from running > > --repair? > > It's more probably a remnant of an incomplete balance operation or an > older mkfs version. I'd simply rebalance metadata to fix this. > > I don't think that btrfs-repair would migrate missing metadata > duplicates back to single profile, it would more likely trigger > recreating the missing duplicates. But I'm not sure. I'm fairly certain it's from the repair operation, the device didn't have any single metadata before (and it still didn't really, notice that used=0.00B). So my best guess is that it was allocated during the --repair. > If it is a result of the repair operation, that could be an > interesting clue. Could it explain "error -17" from your logs? But that > would mean the duplicates were already missing before the repair > operation and triggered that problem. So the question is, why are those > duplicates missing in the first place as a result of normal operation? > From your logs: I certainly didn't create them myself, the device has always had metadata=dup (you can check previous threads of mine, which also contain "btrfs fi df" output). So I don't think it has anything to do with the failures. > ---8<--- snip > Feb 02 22:49:14 thetick kernel: BTRFS: device label MARCEC_BACKUP devid > 1 transid 283903 /dev/sdd2 > Feb 02 22:49:19 thetick kernel: EXT4-fs (sdd1): mounted filesystem with > ordered data mode. Opts: (null) > Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): use zlib > compression Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): > disk space caching is enabled > Feb 03 00:18:52 thetick kernel: BTRFS info (device sdd2): has skinny > extents Feb 03 00:20:09 thetick kernel: BTRFS info (device sdd2): The > free space cache file (3967375376384) is invalid. skip it > Feb 03 01:05:58 thetick kernel: [ cut here ] > Feb 03 01:05:58 thetick kernel: WARNING: CPU: 1 PID: 26544 at > fs/btrfs/extent- tree.c:2967 btrfs_run_delayed_refs+0x26c/0x290 > Feb 03 01:05:58 thetick kernel: BTRFS: Transaction aborted (error -17) > --->8--- snap > > "error -17" being "object already exists". My only theory would be this > has a direct connection to you finding the single metadata profile. > Like in "the kernel thinks the objects already exists when it really > didn't, and as a result the object is there only once now" aka "single > metadata". > > But I'm no dev and no expert on the internals. Again, I don't think it has anything to do with the single metadata. A "btrfs balance start -mprofile=single" got rid of the (empty) single metadata blocks. What I *can* say is that the error seems to be transient. For example, I ran an rsync last night that failed with the same error as before. After unmounting and mounting the FS again, I could run it to completion (and verified some files). Anyway, here's the whole log since this morning. I can't correlate individual stack traces to specific actions any more, but the rough order of actions on my part (not counting unmount/mount cycles, those are easy enough to find): - backups from the home server run, then - btrfs-cleaner drops old snapshots, after which - I run rsync (which fails), then - I run rsync again (which succeeds), then - I run "balance start -mprofile=single", which succeeds (I might have run this before rsync, I'm not sure), after which - I start backups on my laptop (which fails after I finally went to bed, at about 3:58). After that I just try "btrbk -r run" a few times, which managed to transfer the first two snapshots before failing. Feb 11 00:00:06 diefledermaus kernel: usb 1-1: new high-speed USB device number 4 using ehci-pci Feb 11 00:00:06 diefledermaus kernel: usb 1-1: New USB device found, idVendor=0480, idProduct=d010 Feb 11 00:00:06 diefledermaus kernel: usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 Feb 11 00:00:06 diefledermaus kernel: usb 1-1: Product: External USB 3.0 Feb 11 00:00:06 diefledermaus kernel: usb 1-1: Manufacturer: Toshiba Feb 11 00:00:06 diefledermaus kernel: usb 1-1: SerialNumber: 20130421020612 Feb 11 00:00:07 diefledermaus kernel: usb-storage 1-1:1.0: USB Mass Storage device detected Feb 11 00:00:07 diefledermaus kernel: usb-storage 1-1:1.0: Quirks match for vid 0480 pid d010: 2000 Feb 11 00:00:07 diefledermaus kernel: scsi host6: usb-storage 1-1:1.0 Feb 11 00:00:07 diefledermaus kernel: usbcore: registered new interface driver usb-storage Feb 11 00:00:16 diefledermaus kernel: scsi 6:0:0:0: Direct-Access Toshiba
