Re: btrfs and systemd
Hi Qu, Am 29.08.2016 um 03:48 schrieb Qu Wenruo: > > > At 08/29/2016 04:15 AM, Stefan Priebe - Profihost AG wrote: >> Hi, >> >> i'm trying to get my 60TB btrfs volume to mount with systemd at boot. >> But this always fails with: "mounting timed out. Stopping." after 90s. > > 60TB is quite large, and under most case it will already cause mount > speed problem. > > In our test environment, filling a fs with 16K small files to 2T (just > 128K files)will already slow the mount process to 10s. > > For larger fs, or more specifically, large extent tree, will slow the > mount process obviously. > > The root fix will need a rework of extent tree. > AFAIK Josef is working on the rework. > > So the btrfs fix will need some time. thanks but i've no problem with the long mount time (in my case 6 minutes) i'm just wondering how to live with it with systemd. As it always cancels the mount process after 90s and i see no fstab option to change this. Greets, Stefan > > Thanks, > Qu >> >> I can't find any fstab setting for systemd to higher this timeout. >> There's just the x-systemd.device-timeout but this controls how long to >> wait for the device and not for the mount command. >> >> Is there any solution for big btrfs volumes and systemd? >> >> Greets, >> Stefan >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> > > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Multiple bugs found by fuzzing BTRFS
Thanks for your fuzzing images. Quite helpful. At 08/29/2016 02:06 PM, Lukas Lueg wrote: Hi, I've now spent around 160 hours of fuzzing BTRFS, here are the crashes I found so far. Every type of crash is reported only once although there are usually multiple locations where they show up (especially heap-use-after-free and calls to abort()). The following bug reports have attached to them images of ±18kb which expand to 16mb and reproduce a crash when running btrfsck; they all have been revirginized so CRC- and FSID-checks pass by a vanilla btrfsck. Use-after-free, shows up all over the place: https://bugzilla.kernel.org/show_bug.cgi?id=153641 Segfault in memcpy, yeah: https://bugzilla.kernel.org/show_bug.cgi?id=154021 Run-off-the-mill buffer-overflow: https://bugzilla.kernel.org/show_bug.cgi?id=154961 Endless loop in btrfsck: https://bugzilla.kernel.org/show_bug.cgi?id=155151 Calls to abort() by lack of error paths: https://bugzilla.kernel.org/show_bug.cgi?id=155181 Division by zero, the old problem of computing stripe_size: https://bugzilla.kernel.org/show_bug.cgi?id=155201 Digging, while it's a little different from the original one. BTW, for btrfsck bugs, would you please also try the new low memory mode? For example, the above image won't trigger bug in low memory mode. Thanks, Qu There are many more crashes like the above; how do you guys want them to be reported? Best regards -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Multiple bugs found by fuzzing BTRFS
Hi, I've now spent around 160 hours of fuzzing BTRFS, here are the crashes I found so far. Every type of crash is reported only once although there are usually multiple locations where they show up (especially heap-use-after-free and calls to abort()). The following bug reports have attached to them images of ±18kb which expand to 16mb and reproduce a crash when running btrfsck; they all have been revirginized so CRC- and FSID-checks pass by a vanilla btrfsck. Use-after-free, shows up all over the place: https://bugzilla.kernel.org/show_bug.cgi?id=153641 Segfault in memcpy, yeah: https://bugzilla.kernel.org/show_bug.cgi?id=154021 Run-off-the-mill buffer-overflow: https://bugzilla.kernel.org/show_bug.cgi?id=154961 Endless loop in btrfsck: https://bugzilla.kernel.org/show_bug.cgi?id=155151 Calls to abort() by lack of error paths: https://bugzilla.kernel.org/show_bug.cgi?id=155181 Division by zero, the old problem of computing stripe_size: https://bugzilla.kernel.org/show_bug.cgi?id=155201 There are many more crashes like the above; how do you guys want them to be reported? Best regards -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send extremely slow (almost stuck)
At 08/29/2016 10:11 AM, Qu Wenruo wrote: At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: Dear btrfs experts, I just tried to make use of btrfs send / receive for incremental backups (using btrbk to simplify the process). It seems that on my two machines, btrfs send gets stuck after transferring some GiB - it's not fully halted, but instead of making full use of the available I/O, I get something < 500 kiB on average, which are just some "full speed spikes" with many seconds / minutes of no I/O in between. During this "halting", btrfs send eats one full CPU core. A "perf top" shows this is spent in "find_parent_nodes" and "__merge_refs" inside the kernel. I am using btrfs-progs 4.7 and kernel 4.7.0. Unknown bug, while unfortunately no good idea to solve yet. Sorry, known bug, not unknown Thanks, Qu I sent a RFC patch to completely disable shared extent detection, while got strong objection. I also submitted some other ideas on fixing it, while still got strong objection. Objection includes this is a performance problem, not a function problem and we should focus on function problem first and postpone such performance problem. And further more, Btrfs from the beginning of its design, focuses on fast snapshot creation, and takes backref walk as sacrifice. So it's not an easy thing to fix. I googled a bit and found related patchwork (https://patchwork.kernel.org/patch/9238987/) which seems to workaround high load in this area and mentions a real solution is proposed but not yet there. Since this affects two machines of mine and backupping my root volume would take about 80 hours in case I can extrapolate the average rate, this means btrfs send is unusable to me. Can I assume this is a common issue which will be fixed in a later kernel release (4.8, 4.9) or can I do something to my FS's to workaround this issue? I don't expect there will be even an agreement on how to fix the problem in v4.1x. Fixes in send will lead to obvious speed improvement, while cause incompatibility or super complex design. Fixes in backref will lead to a backref rework, which normally comes with new regression, and we are even unsure if it will really help. If you just hate the super slow send, and can accept the extra space usage, please try this RFC patch: https://patchwork.kernel.org/patch/9245287/ This patch, just as its name, will completely stop same extent(reflink) detection. Which will cause more space usage, while it skipped the super time consuming find_parent_nodes(), it should at least workaround your problem. I have some other idea to fix it with less aggressive idea, while since there is objection against it, I didn't code it further. But, since there are *REAL* *WORLD* users reporting such problem, I think I'd better restart the fix as an RFC. Thanks, Qu One FS is only two weeks old, the other one now about 1 year. I did some balancing at some points of time to have more unallocated space for trimming, and used duperemove regularly to free space. One FS has skinny extents, the other has not. Mount options are "rw,noatime,compress=zlib,ssd,space_cache,commit=120". Apart from that: No RAID or any other special configuration involved. Cheers and any help appreciated, Oliver -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send extremely slow (almost stuck)
At 08/28/2016 11:38 AM, Oliver Freyermuth wrote: Dear btrfs experts, I just tried to make use of btrfs send / receive for incremental backups (using btrbk to simplify the process). It seems that on my two machines, btrfs send gets stuck after transferring some GiB - it's not fully halted, but instead of making full use of the available I/O, I get something < 500 kiB on average, which are just some "full speed spikes" with many seconds / minutes of no I/O in between. During this "halting", btrfs send eats one full CPU core. A "perf top" shows this is spent in "find_parent_nodes" and "__merge_refs" inside the kernel. I am using btrfs-progs 4.7 and kernel 4.7.0. Unknown bug, while unfortunately no good idea to solve yet. I sent a RFC patch to completely disable shared extent detection, while got strong objection. I also submitted some other ideas on fixing it, while still got strong objection. Objection includes this is a performance problem, not a function problem and we should focus on function problem first and postpone such performance problem. And further more, Btrfs from the beginning of its design, focuses on fast snapshot creation, and takes backref walk as sacrifice. So it's not an easy thing to fix. I googled a bit and found related patchwork (https://patchwork.kernel.org/patch/9238987/) which seems to workaround high load in this area and mentions a real solution is proposed but not yet there. Since this affects two machines of mine and backupping my root volume would take about 80 hours in case I can extrapolate the average rate, this means btrfs send is unusable to me. Can I assume this is a common issue which will be fixed in a later kernel release (4.8, 4.9) or can I do something to my FS's to workaround this issue? I don't expect there will be even an agreement on how to fix the problem in v4.1x. Fixes in send will lead to obvious speed improvement, while cause incompatibility or super complex design. Fixes in backref will lead to a backref rework, which normally comes with new regression, and we are even unsure if it will really help. If you just hate the super slow send, and can accept the extra space usage, please try this RFC patch: https://patchwork.kernel.org/patch/9245287/ This patch, just as its name, will completely stop same extent(reflink) detection. Which will cause more space usage, while it skipped the super time consuming find_parent_nodes(), it should at least workaround your problem. I have some other idea to fix it with less aggressive idea, while since there is objection against it, I didn't code it further. But, since there are *REAL* *WORLD* users reporting such problem, I think I'd better restart the fix as an RFC. Thanks, Qu One FS is only two weeks old, the other one now about 1 year. I did some balancing at some points of time to have more unallocated space for trimming, and used duperemove regularly to free space. One FS has skinny extents, the other has not. Mount options are "rw,noatime,compress=zlib,ssd,space_cache,commit=120". Apart from that: No RAID or any other special configuration involved. Cheers and any help appreciated, Oliver -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs and systemd
At 08/29/2016 04:15 AM, Stefan Priebe - Profihost AG wrote: Hi, i'm trying to get my 60TB btrfs volume to mount with systemd at boot. But this always fails with: "mounting timed out. Stopping." after 90s. 60TB is quite large, and under most case it will already cause mount speed problem. In our test environment, filling a fs with 16K small files to 2T (just 128K files)will already slow the mount process to 10s. For larger fs, or more specifically, large extent tree, will slow the mount process obviously. The root fix will need a rework of extent tree. AFAIK Josef is working on the rework. So the btrfs fix will need some time. Thanks, Qu I can't find any fstab setting for systemd to higher this timeout. There's just the x-systemd.device-timeout but this controls how long to wait for the device and not for the mount command. Is there any solution for big btrfs volumes and systemd? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs and systemd
Stefan Priebe - Profihost AG posted on Sun, 28 Aug 2016 22:15:21 +0200 as excerpted: > Hi, > > i'm trying to get my 60TB btrfs volume to mount with systemd at boot. > But this always fails with: "mounting timed out. Stopping." after 90s. > > I can't find any fstab setting for systemd to higher this timeout. > There's just the x-systemd.device-timeout but this controls how long to > wait for the device and not for the mount command. > > Is there any solution for big btrfs volumes and systemd? Yes. The key here is to realize that systemd creates mount units dynamically based on the fstab file, placing them in /run/systemd/generator/ (take a look =:^), and then uses them to process mounts. So the normal method for reconfiguring/overriding systemd units applies. See the systemd.unit manpage, example section, example 2 (as of systemd 230), for override details, but in general you choose whether you want to override the entire default unit (probably not, as you'd replace the generated file and override the fstab options) or just specific options (likely what you want), and place your override file in the appropriate subdir of /etc/systemd/ accordingly. Now that you know where to put the file with the override, take a look at the systemd.mount manpage, options section. Based on that (IOW, I've not actually tried this), the setting you need to change is TimeoutSec. You can also consider changing the global timeout setting by setting DefaultTimeoutStart in /etc/systemd/system.conf, tho that's going to affect other units as well. But it may be easier, if your other units start up fine without getting anywhere close to the timeout, and thus never need to use the timeout setting. Of course if they break, you'll then be dealing with the longer timeouts, but you may find it easier to simply set that default and deal with the long timeouts on anything else when and if some other unit does actually break and start following the longer default timeouts. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Recommendation on raid5 drive error resolution
Am I right that the wr: 0 means that the disks should at least be in a nice consistent state? I know that overlapping read fails can still cause everything to fail. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Recommendation on raid5 drive error resolution
Current status: Knowing things were bad I did set the scterc values sanely, but the box was getting less stable so I thought a reboot was a good idea. That reboot failed to mount the partition at all and eveything triggered my 'is this a psu issue' sense so I've left the box off till I've got time to check if a psu replacement makes anything happier. That might happen tonight or tomorrow. I'll update the thread when I do that. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send extremely slow (almost stuck)
On Sun, Aug 28, 2016 at 12:15 PM, Oliver Freyermuth wrote: > For me, this means I have to stay with rsync backups, which are sadly > incomplete since special FS attrs > like "C" for nocow are not backed up. Should be able to make a script that creates a textfile with lsattr for every file. Then either just leave that file as part of the backup in case it's needed some day, or making a corresponding script on the backup machine to restore those. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: does btrfs-receive use/compare the checksums from the btrfs-send side?
On Sun, 2016-08-28 at 22:19 +0200, Adam Borowski wrote: > Transports over which you're likely to send a filesystem stream > already > protect against corruption. Well... in some cases,... but not always... just consider a plain old netcat... > It'd still be nice to have something for those which don't, of > course. And it would be even more nice in the case of doing e.g. backups, even if it's from local fs to another local fs so that one doesn't have to do another round of diff'ing, because one already knows the copy is guaranteed to be valid (or at least the checksum is from the source, and a further scrub on the copy would reveal any silent block corruption). Cheers. smime.p7s Description: S/MIME cryptographic signature
Re: does btrfs-receive use/compare the checksums from the btrfs-send side?
On Sun, Aug 28, 2016 at 07:50:42PM +0200, Christoph Anton Mitterer wrote: > On Sun, 2016-08-28 at 11:35 -0600, Chris Murphy wrote: > > I don't see evidence of them in the btrfs send file, so I don't think > > csums are in the stream. > > hmm... isn't that kinda unfortunate not to make use of the information > that's already there? Transports over which you're likely to send a filesystem stream already protect against corruption. It'd still be nice to have something for those which don't, of course. -- An imaginary friend squared is a real enemy. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs and systemd
Hi, i'm trying to get my 60TB btrfs volume to mount with systemd at boot. But this always fails with: "mounting timed out. Stopping." after 90s. I can't find any fstab setting for systemd to higher this timeout. There's just the x-systemd.device-timeout but this controls how long to wait for the device and not for the mount command. Is there any solution for big btrfs volumes and systemd? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfstune settings
Am Sun, 28 Aug 2016 20:10:38 +0200
schrieb Oliver Freyermuth :
> > Try borgbackup, I'm using it very successfully. It is very fast,
> > supports very impressive deduplication and compression, retention
> > policies, and remote backups - and it is available as a single
> > binary version so you can more easily use it for disaster recovery.
> > One downside: while it seems to restore nocow attributes, it seems
> > to do it in a silly way (it first creates the file, then sets the
> > attributes, which of course won't work for nocow). I have not
> > checked that extensively, only had to restore once yet.
>
> Wow - this looks like the holy grail I've been waiting for, not sure
> how I have missed that up to now.
> Especially the deduplication across several backupped systems on the
> backup target is interesting, I originally planned to do that using
> duperemove on the backup target to dedupe across the readonly
> snapshots. I'll certainly give borgbackup a spin as soon as I have
> time to look into it!
Apparently, only one client can access the repo at the same time. So
you have to fiddle around with backup time windows and maybe the option
for lock timeouts and retries via your service manager. I'm using it to
backup a multi server shop system, it will first create an SQL dump,
then start backing up each host one after the next to a local
repository. In the end, I'll rsync it to an offsite location. It
requires some scripting and ssh foo but it works reliable.
At home, I'm using a simple systemd job to wake my system up at night
and put it back to sleep later:
$ systemctl cat internal-backup.{timer,service}
# /etc/systemd/system/internal-backup.timer
[Unit]
Description=Daily Backup Timer
[Timer]
OnCalendar=03:00
WakeSystem=true
[Install]
WantedBy=timers.target
# /etc/systemd/system/internal-backup.service
[Unit]
Description=Daily Backup Service
[Service]
Environment=BORG_REPO=/mnt/private/backup/jupiter.sol.local.borg
Environment=BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
Environment=BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
Type=oneshot
IOSchedulingClass=idle
IOSchedulingPriority=7
CPUSchedulingPolicy=batch
Nice=3
ProtectHome=read-only
ProtectSystem=full
PrivateTmp=yes
ReadWriteDirectories=/root/.cache/borg
ReadOnlyDirectories=/mnt/btrfs-pool
WorkingDirectory=/mnt/btrfs-pool
ExecStart=/usr/bin/systemd-inhibit /usr/bin/borg create -v --stats
--exclude-caches --compression zlib --exclude
'pp:gentoo/usr/portage/distfiles' --exclude 'pp:gentoo/rootfs/var/tmp'
--exclude
'pp:gentoo/usr/portage/packages' ::'system@{now:%%Y%%m%%d-%%H%%M}' .
ExecStartPost=/usr/bin/borg prune -v --prefix 'system@' --keep-daily 7
--keep-weekly 5 --keep-monthly 12 --list --stats --save-space
The result then looks like this:
$ sudo -H borg
info /mnt/private/backup/jupiter.sol.local.borg::system@20160828-0255
Name: system@20160828-0255 Fingerprint:
786c9c8a18d5d04ba2a745d91777cf30345613b78d2afaa74192db871aa220de
Hostname: jupiter.sol.local Username: root
Time (start): Sun, 2016-08-28 02:55:24
Time (end): Sun, 2016-08-28 03:08:13
Command line: /usr/lib/python-exec/python3.4/borg create -v --stats
--exclude-caches --compression zlib --exclude
pp:gentoo/usr/portage/distfiles --exclude pp:gentoo/rootfs/var/tmp
--exclude pp:gentoo/usr/portage/packages ::system@{now:%Y%m%d-%H%M} .
Number of files: 2356372
Original size Compressed size Deduplicated size
This archive: 1.73 TB 1.49 TB1.02 GB
All archives:17.26 TB 14.94 TB1.78 TB
Unique chunks Total chunks
Chunk index: 2858371 28473400
Initial backup creation is much slower than rsync but successive runs
are much faster (around 15-20 minutes instead of 1 hour).
--
Regards,
Kai
Replies to list-only preferred.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfstune settings
Hi, happy borgbackup user here. This is probably off-topic for most but as many users probably are evaluating send/receive versus other backup solutions, I'll keep linux-btrfs in the loop. On 28/08/2016 20:10, Oliver Freyermuth wrote: >> Try borgbackup, I'm using it very successfully. It is very fast, >> supports very impressive deduplication and compression, retention >> policies, and remote backups - and it is available as a single binary >> version so you can more easily use it for disaster recovery. One >> downside: while it seems to restore nocow attributes, it seems to do it >> in a silly way (it first creates the file, then sets the attributes, >> which of course won't work for nocow). I have not checked that >> extensively, only had to restore once yet. > Wow - this looks like the holy grail I've been waiting for, not sure how > I have missed that up to now. > Especially the deduplication across several backupped systems on the backup > target > is interesting, I originally planned to do that using duperemove on the > backup target > to dedupe across the readonly snapshots. Note that only one backup can happen at a given time on a single repository. You'll probably have to both schedule the backups to avoid collisions and use "--lock-wait" with a large enough parameter to avoid backup failures. There's another twist : borgbackup maintains a local index of the repositories' content, when it detects that it is out of sync (if several systems use the same repository it will) it has to update its index from remote. I'm not sure how heavy this update can be (it seems it uses some form of delta). I have a ~user/.cache/borg of ~2GB for a user with ~9TB of data to backup in ~8M files but I don't share repositories so I'm not affected by this). Best regards, Lionel -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfstune settings
> Try borgbackup, I'm using it very successfully. It is very fast, > supports very impressive deduplication and compression, retention > policies, and remote backups - and it is available as a single binary > version so you can more easily use it for disaster recovery. One > downside: while it seems to restore nocow attributes, it seems to do it > in a silly way (it first creates the file, then sets the attributes, > which of course won't work for nocow). I have not checked that > extensively, only had to restore once yet. Wow - this looks like the holy grail I've been waiting for, not sure how I have missed that up to now. Especially the deduplication across several backupped systems on the backup target is interesting, I originally planned to do that using duperemove on the backup target to dedupe across the readonly snapshots. I'll certainly give borgbackup a spin as soon as I have time to look into it! > It's probably not worth the hassle... Wait until you are forced to > recreate the filesystem or you are having enough spare time to do it. > However, some settings can be changed for existing filesystems and I > would simply ignore the fact existing meta data needs to be converted. Thanks a lot for this hint and also your other points. I think I'll go that way, comparing a "similar" system with skinny metadata and one without the space usage for metadata seems not to be different by a noticeable factor. Since my systems are SSD only, to avoid unnecessary writes I'll then just wait until I really need to replay a backup. Thanks a lot and best regards, Oliver -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re[2]: btrfs check "Couldn't open file system" after error in transaction.c
Hi Chris, thanks for your reply -especially on a Sunday. I have a filesystem (three disks with no raid) So it's data single *and* metadata single? No: Data, single: total=8.14TiB, used=7.64TiB System, RAID1: total=32.00MiB, used=912.00KiB Metadata, RAID1: total=18.00GiB, used=16.45GiB GlobalReserve, single: total=512.00MiB, used=0.00B btrfs check will lead to "Couldn't open file system" Try btrfs-progs all the most recent btrfs-progs to see if it's any different: 4.5.3, 4.6.1, 4.7 (or 4.7.1 if you can get it, it's days old). Ok, I will try. [ 98.534830] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 That's pretty weird. It wants a LOWER generation number than what it found? By quite a bit. It's nearly 1500 generations different. I don't know what can cause this kind of confusion or how to fix it. Ok, time to get the data off it (I do have backups, but of course some weeks old). Answering this question: -If possible repair the filesystem NO Maybe take advantage of the fact it does read only and recreate it. You could take a btrfs-image and btrfs-debug-tree first, And what do I do with it? because there's some bug somewhere: somehow it became inconsistent, and can't be fixed at mount time or even with btrfs check. Ok, so is there any way to help you finding this bug? Coming back to my objectives: -Understand the reason behind the issue and prevent it in future Finding the but would help on the above -If not possible to repair the filesystem: -understand if the data that I read from the drive is valid or corrupted Can you answer this? As mentioned: I do have a backup, a month old. The data does not change so regularly, so most should be ok. Now I have two sources of data: the backup and the current degraded filesystem. If data differs, which one do I take? Is it safe to use the more recent one from the degraded filesystem? Greetings, Hendrik -- Chris Murphy --- Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft. https://www.avast.com/antivirus -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfstune settings
Am Sun, 28 Aug 2016 18:18:52 +0200 schrieb Oliver Freyermuth : > That would certainly be the best option, however, > I have two issues with that: > - Any replay of a backup will do a lot of writes on the SSD, > reducing lifetime. I'm using bcache in front of my harddisks (writeback mode) to get almost-SSD performance. That way, when I restore, I can simply switch to writearound mode, restore, then switch back to writeback mode. But in the long term, writeback mode will be extremely heavy on SSD wearing when the bcache is small because it will have to evict a lot of data from the cache. I'm using 500GB bcache for 3x 1TB harddisk. 500GB is more than my usual working set so bcache almost never has to evict data from the cache. -- Regards, Kai Replies to list-only preferred. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: does btrfs-receive use/compare the checksums from the btrfs-send side?
On Sun, 2016-08-28 at 11:35 -0600, Chris Murphy wrote: > I don't see evidence of them in the btrfs send file, so I don't think > csums are in the stream. hmm... isn't that kinda unfortunate not to make use of the information that's already there? IMO, to the extent this is possibly, btrfs should generally re-use csums (or compare freshly created ones with the ones already known) on copy-like operations. Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: btrfstune settings
Am Sun, 28 Aug 2016 18:18:52 +0200 schrieb Oliver Freyermuth : > (sorry if my Message-ID header is missing, I am not subscribed to the > mailing list, so I reply using mail-archive) > > > Try btrfs-show-super . The incompat_flags section > > enumerates the flags that are on (at least with a reasonably new > > btrfs-progs). > > Thanks a lot for this hint - I totally missed that, since I only > looked at manpages and features of the "all-in-one" btrfs tool up to > now. Also thanks for the hint to /sys/fs/btrfs, this is even more > readable! > > And thanks a lot for your detailed answer(s) in general! > > > In general it should be safe to activate flags via btrfstune, but > > since they'll generally apply only to newly written data, any > > benefits on a mature filesystem will be limited. For that reason > > as well as to get the benefit of 16K nodesize which you can't > > except at creation, I recommend backing up and recreating the > > filesystem from a fresh mkfs.btrfs. > > That would certainly be the best option, however, > I have two issues with that: > - Any replay of a backup will do a lot of writes on the SSD, > reducing lifetime. > - I did not yet figure out a working way to get a "complete" backup > for a btrfs. btrfs send is too slow for me, and rsync does an > incomplete backup, since all FS-special attrs (like the NOCOW attr > from 'chattr +C') are not copied, even when xattr copy is on. Try borgbackup, I'm using it very successfully. It is very fast, supports very impressive deduplication and compression, retention policies, and remote backups - and it is available as a single binary version so you can more easily use it for disaster recovery. One downside: while it seems to restore nocow attributes, it seems to do it in a silly way (it first creates the file, then sets the attributes, which of course won't work for nocow). I have not checked that extensively, only had to restore once yet. > My last backup replay was from an rsync backup (these I do regularly) > and afterwards I did a manual fixing of these special attrs, which is > some bit of extra work. Also, the machine is offline during backup > replay, while I could still use it during a simple rebalance. It's hard to circumvent this. I had once the idea to rsync backups to a btrfs device, then during recovery use it as a seeding device. I never tried it but it could work. Your system could be almost immediately back online this way but you'll have to do some special extra steps to get the seeding device back into a state which you can use again for your backups. > The good news is my old FS already has 16K leafsize, it's only > missing skinny metadata (and no-holes, apparently that's not default > in mkfs.btrfs yet?). AFAIR no-holes and skinny-metadata can be enabled on existing filesystems... > This reduces the scope of my question a bit - is it sufficient and > worthwhile to "btrfstune -x" and then "btrfs balance start > -musage=0 /" to convert all metadata, and thus gain some space? Not sure... Especially about no-holes. You could also try to "defragment" meta data using $ find /mnt/btrfs-pool -type d -print0 | xargs -0 btrfs fi defrag > Or are the gains not worth it and I should just wait until the next > time I need to replay a backup for other reasons? It's probably not worth the hassle... Wait until you are forced to recreate the filesystem or you are having enough spare time to do it. However, some settings can be changed for existing filesystems and I would simply ignore the fact existing meta data needs to be converted. -- Regards, Kai Replies to list-only preferred. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: does btrfs-receive use/compare the checksums from the btrfs-send side?
On Sat, Aug 27, 2016 at 9:46 PM, Christoph Anton Mitterer wrote: > Hey. > > I've often wondered: > When I do a send/receive, does the receiving side use the checksums > from the sending side (either by directly storing them or by comparing > them with calculated checksums and failing if they don't match after > the transfer)? > > Cause that would effectively secure any transport in between against > transmission errors ... and also against badblocks/etc. on the > receiving fs. > > If btrfs is already that smart, then I think this feature should be > mentioned in the send/receive manpages. I don't see evidence of them in the btrfs send file, so I don't think csums are in the stream. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check "Couldn't open file system" after error in transaction.c
On Sun, Aug 28, 2016 at 4:04 AM, Hendrik Friedel wrote: > Hello, > > I have a filesystem (three disks with no raid) So it's data single *and* metadata single? > btrfs check will lead to "Couldn't open file system" Try btrfs-progs all the most recent btrfs-progs to see if it's any different: 4.5.3, 4.6.1, 4.7 (or 4.7.1 if you can get it, it's days old). > > > [ 98.534830] BTRFS error (device sde): parent transid verify failed on > 22168481054720 wanted 1826943 found 1828546 That's pretty weird. It wants a LOWER generation number than what it found? By quite a bit. It's nearly 1500 generations different. I don't know what can cause this kind of confusion or how to fix it. Maybe take advantage of the fact it does read only and recreate it. You could take a btrfs-image and btrfs-debug-tree first, because there's some bug somewhere: somehow it became inconsistent, and can't be fixed at mount time or even with btrfs check. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs check "Couldn't open file system" after error in transaction.c
Hello, some more info: The system is Debian jessie with kernel 4.6.0 and btrfs-tools 4.6. I did go through the recovery steps from the wiki: -btrfs scrub to detect issues on live filesystems see my original mail. Is aborted immediately -look at btrfs detected errors in syslog (look at Marc's blog above on how to use sec.pl to do this) see my original mail -mount -o ro,recovery to mount a filesystem with issues does work, but I get many errors like this one [ 325.360115] BTRFS info (device sdd): no csum found for inode 1703 start 2072977408 -btrfs-zero-log might help in specific cases. Go read Btrfs-zero-log I would like to get your ok and instructions on this first -btrfs restore will help you copy data off a broken btrfs filesystem. See its page: Restore see above. Recovering the data does seem to work with ro,recovery. By the way: can I be sure somehow that the Data is correct when I read it this way, despite the "no csum found for inode" ? -btrfs check --repair, aka btrfsck is your last option if the ones above have not worked. Does not work: "Couldn't open file system" I also went through Marcs page: http://marc.merlins.org/perso/btrfs/post_2014-03-19_Btrfs-Tips_-Btrfs-Scrub-and-Btrfs-Filesystem-Repair.html but without further hints. So I have now these objectives: -If possible repair the filesystem -Understand the reason behind the issue and prevent it in future -If not possible to repair the filesystem: -understand if the data that I read from the drive is valid or corrupted I'd appreciate your help on this. Greetings, Hendrik -- Originalnachricht -- Von: "Hendrik Friedel" An: "Btrfs BTRFS" Gesendet: 28.08.2016 12:04:18 Betreff: btrfs check "Couldn't open file system" after error in transaction.c Hello, I have a filesystem (three disks with no raid) that I can still mount ro, but I cannot check or scrub it. In dmesg I see: [So Aug 28 11:33:22 2016] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [So Aug 28 11:33:22 2016] BTRFS warning (device sde): Skipping commit of aborted transaction. (more complete at the end of this mail) What I did up to now in order to recover: - mount ro,recovery (http://marc.merlins.org/perso/btrfs/post_2014-03-19_Btrfs-Tips_-Btrfs-Scrub-and-Btrfs-Filesystem-Repair.html) that works. btrfs check will lead to "Couldn't open file system" root@homeserver:~# btrfs scrub start /mnt/test scrub started on /mnt/test, fsid a8af3832-48c7-4568-861f-e80380dd7e0b (pid=18953) root@homeserver:~# btrfs scrub status /mnt/test scrub status for a8af3832-48c7-4568-861f-e80380dd7e0b scrub started at Sun Aug 28 12:02:46 2016 and was aborted after 00:00:00 total bytes scrubbed: 0.00B with 0 errors First thing to do now is probably to check the backups. But is there a way to repair this filesystem? Besides this: What could be the reason for this error? Scrubs were regular and good. There was no power loss and also smartctl looks fine on all three drives. Greetings, Hendrik root@homeserver:~# btrfs fi show Label: 'BigStorage' uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 7.66TiB devid1 size 2.73TiB used 2.72TiB path /dev/sde devid2 size 2.73TiB used 2.72TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd [ 98.534830] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [ 98.534866] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [ 98.534891] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [ 98.534920] BTRFS warning (device sde): Skipping commit of aborted transaction. [ 98.534921] [ cut here ] [ 98.534939] WARNING: CPU: 1 PID: 3643 at /home/zumbi/linux-4.6.4/fs/btrfs/transaction.c:1771 cleanup_transaction+0x96/0x300 [btrfs] [ 98.534940] BTRFS: Transaction aborted (error -5) [ 98.534940] Modules linked in: xt_nat(E) xt_tcpudp(E) veth(E) ftdi_sio(E) usbserial(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) xfrm_user(E) xfrm_algo(E) iptable_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) nf_nat_ipv4(E) xt_addrtype(E) iptable_filter(E) ip_tables(E) xt_conntrack(E) x_tables(E) nf_nat(E) nf_conntrack(E) br_netfilter(E) bridge(E) stp(E) llc(E) cpufreq_stats(E) cpufreq_userspace(E) cpufreq_conservative(E) cpufreq_powersave(E) binfmt_misc(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) fscache(E) sunrpc(E) snd_hda_codec_hdmi(E) iTCO_wdt(E) iTCO_vendor_support(E) stv6110x(E) lnbp21(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) cryptd(E) pcspkr(E) serio_raw(E) [ 98.534963] snd_hda_codec_realtek(E) snd_hda_codec_generic(E) i2c_i801(E) i915(E) stv090x(E)
Re: Recommendation on raid5 drive error resolution
On Thu, Aug 25, 2016 at 1:23 AM, Gareth Pye wrote: > So I've been living on the reckless-side (meta RAID6, data RAID5) and > I have a drive or two that isn't playing nicely any more. > > dmesg of the system running for a few minutes: http://pastebin.com/9pHBRQVe > > Everything of value is backed up, but I'd rather keep data than > download it all again. When I only saw one disk having troubles I was > concerned. Now I notice both sda and sdc having issues I'm thinking I > might be about to have a bad time. > > What else should I provide? [ 72.555921] BTRFS info (device sda7): bdev /dev/sdc errs: wr 0, rd 9091, flush 0, corrupt 0, gen 0 [ 72.555941] BTRFS info (device sda7): bdev /dev/sdh errs: wr 0, rd 74, flush 0, corrupt 0, gen 0 Two devices with read errors, bad. If they overlap, it's basically a dead raid5. And it also means you *CANNOT* remove either drive. So now you have a problem, and I highly advise that you fresh your backup because this is a really fragile state for any raid5. What's the result from these two commands for every drive in this array? smarctl -l scterc cat /sys/block/sdX/device/timeout The SCTERC value must be less than the timeout. This really must be the first thing you do, even before starting your backup, because otherwise a misconfiguration here has a very good chance of preventing the success of getting a backup. Note these are not persistent settings. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Is there a performance advantage in using Btrfs subvolumes over raw (MBR/GPT) partitions on rotational disks?
On Sun, Aug 28, 2016 at 5:17 AM, Alexander Tomokhov wrote: > Let's say I use a 128GiB MBR partition for OS and remaining 800GiB+ on disk > serves another purpose. Then all OS files will be physically close to each > other on the disk, thus reducing seek times. > If I use the whole disk for OS partition, then files will be spread across > the disk and even OS-only operations will require hard drive heads to perform > longer seeks. > > What effect have Btrfs subvolumes in this case if I use them instead of MBR > partitions? Writes to subvolumes doesn't directly affect where on disk the data is written, those data extents will be put into a block group by the allocator regardless of what subvolume the data originated from. > Btrfs wiki says: "btrfs does write optimisation (sequential writes) across a > filesystem", "subvolume write performance will benefit from this algorithm". I'm pretty sure this means it's aggregating random writes such that they end up being more like sequential writes, and is related to delayed allocation. It may also be referring to the use of subvolumes instead of directories for multiple write streams has better concurrency handling and less contention. -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfstune settings
(sorry if my Message-ID header is missing, I am not subscribed to the mailing list, so I reply using mail-archive) > Try btrfs-show-super . The incompat_flags section enumerates the > flags that are on (at least with a reasonably new btrfs-progs). Thanks a lot for this hint - I totally missed that, since I only looked at manpages and features of the "all-in-one" btrfs tool up to now. Also thanks for the hint to /sys/fs/btrfs, this is even more readable! And thanks a lot for your detailed answer(s) in general! > In general it should be safe to activate flags via btrfstune, but since > they'll generally apply only to newly written data, any benefits on a > mature filesystem will be limited. For that reason as well as to get the > benefit of 16K nodesize which you can't except at creation, I recommend > backing up and recreating the filesystem from a fresh mkfs.btrfs. That would certainly be the best option, however, I have two issues with that: - Any replay of a backup will do a lot of writes on the SSD, reducing lifetime. - I did not yet figure out a working way to get a "complete" backup for a btrfs. btrfs send is too slow for me, and rsync does an incomplete backup, since all FS-special attrs (like the NOCOW attr from 'chattr +C') are not copied, even when xattr copy is on. My last backup replay was from an rsync backup (these I do regularly) and afterwards I did a manual fixing of these special attrs, which is some bit of extra work. Also, the machine is offline during backup replay, while I could still use it during a simple rebalance. The good news is my old FS already has 16K leafsize, it's only missing skinny metadata (and no-holes, apparently that's not default in mkfs.btrfs yet?). This reduces the scope of my question a bit - is it sufficient and worthwhile to "btrfstune -x" and then "btrfs balance start -musage=0 /" to convert all metadata, and thus gain some space? Or are the gains not worth it and I should just wait until the next time I need to replay a backup for other reasons? Cheers and thanks for your reply, Oliver -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send extremely slow (almost stuck)
(sorry if my Message-ID header is missing, I am not subscribed to the mailing list, so I reply using mail-archive) > So a workaround would be reducing your duperemove usage and possibly > rewriting (for instance via defrag) the deduped files to kill the > multiple reflinks. Or simply delete the additional reflinked copies, if > your use-case allows it. Sadly, I need the extra space (that's why I was using duperemove in the first place) and can not delete all duped copies. These are mainly several checkouts of different repositories with partially common (partially large binary) content. > And thin down your snapshot retention if you have many snapshots per > subvolume. With the geometric scaling issues, thinning to under 300 per > subvolume should be quite reasonable in nearly all circumstances, and > thinning to under 100 per subvolume may be possible and should result in > dramatically reduced scaling issues. In addition, I have only ~ 5 snapshots for both those volumes, which should certainly not be too much. So in short, this just means btrfs send is (still) unusable for filesystems which rely on the offline dedupe feature (in the past 'btrfs send' got broken after dedupe which got fixed, now it is just extremely slow). For me, this means I have to stay with rsync backups, which are sadly incomplete since special FS attrs like "C" for nocow are not backed up. Cheers and thanks for your reply, Oliver -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfstune settings
Am Sun, 28 Aug 2016 08:27:59 + (UTC) schrieb Duncan <1i5t5.dun...@cox.net>: > > However, I miss two things: > > - How do I see on an existing FS which of these features are on? > > btrfstune (it seems) can only "set", but not "get" the feature > > flags. > > Try btrfs-show-super . The incompat_flags section enumerates > the flags that are on (at least with a reasonably new btrfs-progs). It's also in the sub directories of /sys/fs/btrfs... -- Regards, Kai Replies to list-only preferred. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[4.7.2] btrfs_run_delayed_refs:2963: errno=-17 Object already exists
Hello list! It happened again. While using VirtualBox the following crash happened, btrfs check found a lot of errors which it couldn't repair. Earlier that day my system crashed which may already introduced errors into my filesystem. Apparently, I couldn't create an image (not enough space available), I only can give this trace from dmesg: [44819.903435] [ cut here ] [44819.903443] WARNING: CPU: 3 PID: 2787 at fs/btrfs/extent-tree.c:2963 btrfs_run_delayed_refs+0x26c/0x290 [44819.903444] BTRFS: Transaction aborted (error -17) [44819.903445] Modules linked in: nls_iso8859_15 nls_cp437 vfat fat fuse rfcomm veth af_packet ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge stp llc w83627ehf bnep hwmon_vid cachefiles btusb btintel bluetooth snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec rfkill snd_hwdep snd_hda_core snd_pcm snd_timer coretemp hwmon snd r8169 mii kvm_intel kvm iTCO_wdt iTCO_vendor_support rtc_cmos irqbypass soundcore ip_tables uas usb_storage nvidia_drm(PO) vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) nvidia_modeset(PO) nvidia(PO) efivarfs unix ipv6 [44819.903484] CPU: 3 PID: 2787 Comm: BrowserBlocking Tainted: P O 4.7.2-gentoo #2 [44819.903485] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z68 Pro3, BIOS L2.16A 02/22/2013 [44819.903487] 8130af2d 8800b7d03d20 [44819.903489] 810865fa 880409374428 8800b7d03d70 8803bf299760 [44819.903491] ffef 8803f677f000 8108666a [44819.903493] Call Trace: [44819.903496] [] ? dump_stack+0x46/0x59 [44819.903500] [] ? __warn+0xba/0xe0 [44819.903502] [] ? warn_slowpath_fmt+0x4a/0x50 [44819.903504] [] ? btrfs_run_delayed_refs+0x26c/0x290 [44819.903507] [] ? btrfs_release_path+0xe/0x80 [44819.903509] [] ? btrfs_start_dirty_block_groups+0x2da/0x420 [44819.903511] [] ? btrfs_commit_transaction+0x143/0x990 [44819.903514] [] ? kmem_cache_free+0x165/0x180 [44819.903516] [] ? btrfs_wait_ordered_range+0x7c/0x110 [44819.903518] [] ? btrfs_sync_file+0x286/0x360 [44819.903522] [] ? do_fsync+0x33/0x60 [44819.903524] [] ? SyS_fdatasync+0xa/0x10 [44819.903528] [] ? entry_SYSCALL_64_fastpath+0x13/0x8f [44819.903529] ---[ end trace 6944811e170a0e57 ]--- [44819.903531] BTRFS: error (device bcache2) in btrfs_run_delayed_refs:2963: errno=-17 Object already exists [44819.903533] BTRFS info (device bcache2): forced readonly Since I had to get back up and running fast, I restored from backup. I now bought some extra 3TB backup space and created a rescue system including all tools on a USB stick, so next time it happens I may be able to create an image of the broken filesystem. Btrfs --repair refused to repair the filesystem telling me something about compressed extents and an unsupported case, wanting me to take an image and send it to the devs. *sigh* System is kernel 4.7.2, Gentoo Linux, latest VirtualBox stable. VirtualBox was using VDI image format without nocow. I now reverted back to using nocow on VDI files and hope it doesn't strike again too soon. I didn't try again yet, first I need to refresh my backup which takes a while. The filesystem runs on 3x SATA 1TB mraid1 draid0 through bcache in writeback mode, backed by a 500GB 850 Evo - if that matters. The problem occurred during high IO on 4.7.2. I previously ran 4.6.6 which didn't show this problem. Part of the culprit may be that I was using bfq patches - I removed them for now and went back to deadline io scheduler. The bfq patches froze my system a few times when I booted 4.7.2 which may already have broken my btrfs (although it shouldn't, right? btrfs is transactional). Last time this happened (on an earlier kernel), bfq may have been part of the problem, too. So I think bfq does something to btrfs which may break the fs, or at least interferes badly with the transaction as otherwise it shouldn't break. You may want to run your test suites with bfq also (or different io schedulers in general). My home partition is mounted as a subvolume: /dev/bcache0 on /home type btrfs (rw,noatime,compress=lzo,nossd,space_cache,autodefrag,subvolid=261,subvol=/home) -- Regards, Kai Replies to list-only preferred. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Is there a performance advantage in using Btrfs subvolumes over raw (MBR/GPT) partitions on rotational disks?
Let's say I use a 128GiB MBR partition for OS and remaining 800GiB+ on disk serves another purpose. Then all OS files will be physically close to each other on the disk, thus reducing seek times. If I use the whole disk for OS partition, then files will be spread across the disk and even OS-only operations will require hard drive heads to perform longer seeks. What effect have Btrfs subvolumes in this case if I use them instead of MBR partitions? Btrfs wiki says: "btrfs does write optimisation (sequential writes) across a filesystem", "subvolume write performance will benefit from this algorithm". I don't understand what it means and couldn't find explanation. Does Btrfs optimize file data placement on rotational disk inside single subvolume, so that storing similar (accessed together) data inside single subvolume will benefit in terms of performance? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs check "Couldn't open file system" after error in transaction.c
Hello, I have a filesystem (three disks with no raid) that I can still mount ro, but I cannot check or scrub it. In dmesg I see: [So Aug 28 11:33:22 2016] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [So Aug 28 11:33:22 2016] BTRFS warning (device sde): Skipping commit of aborted transaction. (more complete at the end of this mail) What I did up to now in order to recover: - mount ro,recovery (http://marc.merlins.org/perso/btrfs/post_2014-03-19_Btrfs-Tips_-Btrfs-Scrub-and-Btrfs-Filesystem-Repair.html) that works. btrfs check will lead to "Couldn't open file system" root@homeserver:~# btrfs scrub start /mnt/test scrub started on /mnt/test, fsid a8af3832-48c7-4568-861f-e80380dd7e0b (pid=18953) root@homeserver:~# btrfs scrub status /mnt/test scrub status for a8af3832-48c7-4568-861f-e80380dd7e0b scrub started at Sun Aug 28 12:02:46 2016 and was aborted after 00:00:00 total bytes scrubbed: 0.00B with 0 errors First thing to do now is probably to check the backups. But is there a way to repair this filesystem? Besides this: What could be the reason for this error? Scrubs were regular and good. There was no power loss and also smartctl looks fine on all three drives. Greetings, Hendrik root@homeserver:~# btrfs fi show Label: 'BigStorage' uuid: a8af3832-48c7-4568-861f-e80380dd7e0b Total devices 3 FS bytes used 7.66TiB devid1 size 2.73TiB used 2.72TiB path /dev/sde devid2 size 2.73TiB used 2.72TiB path /dev/sdc devid3 size 2.73TiB used 2.73TiB path /dev/sdd [ 98.534830] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [ 98.534866] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [ 98.534891] BTRFS error (device sde): parent transid verify failed on 22168481054720 wanted 1826943 found 1828546 [ 98.534920] BTRFS warning (device sde): Skipping commit of aborted transaction. [ 98.534921] [ cut here ] [ 98.534939] WARNING: CPU: 1 PID: 3643 at /home/zumbi/linux-4.6.4/fs/btrfs/transaction.c:1771 cleanup_transaction+0x96/0x300 [btrfs] [ 98.534940] BTRFS: Transaction aborted (error -5) [ 98.534940] Modules linked in: xt_nat(E) xt_tcpudp(E) veth(E) ftdi_sio(E) usbserial(E) ipt_MASQUERADE(E) nf_nat_masquerade_ipv4(E) xfrm_user(E) xfrm_algo(E) iptable_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) nf_nat_ipv4(E) xt_addrtype(E) iptable_filter(E) ip_tables(E) xt_conntrack(E) x_tables(E) nf_nat(E) nf_conntrack(E) br_netfilter(E) bridge(E) stp(E) llc(E) cpufreq_stats(E) cpufreq_userspace(E) cpufreq_conservative(E) cpufreq_powersave(E) binfmt_misc(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) nfs(E) lockd(E) grace(E) fscache(E) sunrpc(E) snd_hda_codec_hdmi(E) iTCO_wdt(E) iTCO_vendor_support(E) stv6110x(E) lnbp21(E) intel_rapl(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) cryptd(E) pcspkr(E) serio_raw(E) [ 98.534963] snd_hda_codec_realtek(E) snd_hda_codec_generic(E) i2c_i801(E) i915(E) stv090x(E) snd_hda_intel(E) snd_hda_codec(E) snd_hda_core(E) snd_hwdep(E) snd_pcm(E) drm_kms_helper(E) ngene(E) ddbridge(E) snd_timer(E) snd(E) lpc_ich(E) mei_me(E) dvb_core(E) mfd_core(E) drm(E) soundcore(E) mei(E) i2c_algo_bit(E) shpchp(E) evdev(E) battery(E) tpm_tis(E) video(E) tpm(E) processor(E) button(E) fuse(E) autofs4(E) btrfs(E) xor(E) raid6_pq(E) dm_mod(E) md_mod(E) hid_generic(E) usbhid(E) hid(E) sg(E) sd_mod(E) ahci(E) libahci(E) crc32c_intel(E) libata(E) psmouse(E) scsi_mod(E) fan(E) thermal(E) xhci_pci(E) xhci_hcd(E) fjes(E) e1000e(E) ptp(E) pps_core(E) ehci_pci(E) ehci_hcd(E) usbcore(E) usb_common(E) [ 98.534988] CPU: 1 PID: 3643 Comm: btrfs-transacti Tainted: G E 4.6.0-0.bpo.1-amd64 #1 Debian 4.6.4-1~bpo8+1 [ 98.534989] Hardware name: /DH87RL, BIOS RLH8710H.86A.0325.2014.0417.1800 04/17/2014 [ 98.534990] 0286 007b2061 813124c5 8804a15f3d40 [ 98.534992] 8107af94 8804e884ac10 8804a15f3d98 [ 98.534993] 8804b9bbc500 fffb 8804e884ac10 fffb [ 98.534995] Call Trace: [ 98.535000] [] ? dump_stack+0x5c/0x77 [ 98.535003] [] ? __warn+0xc4/0xe0 [ 98.535005] [] ? warn_slowpath_fmt+0x5f/0x80 [ 98.535014] [] ? cleanup_transaction+0x96/0x300 [btrfs] [ 98.535017] [] ? wait_woken+0x90/0x90 [ 98.535026] [] ? btrfs_commit_transaction+0x2b3/0xa30 [btrfs] [ 98.535028] [] ? wait_woken+0x90/0x90 [ 98.535036] [] ? transaction_kthread+0x1ce/0x1f0 [btrfs] [ 98.535043] [] ? btrfs_cleanup_transaction+0x590/0x590 [btrfs] [ 98.535045] [] ? kthread+0xdf/0x100 [ 98.535048] [] ? ret_from_fork+0x22/0x40 [ 98.535049] [] ? kthread_park+0x50/0x50 [ 98.535050] ---[ end trace 91a2f
Re: btrfstune settings
Oliver Freyermuth posted on Sun, 28 Aug 2016 05:42:39 +0200 as excerpted: > Dear btrfs experts, > > I hope this is the correct place to ask, the wiki and manpages did not > help me on these questions. > > BTRFS has gained extended inode refs, skinny metadata and no-holes quite > a while ago and these are now the defaults for new mkfs.btrfs. btrfstune > can activate those features. > > However, I miss two things: > - How do I see on an existing FS which of these features are on? > btrfstune (it seems) can only "set", but not "get" the feature flags. Try btrfs-show-super . The incompat_flags section enumerates the flags that are on (at least with a reasonably new btrfs-progs). > - Is it worthwhile / recommended / safe to activate those on existing > FS? > Are there any steps (e.g. balancing metadata with -musage=0, I'd > guess) needed to make them become active afterwards? If they're intended to be changed on an existing filesystem, btrfstune should allow it, otherwise it doesn't. Node/leaf-size used to default to sectorsize (4K on x86/amd64), for instance, while now defaulting to 16K, but can't be changed on an existing filesystem, only at mkfs time, so that's the only place with the option. In general it should be safe to activate flags via btrfstune, but since they'll generally apply only to newly written data, any benefits on a mature filesystem will be limited. For that reason as well as to get the benefit of 16K nodesize which you can't except at creation, I recommend backing up and recreating the filesystem from a fresh mkfs.btrfs. Since btrfs is considered still stabilizing, having a backup is very strongly recommended in any case unless the value of the data simply isn't worth the hassle, and once you have a backup tested, available and ready to use should it be necessary, blowing the existing filesystem away and starting with a fresh filesystem created with the options you want isn't much more difficult and may well be faster than a full balance anyway. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Crush on btrfs device delete missing
Hi Duncan. Sorry, I was careless. (And sorry illrelevant To header.) System is a Kernel 4.7.2 Manjaro Linux default setting. === $ uname -r 4.7.2-1-MANJARO === and btrfs progs. === $ pacman -Q btrfs-progs btrfs-progs 4.7-1 === result of btrfs fi usage /path/to/volume === Overall: Device size: 19.10TiB Device allocated: 10.71TiB Device unallocated:8.39TiB Device missing:2.73TiB Used: 10.71TiB Free (estimated): 4.19TiB (min: 4.19TiB) Data ratio: 2.00 Metadata ratio: 1.99 Global reserve: 512.00MiB (used: 0.00B) Data,RAID1: Size:5.34TiB, Used:5.34TiB /dev/dm-1 1.78TiB /dev/dm-2 1.78TiB /dev/dm-3 1.78TiB /dev/dm-4 1.78TiB /dev/dm-5 1.78TiB /dev/mapper/hymaster_6 2.00GiB missing 1.78TiB Metadata,single: Size:112.00MiB, Used:0.00B /dev/dm-1 16.00MiB /dev/dm-2 16.00MiB /dev/dm-3 16.00MiB /dev/dm-4 16.00MiB /dev/dm-5 24.00MiB missing24.00MiB Metadata,RAID1: Size:14.00GiB, Used:12.40GiB /dev/dm-1 3.00GiB /dev/dm-2 3.00GiB /dev/dm-3 6.00GiB /dev/dm-4 5.00GiB /dev/dm-5 5.00GiB missing 6.00GiB System,RAID1: Size:32.00MiB, Used:784.00KiB /dev/dm-1 32.00MiB /dev/dm-2 32.00MiB Unallocated: /dev/dm-1 966.47GiB /dev/dm-2 965.47GiB /dev/dm-3 966.50GiB /dev/dm-4 965.50GiB /dev/dm-5 965.50GiB /dev/mapper/hymaster_6 2.73TiB missing 967.50GiB === Result of btrfs filesystem show /path/to/volume === Label: 'HyMasterBtr' uuid: c78a98fb-4718-4f30-abae-466ebf6e2d8f Total devices 7 FS bytes used 5.36TiB devid1 size 2.73TiB used 1.79TiB path /dev/mapper/hymaster_1 devid2 size 2.73TiB used 1.79TiB path /dev/mapper/hymaster_2 devid4 size 2.73TiB used 1.79TiB path /dev/mapper/hymaster_3 devid5 size 2.73TiB used 1.79TiB path /dev/mapper/hymaster_4 devid6 size 2.73TiB used 1.79TiB path /dev/mapper/hymaster_5 devid7 size 2.73TiB used 2.00GiB path /dev/mapper/hymaster_6 *** Some devices missing === Result of btrfs filesystem df === Data, RAID1: total=5.34TiB, used=5.34TiB System, RAID1: total=32.00MiB, used=784.00KiB Metadata, RAID1: total=14.00GiB, used=12.40GiB Metadata, single: total=112.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B === Thank you. On Sun, 28 Aug 2016 08:06:47 + (UTC) Duncan <1i5t5.dun...@cox.net> wrote: > MASAKI Yuhsuke posted on Sun, 28 Aug 2016 14:22:52 +0900 as excerpted: > > > btrfs device delete missing always crushes and stop system. > > It log console only not STDOUT/STDERR or core file. > > While I'm unlikely to be able to help with the problem, for those who > can, you're missing some critical information, including kernel and > btrfs- progs versions (if they're old, please try a current 4.6 or > 4.7 kernel if on current, or 4.4 if on LTS, and an appropriately > matched or newer btrfs userspace as well), and btrfs filesystem usage > or btrfs filesystem show and btrfs filesystem df outputs. > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Crush on btrfs device delete missing
MASAKI Yuhsuke posted on Sun, 28 Aug 2016 14:22:52 +0900 as excerpted: > btrfs device delete missing always crushes and stop system. > It log console only not STDOUT/STDERR or core file. While I'm unlikely to be able to help with the problem, for those who can, you're missing some critical information, including kernel and btrfs- progs versions (if they're old, please try a current 4.6 or 4.7 kernel if on current, or 4.4 if on LTS, and an appropriately matched or newer btrfs userspace as well), and btrfs filesystem usage or btrfs filesystem show and btrfs filesystem df outputs. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send extremely slow (almost stuck)
Oliver Freyermuth posted on Sun, 28 Aug 2016 05:38:00 +0200 as excerpted: > Dear btrfs experts, > > I just tried to make use of btrfs send / receive for incremental backups > (using btrbk to simplify the process). > It seems that on my two machines, btrfs send gets stuck after > transferring some GiB - it's not fully halted, but instead of making > full use of the available I/O, I get something < 500 kiB on average, > which are just some "full speed spikes" with many seconds / minutes of > no I/O in between. > > During this "halting", btrfs send eats one full CPU core. > A "perf top" shows this is spent in "find_parent_nodes" and > "__merge_refs" inside the kernel. > I am using btrfs-progs 4.7 and kernel 4.7.0. > > I googled a bit and found related patchwork > (https://patchwork.kernel.org/patch/9238987/) which seems to workaround > high load in this area and mentions a real solution is proposed but not > yet there. > > Since this affects two machines of mine and backupping my root volume > would take about 80 hours in case I can extrapolate the average rate, > this means btrfs send is unusable to me. > > Can I assume this is a common issue which will be fixed in a later > kernel release (4.8, 4.9) or can I do something to my FS's to workaround > this issue? > > One FS is only two weeks old, the other one now about 1 year. I did some > balancing at some points of time to have more unallocated space for > trimming, > and used duperemove regularly to free space. One FS has skinny extents, > the other has not. The problem is as the patch says, multiple references per extent increases process time geometrically. And dupremove works by doing just that, pointing multiple duplications to the same extents, increasing the reference count per extent, thereby exacerbating the problem on your system, if dupremove is actually finding a reasonable number of duplicates to reflink to the same extents. The other common multi-reflink usage is snapshots, since each snapshot creates another reflink to each extent it snapshots. However, being just a list regular and btrfs user, not a dev, and using neither dedupe nor snapshots nor send/receive in my own use-case, I'm not absolutely sure whether other snapshot references affect send/receive or whether it's only multiple reflinks per sent snapshot. Either way, over a few hundred snapshots per subvolume or a couple thousand snapshots per filesystem, they do seriously affect scaling of balance and fsck, even if they don't actually affect send/receive so badly. So a workaround would be reducing your duperemove usage and possibly rewriting (for instance via defrag) the deduped files to kill the multiple reflinks. Or simply delete the additional reflinked copies, if your use-case allows it. And thin down your snapshot retention if you have many snapshots per subvolume. With the geometric scaling issues, thinning to under 300 per subvolume should be quite reasonable in nearly all circumstances, and thinning to under 100 per subvolume may be possible and should result in dramatically reduced scaling issues. Note that the current patch doesn't really workaround the geometric scaling issues or extreme cpu usage bottlenecking send/receive, but rather, addresses the soft lockups problem due to not scheduling often enough to give other threads time to process. You didn't mention problems with soft lockups, so it's likely to be of limited help for the send/receive problem. As for the longer term, yes, it should be fixed, eventually, but keep in mind that btrfs isn't considered fully stable and mature yet, so this sort of problem isn't unexpected and indeed scaling issues like this are known to still be an issue, and while I haven't been tracking that red/ black tree work, in general it can be noted that btrfs fixes for this sort of problem often take rather longer than might be expected, so a fix may be more like a year or two out than a kernel cycle or two out. Unless of course you see otherwise from someone working on this problem specifically, and even then, sometimes the first fix doesn't get it quite right, and the problem may remain for some time as more is learned about the ultimate issue via multiple attempts to fix it. This has happened to the quota code a number of times for instance, as it as turned out to be a /really/ hard problem, with multiple rewrites necessary, such that even now, the practical recommendation is often to either just turn off quotas and not worry about them if you don't need them, or use a more mature filesystem where the quota code is known to be stable and mature, if your use-case depends on them. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majord
Re: Recommendation on raid5 drive error resolution
Hi Gareth, I'm interested in how you go with this as I'm somewhat similar with RAID5 with both. Don't take this as advice as I have never done it; however if I were in your shoes, I would take out one of the disks that isn't playing nicely and rebuild the array. Once it is running smooth then I would take the other disk that isn't playing nice and replace it and rebuild again. The whole process will take a fair bit of time but better to be safe than sorry. Like I said I have never done it so do so at your own risk. DanglingPointer On 25/08/16 17:23, Gareth Pye wrote: So I've been living on the reckless-side (meta RAID6, data RAID5) and I have a drive or two that isn't playing nicely any more. dmesg of the system running for a few minutes: http://pastebin.com/9pHBRQVe Everything of value is backed up, but I'd rather keep data than download it all again. When I only saw one disk having troubles I was concerned. Now I notice both sda and sdc having issues I'm thinking I might be about to have a bad time. What else should I provide? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
