[PATCH] Btrfs: incremental send, apply asynchronous page cache readahead

2017-09-12 Thread peterh
From: Kuanling Huang By analyzing the perf on btrfs send, we found it take large amount of cpu time on page_cache_sync_readahead. This effort can be reduced after switching to asynchronous one. Overall performance gain on HDD and SSD were 9 and 15 respectively if simply send a large file. Signed

Re: [PATCH] btrfs: prevent to set invalid default subvolid

2017-09-12 Thread Qu Wenruo
On 2017年09月12日 21:42, Satoru Takeuchi wrote: `btrfs sub set-default` succeeds to set an ID which isn't corresponding to any fs/file tree. If such the bad ID is set to a filesystem, we can't mount this filesystem without specifying `subvol` or `subvolid` mount options. Signed-off-by: Satoru Tak

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Timofey Titovets
2017-09-13 0:13 GMT+03:00 Adam Borowski : > On Tue, Sep 12, 2017 at 04:12:32PM -0400, Austin S. Hemmelgarn wrote: >> On 2017-09-12 16:00, Adam Borowski wrote: >> > Noted. Both Marat's and my use cases, though, involve VMs that are off >> > most >> > of the time, and at least for me, turned on onl

Re: [PATCH v2 7/7] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-12 Thread Qu Wenruo
On 2017年09月13日 02:07, David Sterba wrote: On Tue, Sep 12, 2017 at 07:50:19PM +0200, Goffredo Baroncelli wrote: On 09/12/2017 07:03 PM, David Sterba wrote: Say I want to prepare a minimal image but will provide a large file

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Adam Borowski
On Tue, Sep 12, 2017 at 04:12:32PM -0400, Austin S. Hemmelgarn wrote: > On 2017-09-12 16:00, Adam Borowski wrote: > > Noted. Both Marat's and my use cases, though, involve VMs that are off most > > of the time, and at least for me, turned on only to test something. > > Touching mtime makes rsync r

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-12 16:00, Adam Borowski wrote: On Tue, Sep 12, 2017 at 03:11:52PM -0400, Austin S. Hemmelgarn wrote: On 2017-09-12 14:43, Adam Borowski wrote: On Tue, Sep 12, 2017 at 01:36:48PM -0400, Austin S. Hemmelgarn wrote: On 2017-09-12 13:21, Adam Borowski wrote: There's fallocate -d, but t

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Adam Borowski
On Tue, Sep 12, 2017 at 03:11:52PM -0400, Austin S. Hemmelgarn wrote: > On 2017-09-12 14:43, Adam Borowski wrote: > > On Tue, Sep 12, 2017 at 01:36:48PM -0400, Austin S. Hemmelgarn wrote: > > > On 2017-09-12 13:21, Adam Borowski wrote: > > > > There's fallocate -d, but that for some reason touches

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-12 14:47, Christoph Hellwig wrote: On Tue, Sep 12, 2017 at 08:43:59PM +0200, Adam Borowski wrote: For now, though, I wonder -- should we send fine folks at util-linux a patch to make fallocate -d restore mtime, either always or on an option? Don't do that. Please just add a new ioc

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-12 14:43, Adam Borowski wrote: On Tue, Sep 12, 2017 at 01:36:48PM -0400, Austin S. Hemmelgarn wrote: On 2017-09-12 13:21, Adam Borowski wrote: There's fallocate -d, but that for some reason touches mtime which makes rsync go again. This can be handled manually but is still not nice.

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Christoph Hellwig
On Tue, Sep 12, 2017 at 08:43:59PM +0200, Adam Borowski wrote: > For now, though, I wonder -- should we send fine folks at util-linux a patch > to make fallocate -d restore mtime, either always or on an option? Don't do that. Please just add a new ioctl or fallocate command that punches a hole if

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Adam Borowski
On Tue, Sep 12, 2017 at 01:36:48PM -0400, Austin S. Hemmelgarn wrote: > On 2017-09-12 13:21, Adam Borowski wrote: > > There's fallocate -d, but that for some reason touches mtime which makes > > rsync go again. This can be handled manually but is still not nice. > It touches mtime because it upda

Re: [PATCH v2 7/7] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-12 Thread David Sterba
On Tue, Sep 12, 2017 at 07:50:19PM +0200, Goffredo Baroncelli wrote: > On 09/12/2017 07:03 PM, David Sterba wrote: > > Say I want to prepare a minimal image but will provide a large file > ^^ > > I think that if the target is a f

Re: [PATCH v2 7/7] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-12 Thread Goffredo Baroncelli
On 09/12/2017 07:03 PM, David Sterba wrote: > Say I want to prepare a minimal image but will provide a large file ^^ I think that if the target is a file AND --minimize is passed, it is a reasonable expectation that the file is

Re: [PATCH 4/4 v2] btrfs-progs: mkfs: add subvolume support to mkfs

2017-09-12 Thread David Sterba
On Wed, Aug 30, 2017 at 10:24:50AM -0700, Yingyi Luo wrote: > From: yingyil > > Add -S/--subvol [NAME] option to configure. It enables users to create a > subvolume under the toplevel volume and populate the created subvolume > with files from the rootdir specified by -r/--rootdir option. Sounds

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-12 13:21, Adam Borowski wrote: On Tue, Sep 12, 2017 at 02:26:39PM +0300, Marat Khalili wrote: On 12/09/17 14:12, Adam Borowski wrote: Why would you need support in the hypervisor if cp --reflink=always is enough? +1 :) But I've already found one problem: I use rsync snapshots for b

Re: [PATCH 3/4 v2] btrfs-progs: mkfs: refactor create_data_reloc_tree

2017-09-12 Thread David Sterba
On Wed, Aug 30, 2017 at 10:24:49AM -0700, Yingyi Luo wrote: > From: yingyil > > Add an objectid parameter to make the function a general one for > inserting root items and rename it to create_tree. The change cascades > down to the callchain. > > Signed-off-by: yingyil The patch is independent

Re: [PATCH 2/4 v2] btrfs-progs: add a parameter to btrfs_link_subvol

2017-09-12 Thread David Sterba
On Wed, Aug 30, 2017 at 10:24:48AM -0700, Yingyi Luo wrote: > From: yingyil > > A convert parameter is added as a flag to indicate if btrfs_link_subvol() > is used for btrfs-convert. The change cascades down to the callchain. > > Signed-off-by: yingyil > --- > v2: Added a flag for btrfs_link_su

Re: [PATCH 1/4 v2] btrfs-progs: convert: move link_subvol out of main

2017-09-12 Thread David Sterba
On Wed, Aug 30, 2017 at 10:24:47AM -0700, Yingyi Luo wrote: > From: yingyil > > link_subvol() is moved to utils.c and renamed as btrfs_link_subvol(). The file utils.c is a catch-all file, but the subvolume function belongs to inode features, similar as in the kernel sources (although there it's

Re: defragmenting best practice?

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-12 12:28, Ulli Horlacher wrote: On Thu 2017-08-31 (09:05), Ulli Horlacher wrote: When I do a btrfs filesystem defragment -r /directory does it defragment really all files in this directory tree, even if it contains subvolumes? The man page does not mention subvolumes on this topic.

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Adam Borowski
On Tue, Sep 12, 2017 at 02:26:39PM +0300, Marat Khalili wrote: > On 12/09/17 14:12, Adam Borowski wrote: > > Why would you need support in the hypervisor if cp --reflink=always is > > enough? > +1 :) > > But I've already found one problem: I use rsync snapshots for backups, and > although rsync do

Re: [PATCH v2 7/7] btrfs-progs: Doc/mkfs: Add extra condition for rootdir option

2017-09-12 Thread David Sterba
On Mon, Sep 11, 2017 at 03:36:12PM +0900, Qu Wenruo wrote: > Add extra limitation explained for --rootdir option, including: > 1) Size limitation >Now I decide to follow "mkfs.ext4 -d" behavior, so we user is >responsible to make sure the block device/file is large enough. That's fine and

Re: [PATCH 08/10] btrfs: switch args for comp_*_refs

2017-09-12 Thread Timofey Titovets
2017-09-12 19:21 GMT+03:00 Timofey Titovets : > 2017-09-12 0:12 GMT+03:00 : >> From: Josef Bacik >> >> Because seriously? ref2 and then ref1? >> >> Signed-off-by: Josef Bacik >> --- >> fs/btrfs/delayed-ref.c | 8 >> 1 file changed, 4 insertions(+), 4 deletions(-) >> >> diff --git a/fs

defragmenting best practice?

2017-09-12 Thread Ulli Horlacher
On Thu 2017-08-31 (09:05), Ulli Horlacher wrote: > When I do a > btrfs filesystem defragment -r /directory > does it defragment really all files in this directory tree, even if it > contains subvolumes? > The man page does not mention subvolumes on this topic. No answer so far :-( But I found an

Re: [PATCH 08/10] btrfs: switch args for comp_*_refs

2017-09-12 Thread Timofey Titovets
2017-09-12 0:12 GMT+03:00 : > From: Josef Bacik > > Because seriously? ref2 and then ref1? > > Signed-off-by: Josef Bacik > --- > fs/btrfs/delayed-ref.c | 8 > 1 file changed, 4 insertions(+), 4 deletions(-) > > diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c > index 7374

Re: [PATCH 07/10] btrfs: remove type argument from comp_tree_refs

2017-09-12 Thread Timofey Titovets
2017-09-12 0:12 GMT+03:00 : > From: Josef Bacik > > We can get this from the ref we've passed in. > > Signed-off-by: Josef Bacik > --- > fs/btrfs/delayed-ref.c | 10 -- > 1 file changed, 4 insertions(+), 6 deletions(-) > > diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c > i

[PATCH] btrfs: prevent to set invalid default subvolid

2017-09-12 Thread Satoru Takeuchi
`btrfs sub set-default` succeeds to set an ID which isn't corresponding to any fs/file tree. If such the bad ID is set to a filesystem, we can't mount this filesystem without specifying `subvol` or `subvolid` mount options. Signed-off-by: Satoru Takeuchi --- fs/btrfs/ioctl.c | 4 1 file cha

Re: Please help with exact actions for raid1 hot-swap

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-11 17:33, Duncan wrote: Austin S. Hemmelgarn posted on Mon, 11 Sep 2017 11:11:01 -0400 as excerpted: On 2017-09-11 09:16, Marat Khalili wrote: Patrik, Duncan, thank you for the help. The `btrfs replace start /dev/sdb7 /dev/sdd7 /mnt/data` worked without a hitch (though I didn't try

Re: [PATCH] btrfs: convert all mount option checking code to use btrfs_test_opt

2017-09-12 Thread David Sterba
On Tue, Sep 12, 2017 at 08:08:08PM +0900, Satoru Takeuchi wrote: > Signed-off-by: Satoru Takeuchi Reviewed-by: David Sterba -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.o

Re: Storage and snapshots as historical yearly

2017-09-12 Thread Austin S. Hemmelgarn
On 2017-09-11 17:36, Pete wrote: On 09/11/2017 07:49 PM, Austin S. Hemmelgarn wrote: Unfortunately, I don't know of any overlay mount implementation that works correctly and reliably with BTRFS.  I know for a fact that OverlayFS (the upstream in-kernel implementation) does not work, and I belie

Re: [PATCH] btrfs-progs: cleanup whitespaces of btrfs-completion

2017-09-12 Thread David Sterba
On Tue, Sep 12, 2017 at 10:04:10AM +0900, Misono, Tomohiro wrote: > Clean up spaces and use only tabs. No functional change. > > Signed-off-by: Tomohiro Misono Applied, thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.ke

Re: dead links in wiki about why not using lz4

2017-09-12 Thread David Sterba
On Tue, Sep 12, 2017 at 09:19:14AM +, Alberto Bursi wrote: > Since zstd was added in btrfs there are people wondering why not lz4 too. > > The wiki paragraph [1] cites a couple dead links as explanations on why > it's not a good idea to have lz4 in btrfs. Regarding dead links, there's no rep

Re: dead links in wiki about why not using lz4

2017-09-12 Thread David Sterba
On Tue, Sep 12, 2017 at 12:33:21PM +0300, Timofey Titovets wrote: > 2017-09-12 12:19 GMT+03:00 Alberto Bursi : > > Since zstd was added in btrfs there are people wondering why not lz4 too. > > > > The wiki paragraph [1] cites a couple dead links as explanations on why > > it's not a good idea to ha

Re: btrfs-progs task tracking on github

2017-09-12 Thread David Sterba
On Sat, Sep 09, 2017 at 10:12:51AM +0530, Lakshmipathi.G wrote: > Existing project list is good. > > Reg: "Project tests: Build and test coverage: pre-built images on > docker hub" If we don't have pre-built images on docker hub, I'll > pick up this task and start working on it. thanks. Great,

Re: btrfs-progs task tracking on github

2017-09-12 Thread David Sterba
On Sat, Sep 09, 2017 at 08:13:04AM +0800, Qu Wenruo wrote: > > [1] https://github.com/kdave/btrfs-progs/projects > > This looks quite good. > > BTW, is that only repo owner be able to modify the status? We'll have to find out, but I think it's possible. There's Collaborators tab in settings, whe

Re: [PATCH] btrfs: avoid null pointer dereference on fs_info when calling btrfs_crit

2017-09-12 Thread David Sterba
On Mon, Sep 11, 2017 at 04:15:28PM +0100, Colin King wrote: > From: Colin Ian King > > There are checks on fs_info in __btrfs_panic to avoid dereferencing a > null fs_info, however, there is a call to btrfs_crit that may also > dereference a null fs_info. Fix this by adding a check to see if fs_i

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Marat Khalili
On 12/09/17 14:12, Adam Borowski wrote: Why would you need support in the hypervisor if cp --reflink=always is enough? +1 :) But I've already found one problem: I use rsync snapshots for backups, and although rsync does have --sparse argument, apparently it conflicts with --inplace. You canno

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Timofey Titovets
2017-09-12 14:12 GMT+03:00 Adam Borowski : > On Tue, Sep 12, 2017 at 02:01:53PM +0300, Timofey Titovets wrote: >> > On 12/09/17 13:32, Adam Borowski wrote: >> >> Just use raw -- btrfs already has every feature that qcow2 has, and >> >> does it better. This doesn't mean btrfs is the best choice for

Re: [PATCH] btrfs: send: fix error code if an unknown inode type is found

2017-09-12 Thread David Sterba
On Tue, Sep 12, 2017 at 12:02:44PM +0900, Tsutomu Itoh wrote: > Hi David, > > On 2017/09/12 1:58, David Sterba wrote: > > There are two types for "not supported", but the correct one is > > EOPNOTSUPP, let's use that one. The error message in the error case is > > not very helpful, enhance it at l

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Adam Borowski
On Tue, Sep 12, 2017 at 02:01:53PM +0300, Timofey Titovets wrote: > > On 12/09/17 13:32, Adam Borowski wrote: > >> Just use raw -- btrfs already has every feature that qcow2 has, and > >> does it better. This doesn't mean btrfs is the best choice for hosting > >> VM files, just that raw-over-btrfs

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Roman Mamedov
On Tue, 12 Sep 2017 12:32:14 +0200 Adam Borowski wrote: > discard in the guest (not supported over ide and virtio, supported over scsi > and virtio-scsi) IDE does support discard in QEMU, I use that all the time. It got broken briefly in QEMU 2.1 [1], but then fixed again. [1] https://bugs.deb

[PATCH] btrfs: convert all mount option checking code to use btrfs_test_opt

2017-09-12 Thread Satoru Takeuchi
Signed-off-by: Satoru Takeuchi --- fs/btrfs/extent-tree.c | 2 +- fs/btrfs/super.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index e2d7e86..ea9b0d2 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Timofey Titovets
2017-09-12 13:39 GMT+03:00 Marat Khalili : > On 12/09/17 13:01, Duncan wrote: >> >> AFAIK that's wrong -- the only time the app should see the error on btrfs >> raid1 is if the second copy is also bad > > So thought I, but... > >> IIRC from what I've read on-list, qcow2 isn't the best alternative f

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Marat Khalili
On 12/09/17 13:01, Duncan wrote: AFAIK that's wrong -- the only time the app should see the error on btrfs raid1 is if the second copy is also bad So thought I, but... IIRC from what I've read on-list, qcow2 isn't the best alternative for hosting VMs on top of btrfs. Yeah, I've seen discussio

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Adam Borowski
On Tue, Sep 12, 2017 at 10:01:07AM +, Duncan wrote: > BTW, I am most definitely /not/ a VM expert, and won't pretend to > understand the details or be able to explain further, but IIRC from what > I've read on-list, qcow2 isn't the best alternative for hosting VMs on > top of btrfs. Somethi

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Duncan
Marat Khalili posted on Tue, 12 Sep 2017 11:42:52 +0300 as excerpted: > On 12/09/17 11:25, Timofey Titovets wrote: >> AFAIK, if while read BTRFS get Read Error in RAID1, application will >> also see that error and if application can't handle it -> you got a >> problems >> >> So Btrfs RAID1 ONLY pr

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Timofey Titovets
2017-09-12 12:29 GMT+03:00 Marat Khalili : > On 12/09/17 12:21, Timofey Titovets wrote: >> >> Can't reproduce that on latest kernel: 4.13.1 > > Great! Thank you very much for the test. Do you know if it's fixed in 4.10? > (or what particular version does?) > -- > > With Best Regards, > Marat Khalil

Re: dead links in wiki about why not using lz4

2017-09-12 Thread Timofey Titovets
2017-09-12 12:19 GMT+03:00 Alberto Bursi : > Since zstd was added in btrfs there are people wondering why not lz4 too. > > The wiki paragraph [1] cites a couple dead links as explanations on why > it's not a good idea to have lz4 in btrfs. > > I think it would be nice if someone can write that info

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Marat Khalili
On 12/09/17 12:21, Timofey Titovets wrote: Can't reproduce that on latest kernel: 4.13.1 Great! Thank you very much for the test. Do you know if it's fixed in 4.10? (or what particular version does?) -- With Best Regards, Marat Khalili -- To unsubscribe from this list: send the line "unsubsc

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Timofey Titovets
2017-09-12 11:42 GMT+03:00 Marat Khalili : > On 12/09/17 11:25, Timofey Titovets wrote: >> >> AFAIK, if while read BTRFS get Read Error in RAID1, application will >> also see that error and if application can't handle it -> you got a >> problems >> >> So Btrfs RAID1 ONLY protect data, not applicati

dead links in wiki about why not using lz4

2017-09-12 Thread Alberto Bursi
Since zstd was added in btrfs there are people wondering why not lz4 too. The wiki paragraph [1] cites a couple dead links as explanations on why it's not a good idea to have lz4 in btrfs. I think it would be nice if someone can write that information back in the wiki, or for the very least wri

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Marat Khalili
On 12/09/17 11:25, Timofey Titovets wrote: AFAIK, if while read BTRFS get Read Error in RAID1, application will also see that error and if application can't handle it -> you got a problems So Btrfs RAID1 ONLY protect data, not application (qemu in your case). That's news to me! Why doesn't it tr

Re: qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Timofey Titovets
2017-09-12 11:02 GMT+03:00 Marat Khalili : > Thanks to the help from the list I've successfully replaced part of btrfs > raid1 filesystem. However, while I waited for best opinions on the course of > actions, the root filesystem of one the qemu-kvm VMs went read-only, and > this root was of course

qemu-kvm VM died during partial raid1 problems of btrfs

2017-09-12 Thread Marat Khalili
Thanks to the help from the list I've successfully replaced part of btrfs raid1 filesystem. However, while I waited for best opinions on the course of actions, the root filesystem of one the qemu-kvm VMs went read-only, and this root was of course based in a qcow2 file on the problematic btrfs