[PATCH] btrfs-progs: doc: fix btrfs-inspect-internal rootid doc
"btrfs inspect-internal rootid " rejects a file to be specified in the implementation. Therefore change "file or directory" to "directory" in the doc. Signed-off-by: Tomohiro Misono --- Documentation/btrfs-inspect-internal.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/btrfs-inspect-internal.asciidoc b/Documentation/btrfs-inspect-internal.asciidoc index 62b1029..a1ed1a3 100644 --- a/Documentation/btrfs-inspect-internal.asciidoc +++ b/Documentation/btrfs-inspect-internal.asciidoc @@ -136,7 +136,7 @@ resize operation, this may be useful before executing the actual resize operatio specify the device 'id' to query, default is 1 if this option is not used *rootid* :: -for a given file or directory, return the containing tree root id, for a +for a given directory, return the containing tree root id, for a subvolume itself return it's own tree id (ie. subvol id) + NOTE: The result is undefined for the so-called empty subvolumes (identified by -- 2.9.5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: doc: fix btrfs-inspect-internal rootid doc
On 2017年08月24日 15:39, Misono, Tomohiro wrote: "btrfs inspect-internal rootid " rejects a file to be specified in the implementation. Therefore change "file or directory" to "directory" in the doc. Signed-off-by: Tomohiro Misono --- Documentation/btrfs-inspect-internal.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/btrfs-inspect-internal.asciidoc b/Documentation/btrfs-inspect-internal.asciidoc index 62b1029..a1ed1a3 100644 --- a/Documentation/btrfs-inspect-internal.asciidoc +++ b/Documentation/btrfs-inspect-internal.asciidoc @@ -136,7 +136,7 @@ resize operation, this may be useful before executing the actual resize operatio specify the device 'id' to query, default is 1 if this option is not used *rootid* :: What about changing "" to ""? Thanks, Qu -for a given file or directory, return the containing tree root id, for a +for a given directory, return the containing tree root id, for a subvolume itself return it's own tree id (ie. subvol id) + NOTE: The result is undefined for the so-called empty subvolumes (identified by -- 2.9.5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs-progs: doc: add figure 1 to btrfs quota doc
The document of btrfs quota is missing figure 1. I notice the body is copy of http://sensille.com/qgroups.pdf (whi ch is linked from https://btrfs.wiki.kernel.org/index.php/Quota_support), and insert the figure. Signed-off-by: Tomohiro Misono --- Documentation/btrfs-quota.asciidoc | 18 +- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/Documentation/btrfs-quota.asciidoc b/Documentation/btrfs-quota.asciidoc index ef2e5d3..5bd13cb 100644 --- a/Documentation/btrfs-quota.asciidoc +++ b/Documentation/btrfs-quota.asciidoc @@ -96,7 +96,23 @@ instead of '0/ID'. For all higher levels, the ID can be chosen freely. Each qgroup can contain a set of lower level qgroups, thus creating a hierarchy of qgroups. Figure 1 shows an example qgroup tree. -// TODO: insert Figure 1 + +---+ + |2/1| + +---+ + / \ + +---+/ \+---+ + |1/1| |1/2| + +---+ +---+ + / \ / \ ++---+/ \+---+/ \+---+ +qgroups |0/1| |0/2| |0/3| ++-+-+ +---+ +---+ + | / \ / \ + | / \ / \ + |/ \ / \ +extents 1 234 + + Figure1: Sample qgroup hierarchy At the bottom, some extents are depicted showing which qgroups reference which extents. It is important to understand the notion of 'referenced' vs -- 2.9.5 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/6] raid5: remove a call to get_start_sect
On Wed, Aug 23, 2017 at 11:23:38AM -0700, Shaohua Li wrote:
> On Wed, Aug 23, 2017 at 07:10:28PM +0200, Christoph Hellwig wrote:
> > The block layer always remaps partitions before calling into the
> > ->make_request methods of drivers. Thus the call to get_start_sect in
> > in_chunk_boundary will always return 0 and can be removed.
> >
> > Signed-off-by: Christoph Hellwig
> > ---
> > drivers/md/raid5.c | 4 +++-
> > 1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> > index 0fc2748aaf95..d687aeb1b538 100644
> > --- a/drivers/md/raid5.c
> > +++ b/drivers/md/raid5.c
> > @@ -5092,10 +5092,12 @@ static int raid5_congested(struct mddev *mddev, int
> > bits)
> > static int in_chunk_boundary(struct mddev *mddev, struct bio *bio)
> > {
> > struct r5conf *conf = mddev->private;
> > - sector_t sector = bio->bi_iter.bi_sector + get_start_sect(bio->bi_bdev);
> > + sector_t sector = bio->bi_iter.bi_sector;
> > unsigned int chunk_sectors;
> > unsigned int bio_sectors = bio_sectors(bio);
> >
> > + WARN_ON_ONCE(bio->bi_partno);
> > +
Meh, of course bi_partno is only added a few patches later, so this
breaks bisectability. But given that the patch is already in there's
probably nothing I can do here.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: rework outstanding_extents
On Thu, Aug 17, 2017 at 01:01:29PM -0400, jo...@toxicpanda.com wrote:
>From: Josef Bacik
>
>Right now we do a lot of weird hoops around outstanding_extents in order
>to keep the extent count consistent. This is because we logically
>transfer the outstanding_extent count from the initial reservation
>through the set_delalloc_bits. This makes it pretty difficult to get a
>handle on how and when we need to mess with outstanding_extents.
>
>Fix this by revamping the rules of how we deal with outstanding_extents.
>Now instead everybody that is holding on to a delalloc extent is
>required to increase the outstanding extents count for itself. This
>means we'll have something like this
>
>btrfs_dealloc_reserve_metadata - outstanding_extents = 1
> btrfs_set_delalloc- outstanding_extents = 2
>btrfs_release_delalloc_extents - outstanding_extents = 1
>
>for an initial file write. Now take the append write where we extend an
>existing delalloc range but still under the maximum extent size
>
>btrfs_delalloc_reserve_metadata - outstanding_extents = 2
> btrfs_set_delalloc
>btrfs_set_bit_hook - outstanding_extents = 3
>btrfs_merge_bit_hook - outstanding_extents = 2
>btrfs_release_delalloc_extents - outstanding_extnets = 1
>
>In order to make the ordered extent transition we of course must now
>make ordered extents carry their own outstanding_extent reservation, so
>for cow_file_range we end up with
>
>btrfs_add_ordered_extent - outstanding_extents = 2
>clear_extent_bit - outstanding_extents = 1
>btrfs_remove_ordered_extent- outstanding_extents = 0
>
>This makes all manipulations of outstanding_extents much more explicit.
>Every successful call to btrfs_reserve_delalloc_metadata _must_ now be
>combined with btrfs_release_delalloc_extents, even in the error case, as
>that is the only function that actually modifies the
>outstanding_extents counter.
s/btrfs_reserve_delalloc_metadata/btrfs_delalloc_reserve_metadata
s/btrfs_release_delalloc_extents/btrfs_delalloc_release_extents
Acked-by: Lu Fengqi
>
>The drawback to this is now we are much more likely to have transient
>cases where outstanding_extents is much larger than it actually should
>be. This could happen before as we manipulated the delalloc bits, but
>now it happens basically at every write. This may put more pressure on
>the ENOSPC flushing code, but I think making this code simpler is worth
>the cost. I have another change coming to mitigate this side-effect
>somewhat.
>
>I also added trace points for the counter manipulation. These were used
>by a bpf script I wrote to help track down leak issues.
>
>Signed-off-by: Josef Bacik
>---
> fs/btrfs/btrfs_inode.h | 22 +++
> fs/btrfs/ctree.h | 2 +
> fs/btrfs/extent-tree.c | 141 ---
> fs/btrfs/file.c | 22 +++
> fs/btrfs/inode-map.c | 3 +-
> fs/btrfs/inode.c | 114 +++---
> fs/btrfs/ioctl.c | 2 +
> fs/btrfs/ordered-data.c | 21 ++-
> fs/btrfs/relocation.c| 3 +
> fs/btrfs/tests/inode-tests.c | 18 ++
> include/trace/events/btrfs.h | 44 ++
> 11 files changed, 236 insertions(+), 156 deletions(-)
>
>diff --git a/fs/btrfs/btrfs_inode.h b/fs/btrfs/btrfs_inode.h
>index 219e971..2894ad6 100644
>--- a/fs/btrfs/btrfs_inode.h
>+++ b/fs/btrfs/btrfs_inode.h
>@@ -267,6 +267,28 @@ static inline bool btrfs_is_free_space_inode(struct
>btrfs_inode *inode)
> return false;
> }
>
>+static inline void btrfs_mod_outstanding_extents(struct btrfs_inode *inode,
>+ int mod)
>+{
>+ ASSERT(spin_is_locked(&inode->lock));
>+ inode->outstanding_extents += mod;
>+ if (btrfs_is_free_space_inode(inode))
>+ return;
>+ trace_btrfs_inode_mod_outstanding_extents(inode->root, btrfs_ino(inode),
>+mod);
>+}
>+
>+static inline void btrfs_mod_reserved_extents(struct btrfs_inode *inode,
>+int mod)
>+{
>+ ASSERT(spin_is_locked(&inode->lock));
>+ inode->reserved_extents += mod;
>+ if (btrfs_is_free_space_inode(inode))
>+ return;
>+ trace_btrfs_inode_mod_reserved_extents(inode->root, btrfs_ino(inode),
>+ mod);
>+}
>+
> static inline int btrfs_inode_in_log(struct btrfs_inode *inode, u64
> generation)
> {
> int ret = 0;
>diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
>index 33e942b..b43114d 100644
>--- a/fs/btrfs/ctree.h
>+++ b/fs/btrfs/ctree.h
>@@ -2735,6 +2735,8 @@ int btrfs_subvolume_reserve_metadata(struct btrfs_root
>*root,
>u64 *qgroup_reserved, bool use_global_rsv);
> void btrfs_subvolume_release_metadata(struct btrfs_fs_info *fs_info,
> struct btrfs_block_rsv *rsv);
>+void btrfs
[PATCH] Btrfs-progs: Check on num_stripes in print_chunk
From: Zhang Yu
[TEST/fuzz] case: 004-simple-dump-tree
Since the wrong key(DATA_RELOC_TREE CHUNK_ITEM 0) in root tree,
error calling print_chunk(), resulting in num_stripes == 0.
ERROR:
[TEST/fuzz] 004-simple-dump-tree
ctree.h:317: btrfs_chunk_item_size: BUG_ON `num_stripes == 0`
triggered, value 1
failed (ignored, ret=134): /myproject/btrfs-progs/btrfs
inspect-internal dump-tree
/myproject/btrfs-progs/tests/fuzz-tests/images/
bko-155201-wrong-chunk-item-in-root-tree.raw.restored
test failed for case 004-simple-dump-tree
Makefile:288: recipe for target 'test-fuzz' failed
make: *** [test-fuzz] Error 1
So, check on num_stripes in print_chunk
Signed-off-by: Zhang Yu
---
print-tree.c | 9 +
1 file changed, 9 insertions(+)
diff --git a/print-tree.c b/print-tree.c
index a0d3395..08f7edb 100644
--- a/print-tree.c
+++ b/print-tree.c
@@ -199,6 +199,15 @@ void print_chunk(struct extent_buffer *eb, struct
btrfs_chunk *chunk)
{
int num_stripes = btrfs_chunk_num_stripes(eb, chunk);
int i;
+ /*
+* check on num_stripes
+* Btrfs_chunk contains at least one stripes
+*/
+ if (num_stripes < 1) {
+ printf("invalid num_stripes: %u\n", num_stripes);
+ return;
+ }
+
u32 chunk_item_size = btrfs_chunk_item_size(num_stripes);
char chunk_flags_str[32] = {0};
--
2.9.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BTRFS RAID 1 not mountable: open_ctree failed, super_num_devices 3 mismatch with num_devices 2 found here
> I rebootet with HWE K4.11 > > and took a pic of the error message (see attachment). > > It seems btrfs still sees the removed NVME. > There is a mismatch from super_num_devices (3) to num_devices (2) > with indicates something strage is going on here, imho. > > Then i returned and booted K4.4, which boots fine. > > root@vHost1:~# btrfs dev stat / > [/dev/nvme0n1p1].write_io_errs 0 > [/dev/nvme0n1p1].read_io_errs0 > [/dev/nvme0n1p1].flush_io_errs 0 > [/dev/nvme0n1p1].corruption_errs 0 > [/dev/nvme0n1p1].generation_errs 0 > [/dev/sda1].write_io_errs 0 > [/dev/sda1].read_io_errs0 > [/dev/sda1].flush_io_errs 0 > [/dev/sda1].corruption_errs 0 > [/dev/sda1].generation_errs 0 > > Btw i edited the subject to match the correct error. > > > Sash Thats very odd, if super_num_devices in superblocks don't match real number of devices then 4.4 kernel shouldn't mount the filesystem too. We probably need help from one of btrfs developers since I'm not one, I'm just btrfs user. Can you provide outpus of: btrfs inspect-internal dump-super -f /dev/sda1 btrfs inspect-internal dump-super -f /dev/nvme0n1p1 Depending on version of btrfs-progs you may need to use btrfs-dump-super instead of "btrfs inspect-internal dump-super" >3rd i saw https://patchwork.kernel.org/patch/9419189/ from Roman. Did >he receive any comments on his patch? This one could help on this >problem, too. Don't know about this patch from Roman per se, but there is a patchset[1] which is aimed to be merged in 4.14 merge window AFAIK. [1] https://www.spinics.net/lists/linux-btrfs/msg66891.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: do not reset bio->bi_ops while writing bio
On Wed, Aug 23, 2017 at 12:15:09PM -0600, Liu Bo wrote: > flush_epd_write_bio() sets bio->bi_ops by itself to honor REQ_SYNC, > but it's not needed at all since bio->bi_ops has set up properly in > both __extent_writepage() and write_one_eb(), and in the case of > write_one_eb(), it also sets REQ_META, which we will lose in > flush_epd_write_bio(). > > This remove this unnecessary bio->bi_ops setting. Nitpick: it's bi_op, not bi_ops. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH][v2] btrfs: change how we decide to commit transactions during flushing
On 22.08.2017 23:00, jo...@toxicpanda.com wrote:
> From: Josef Bacik
>
> Nikolay reported that generic/273 was failing currently with ENOSPC.
> Turns out this is because we get to the point where the outstanding
> reservations are greater than the pinned space on the fs. This is a
> mistake, previously we used the current reservation amount in
> may_commit_transaction, not the entire outstanding reservation amount.
> Fix this to find the minimum byte size needed to make progress in
> flushing, and pass that into may_commit_transaction. From there we can
> make a smarter decision on whether to commit the transaction or not.
> This fixes the failure in generic/273.
>
> Reported-by: Nikolay Borisov
> Signed-off-by: Josef Bacik
Reviewed-and-tested-by: Nikolay Borisov
> ---
> v1->v2:
> - check the ticket bytes in may_commit_transaction instead of copying bytes
> around.
> - clean up may_commit_transaction to remove unused arguments
>
> fs/btrfs/extent-tree.c | 42 --
> 1 file changed, 28 insertions(+), 14 deletions(-)
>
> diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> index a5d59dd..1464678 100644
> --- a/fs/btrfs/extent-tree.c
> +++ b/fs/btrfs/extent-tree.c
> @@ -4836,6 +4836,13 @@ static void shrink_delalloc(struct btrfs_fs_info
> *fs_info, u64 to_reclaim,
> }
> }
>
> +struct reserve_ticket {
> + u64 bytes;
> + int error;
> + struct list_head list;
> + wait_queue_head_t wait;
> +};
> +
> /**
> * maybe_commit_transaction - possibly commit the transaction if its ok to
> * @root - the root we're allocating for
> @@ -4847,18 +4854,29 @@ static void shrink_delalloc(struct btrfs_fs_info
> *fs_info, u64 to_reclaim,
> * will return -ENOSPC.
> */
> static int may_commit_transaction(struct btrfs_fs_info *fs_info,
> - struct btrfs_space_info *space_info,
> - u64 bytes, int force)
> + struct btrfs_space_info *space_info)
> {
> + struct reserve_ticket *ticket = NULL;
> struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_block_rsv;
> struct btrfs_trans_handle *trans;
> + u64 bytes;
>
> trans = (struct btrfs_trans_handle *)current->journal_info;
> if (trans)
> return -EAGAIN;
>
> - if (force)
> - goto commit;
> + spin_lock(&space_info->lock);
> + if (!list_empty(&space_info->priority_tickets))
> + ticket = list_first_entry(&space_info->priority_tickets,
> + struct reserve_ticket, list);
> + else if (!list_empty(&space_info->tickets))
> + ticket = list_first_entry(&space_info->tickets,
> + struct reserve_ticket, list);
> + bytes = (ticket) ? ticket->bytes : 0;
> + spin_unlock(&space_info->lock);
> +
> + if (!bytes)
> + return 0;
>
> /* See if there is enough pinned space to make this reservation */
> if (percpu_counter_compare(&space_info->total_bytes_pinned,
> @@ -4873,8 +4891,12 @@ static int may_commit_transaction(struct btrfs_fs_info
> *fs_info,
> return -ENOSPC;
>
> spin_lock(&delayed_rsv->lock);
> + if (delayed_rsv->size > bytes)
> + bytes = 0;
> + else
> + bytes -= delayed_rsv->size;
> if (percpu_counter_compare(&space_info->total_bytes_pinned,
> -bytes - delayed_rsv->size) < 0) {
> +bytes) < 0) {
> spin_unlock(&delayed_rsv->lock);
> return -ENOSPC;
> }
> @@ -4888,13 +4910,6 @@ static int may_commit_transaction(struct btrfs_fs_info
> *fs_info,
> return btrfs_commit_transaction(trans);
> }
>
> -struct reserve_ticket {
> - u64 bytes;
> - int error;
> - struct list_head list;
> - wait_queue_head_t wait;
> -};
> -
> /*
> * Try to flush some data based on policy set by @state. This is only
> advisory
> * and may fail for various reasons. The caller is supposed to examine the
> @@ -4944,8 +4959,7 @@ static void flush_space(struct btrfs_fs_info *fs_info,
> ret = 0;
> break;
> case COMMIT_TRANS:
> - ret = may_commit_transaction(fs_info, space_info,
> - num_bytes, 0);
> + ret = may_commit_transaction(fs_info, space_info);
> break;
> default:
> ret = -ENOSPC;
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[GIT PULL] Btrfs fix for rc7
Hi, we have one more fixup that stems from the blk_status_t conversion that did not quite cover everything. The normal cases were not affected because the code is 0, but any error and retries could mix up new and old values. Please pull, thanks. The following changes since commit 14ccee78fc82f5512908f4424f541549a5705b89: Linux 4.13-rc6 (2017-08-20 14:13:52 -0700) are available in the git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-4.13-rc7 for you to fetch changes up to 58efbc9f5463308c43d812fd0748a4dee44c8a16: Btrfs: fix blk_status_t/errno confusion (2017-08-24 17:19:02 +0200) Omar Sandoval (1): Btrfs: fix blk_status_t/errno confusion fs/btrfs/disk-io.c | 4 ++-- fs/btrfs/inode.c | 70 +- fs/btrfs/raid56.c | 34 +- fs/btrfs/volumes.c | 10 fs/btrfs/volumes.h | 6 ++--- 5 files changed, 64 insertions(+), 60 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: fix cross-compile build
On Wed, Aug 16, 2017 at 08:17:00AM +0800, Qu Wenruo wrote: > > > On 2017年08月16日 02:11, Eric Sandeen wrote: > > The mktables binary needs to be build with the host > > compiler at built time, not the target compiler, because > > it runs at build time to generate the raid tables. > > > > Copy auto-fu from xfsprogs and modify Makefile to > > accomodate this. > > > > Reported-by: Hallo32 > > Signed-off-by: Eric Sandeen > > Looks better than my previous patch. > With @BUILD_CLFAGS support and better BUILD_CC/CLFAGS detection for > native build environment. The contents of tables.c has practially not changed since its introduction in 2000s. I see no sense in regenerating it each time during build and I don't want to introduce host CC build just for this one file, standard practice or not. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/3] btrfs-progs: fix option handling for some commands
On Thu, Aug 24, 2017 at 02:18:25PM +0900, Misono, Tomohiro wrote: > I found some btrfs commands options are not working because of > inappropriate getopt_long() setting. > > This fixes "btrfs check -Q/-E" > > Signed-off-by: Tomohiro Misono Patches 1-3 and the doc fix applied with some changelog updates, thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: doc: fix btrfs-inspect-internal rootid doc
On Thu, Aug 24, 2017 at 04:39:53PM +0900, Misono, Tomohiro wrote: > "btrfs inspect-internal rootid " rejects a file to be specified in > the implementation. > Therefore change "file or directory" to "directory" in the doc. Is there a reason why a file should not be accepted? The ioctl supports that just fine. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: number of subvolumes
>> Using hundreds or thousands of snapshots is probably fine >> mostly. As I mentioned previously, with a link to the relevant email describing the details, the real issue is reflinks/backrefs. Usually subvolume and snapshots involve them. > We find that typically apt is very slow on a machine with 50 > or so snapshots and raid10. Slow as in probably 10x slower as > doing the same update on a machine with 'single' and no > snapshots. That seems to indicate using snapshots on a '/' volume to provide a "rollback machine" like SUSE. Since '/' usually has many small files and installation of upgraded packages involves only a small part of them, that usually involves a lot of reflinks/backrefs. But that you find that the system has slowed down significantly in ordinary operations is unusual, because what is slow in situations with many relinks/backrefs per extent is not access, but operations like 'balance' or 'delete'. Guessing wildly what you describe seems more the effect of low locality (aka high fragmentation) which is often the result of the 'ssd' option which should always be explicitly disabled (even for volumes on flash SSD storage). I would suggest some use of 'filefrag' to analyze and perhaps use of 'defrag' and 'balance'. Another possibility is having enabled compression with the presence of many in-place updates on some files, which can result also in low locality (high fragmentation). As usual with Btrfs, there are corner cases to avoid: 'defrag' should be done before 'balance' and with compression switched off (IIRC): https://wiki.archlinux.org/index.php/Btrfs#Defragmentation Defragmenting a file which has a COW copy (either a snapshot copy or one made with cp --reflink or bcp) plus using the -c switch with a compression algorithm may result in two unrelated files effectively increasing the disk usage. https://wiki.debian.org/Btrfs Mounting with -o autodefrag will duplicate reflinked or snapshotted files when you run a balance. Also, whenever a portion of the fs is defragmented with "btrfs filesystem defragment" those files will lose their reflinks and the data will be "duplicated" with n-copies. The effect of this is that volumes that make heavy use of reflinks or snapshots will run out of space. Additionally, if you have a lot of snapshots or reflinked files, please use "-f" to flush data for each file before going to the next file. I prefer dump-and-reload. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: doc: add figure 1 to btrfs quota doc
On Thu, Aug 24, 2017 at 04:48:31PM +0900, Misono, Tomohiro wrote: > The document of btrfs quota is missing figure 1. > > I notice the body is copy of http://sensille.com/qgroups.pdf (whi > ch is linked from > https://btrfs.wiki.kernel.org/index.php/Quota_support), and insert the > figure. > > Signed-off-by: Tomohiro Misono Applied, thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs: do not reset bio->bi_ops while writing bio
On Thu, Aug 24, 2017 at 07:10:41AM -0700, Christoph Hellwig wrote: > On Wed, Aug 23, 2017 at 12:15:09PM -0600, Liu Bo wrote: > > flush_epd_write_bio() sets bio->bi_ops by itself to honor REQ_SYNC, > > but it's not needed at all since bio->bi_ops has set up properly in > > both __extent_writepage() and write_one_eb(), and in the case of > > write_one_eb(), it also sets REQ_META, which we will lose in > > flush_epd_write_bio(). > > > > This remove this unnecessary bio->bi_ops setting. > > Nitpick: it's bi_op, not bi_ops. Ah thanks, it's bio->bi_opf ;) thanks, -liubo -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: number of subvolumes
> We find that typically apt is very slow on a machine with 50 or so snapshots > and raid10. Slow as in probably 10x slower as doing the same update on a > machine with 'single' and no snapshots. > > Other operations seem to be the same speed, especially disk benchmarks do not > seem to indicate any performance degradation. For meaningful discussion it is important to take into account the fact that dpkg infamously calls fsync after changing every bit of information, so basically you're measuring fsync speed. Which is slow on btrfs (compared to simpler filesystems), but unrelated to normal work. I've got two near-identical servers here with several containers each different only on in filesystem: btrfs-raid1 on one (for historical reasons) and ext4/mdadm-raid1 on another, no snapshots, no reflinks. Each time containers on ext4 update several times faster, but in everyday operation there's no significant difference. -- With Best Regards, Marat Khalili -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC PATCH] Btrfs: handle unaligned tail of data ranges more efficient
At now while switch page bits in data ranges
we always hande +1 page, for cover case
where end of data range is not page aligned
Let's handle that case more obvious and efficient
Check end aligment directly and touch +1 page
only then needed
Signed-off-by: Timofey Titovets
---
fs/btrfs/extent_io.c | 12 ++--
fs/btrfs/inode.c | 6 +-
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 98d85d2009f8..ccb66c1485e2 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -1360,7 +1360,11 @@ void extent_range_clear_dirty_for_io(struct inode
*inode, u64 start, u64 end)
unsigned long end_index = end >> PAGE_SHIFT;
struct page *page;
- while (index <= end_index) {
+ /* Don't miss unaligned end */
+ if (end%PAGE_SIZE > 0)
+ end_index++;
+
+ while (index < end_index) {
page = find_get_page(inode->i_mapping, index);
BUG_ON(!page); /* Pages should be in the extent_io_tree */
clear_page_dirty_for_io(page);
@@ -1375,7 +1379,11 @@ void extent_range_redirty_for_io(struct inode *inode,
u64 start, u64 end)
unsigned long end_index = end >> PAGE_SHIFT;
struct page *page;
- while (index <= end_index) {
+ /* Don't miss unaligned end */
+ if (end%PAGE_SIZE > 0)
+ end_index++;
+
+ while (index < end_index) {
page = find_get_page(inode->i_mapping, index);
BUG_ON(!page); /* Pages should be in the extent_io_tree */
__set_page_dirty_nobuffers(page);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index cb7779b08aaf..79feaea0fede 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -10745,7 +10745,11 @@ void btrfs_set_range_writeback(void *private_data, u64
start, u64 end)
unsigned long end_index = end >> PAGE_SHIFT;
struct page *page;
- while (index <= end_index) {
+ /* Don't miss unaligned end */
+ if (end%PAGE_SIZE > 0)
+ end_index++;
+
+ while (index < end_index) {
page = find_get_page(inode->i_mapping, index);
ASSERT(page); /* Pages should be in the extent_io_tree */
set_page_writeback(page);
--
2.14.1
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: number of subvolumes
Op Thu, 24 Aug 2017 22:40:54 +0300, schreef Marat Khalili: >> We find that typically apt is very slow on a machine with 50 or so >> snapshots and raid10. Slow as in probably 10x slower as doing the same >> update on a machine with 'single' and no snapshots. >> >> Other operations seem to be the same speed, especially disk benchmarks >> do not seem to indicate any performance degradation. > > For meaningful discussion it is important to take into account the fact Doing daily updates on a desktop is not uncommon and when 3 minutes become 30 then many would call that meaningfull. Similar for a single office server, which is upgraded twice a year, where an upgrade normally would take an hour or 2, but now more than a working day. In the meantime, take samba and postgresql offline, preventing people to work for a few hours. My point is: fsync is not targeted specifically in many common disk bench marks (phoronix?), it might be posible that there is no trigger to spend much time on optimizations in that area. That doesn't make it meaningless. > that dpkg infamously calls fsync after changing every bit of > information, so basically you're measuring fsync speed. Which is slow on > btrfs (compared to simpler filesystems), but unrelated to normal work. OTOH it would be nice if dpkg would at last start making use btrfs snapshot features and abandon these unnecssary fsyncs completely, instead restoring a failed install from a snapshot. This would probably result in a performance improve compared to ext4. > I've got two near-identical servers here with several containers each > different only on in filesystem: btrfs-raid1 on one (for historical > reasons) and ext4/mdadm-raid1 on another, no snapshots, no reflinks. > Each time containers on ext4 update several times faster, but in > everyday operation there's no significant difference. > -- > > With Best Regards, > Marat Khalili -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: fix cross-compile build
On 2017年08月25日 01:01, David Sterba wrote: On Wed, Aug 16, 2017 at 08:17:00AM +0800, Qu Wenruo wrote: On 2017年08月16日 02:11, Eric Sandeen wrote: The mktables binary needs to be build with the host compiler at built time, not the target compiler, because it runs at build time to generate the raid tables. Copy auto-fu from xfsprogs and modify Makefile to accomodate this. Reported-by: Hallo32 Signed-off-by: Eric Sandeen Looks better than my previous patch. With @BUILD_CLFAGS support and better BUILD_CC/CLFAGS detection for native build environment. The contents of tables.c has practially not changed since its introduction in 2000s. I see no sense in regenerating it each time during build and I don't want to introduce host CC build just for this one file, standard practice or not. I understand the worry about adding host CC. But just shown by the patch, the impact is minimal. At least for me and already mentioned before, text file managed by git should be reviewable. No matter if the file get modified in 20 years or not. (BTW, I don't think there is much problem regenerating the RAID6 table. It's way faster than auto-conf/auto-header things) And I think that's the same reason other projects doesn't manage their generated CRC tables using git. Thanks, Qu -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: doc: fix btrfs-inspect-internal rootid doc
On 2017/08/25 2:37, David Sterba wrote: On Thu, Aug 24, 2017 at 04:39:53PM +0900, Misono, Tomohiro wrote: "btrfs inspect-internal rootid " rejects a file to be specified in the implementation. Therefore change "file or directory" to "directory" in the doc. Is there a reason why a file should not be accepted? The ioctl supports that just fine. I thought ioctl would work for file, but current code checks whether is directory or not in the first place. So, I change the doc. If it is better to change the code rather than the doc, I will do that. What do you think? -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: fix cross-compile build
On 8/24/17 12:01 PM, David Sterba wrote: > On Wed, Aug 16, 2017 at 08:17:00AM +0800, Qu Wenruo wrote: >> >> >> On 2017年08月16日 02:11, Eric Sandeen wrote: >>> The mktables binary needs to be build with the host >>> compiler at built time, not the target compiler, because >>> it runs at build time to generate the raid tables. >>> >>> Copy auto-fu from xfsprogs and modify Makefile to >>> accomodate this. >>> >>> Reported-by: Hallo32 >>> Signed-off-by: Eric Sandeen >> >> Looks better than my previous patch. >> With @BUILD_CLFAGS support and better BUILD_CC/CLFAGS detection for >> native build environment. > > The contents of tables.c has practially not changed since its > introduction in 2000s. I see no sense in regenerating it each time > during build Make does know enough to generate it only as needed, of course. > and I don't want to introduce host CC build just for > this one file, standard practice or not. *shrug* it seems like the obviously correct fix to me, but I won't fight over it. -Eric -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Btrfs: use the new helper wbc_to_write_flags
This updates btrfs to use the helper wbc_to_write_flags which has been applied in ext4/xfs/f2fs/block. Please note that, with this, btrfs's dirty pages written by a writeback job will carry the flag REQ_BACKGROUND, which is currently used by writeback-throttle to determine whether it should go to get a request or wait. Signed-off-by: Liu Bo --- fs/btrfs/extent_io.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index a454a10..825fad6 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -3467,8 +3467,7 @@ static int __extent_writepage(struct page *page, struct writeback_control *wbc, int write_flags = 0; unsigned long nr_written = 0; - if (wbc->sync_mode == WB_SYNC_ALL) - write_flags = REQ_SYNC; + write_flags = wbc_to_write_flags(wbc); trace___extent_writepage(page, inode, wbc); @@ -3713,7 +3712,7 @@ static noinline_for_stack int write_one_eb(struct extent_buffer *eb, u32 nritems; unsigned long i, num_pages; unsigned long start, end; - int write_flags = (epd->sync_io ? REQ_SYNC : 0) | REQ_META; + int write_flags = wbc_to_write_flags(wbc) | REQ_META; int ret = 0; clear_bit(EXTENT_BUFFER_WRITE_ERR, &eb->bflags); -- 2.9.4 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs: fix NULL pointer dereference from free_reloc_roots()
__del_reloc_root should be called before freeing up reloc_root->node.
If not, calling __del_reloc_root() dereference reloc_root->node, causing
the system BUG.
Signed-off-by: Naohiro Aota
---
fs/btrfs/relocation.c |2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
index 65661d1aae4e..6445de8e9ece 100644
--- a/fs/btrfs/relocation.c
+++ b/fs/btrfs/relocation.c
@@ -2393,11 +2393,11 @@ void free_reloc_roots(struct list_head *list)
while (!list_empty(list)) {
reloc_root = list_entry(list->next, struct btrfs_root,
root_list);
+ __del_reloc_root(reloc_root);
free_extent_buffer(reloc_root->node);
free_extent_buffer(reloc_root->commit_root);
reloc_root->node = NULL;
reloc_root->commit_root = NULL;
- __del_reloc_root(reloc_root);
}
}
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs: fix NULL pointer dereference from free_reloc_roots()
On 25.08.2017 08:15, Naohiro Aota wrote:
> __del_reloc_root should be called before freeing up reloc_root->node.
> If not, calling __del_reloc_root() dereference reloc_root->node, causing
> the system BUG.
>
> Signed-off-by: Naohiro Aota
This patch should also have:
Fixes: 6bdf131fac23 ("Btrfs: don't leak reloc root nodes on error")
Cc: # 4.9
With that:
Reviewed-by: Nikolay Borisov
> ---
> fs/btrfs/relocation.c |2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
> index 65661d1aae4e..6445de8e9ece 100644
> --- a/fs/btrfs/relocation.c
> +++ b/fs/btrfs/relocation.c
> @@ -2393,11 +2393,11 @@ void free_reloc_roots(struct list_head *list)
> while (!list_empty(list)) {
> reloc_root = list_entry(list->next, struct btrfs_root,
> root_list);
> + __del_reloc_root(reloc_root);
> free_extent_buffer(reloc_root->node);
> free_extent_buffer(reloc_root->commit_root);
> reloc_root->node = NULL;
> reloc_root->commit_root = NULL;
> - __del_reloc_root(reloc_root);
> }
> }
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: number of subvolumes
On Thu, Aug 24, 2017 at 3:56 PM, Ferry Toth wrote: > Op Thu, 24 Aug 2017 22:40:54 +0300, schreef Marat Khalili: > >>> We find that typically apt is very slow on a machine with 50 or so >>> snapshots and raid10. Slow as in probably 10x slower as doing the same >>> update on a machine with 'single' and no snapshots. >>> >>> Other operations seem to be the same speed, especially disk benchmarks >>> do not seem to indicate any performance degradation. >> >> For meaningful discussion it is important to take into account the fact > > Doing daily updates on a desktop is not uncommon and when 3 minutes > become 30 then many would call that meaningfull. > > Similar for a single office server, which is upgraded twice a year, where > an upgrade normally would take an hour or 2, but now more than a working > day. In the meantime, take samba and postgresql offline, preventing > people to work for a few hours. > > My point is: fsync is not targeted specifically in many common disk bench > marks (phoronix?), it might be posible that there is no trigger to spend > much time on optimizations in that area. That doesn't make it meaningless. > >> that dpkg infamously calls fsync after changing every bit of >> information, so basically you're measuring fsync speed. Which is slow on >> btrfs (compared to simpler filesystems), but unrelated to normal work. > > OTOH it would be nice if dpkg would at last start making use btrfs > snapshot features and abandon these unnecssary fsyncs completely, instead > restoring a failed install from a snapshot. This would probably result in > a performance improve compared to ext4. > >> I've got two near-identical servers here with several containers each >> different only on in filesystem: btrfs-raid1 on one (for historical >> reasons) and ext4/mdadm-raid1 on another, no snapshots, no reflinks. >> Each time containers on ext4 update several times faster, but in >> everyday operation there's no significant difference. In the thread "Containers, Btrfs vs Btrfs + overlayfs" there's the idea of nullifying fsyncs in container contexts. Maybe it could be adapted for out of band system software updates, i.e. 1. updater runs in a container 2. takes a snapshot of the system 3. assembles the snapshot per fstab 4. performs the OS update (within the container, on the snapshot), filtering out fsync 5. does a sync() after completion of update 6. update bootloader configuration to point to the updated snapshot/file tree 7. quit container 8. user reboots whenever convenient to run the updated system Basically this is still an atomic update that won't nerf either the file system or the OS if there's a crash or power failure prior to step 7. Any failure, just delete the snapshot (failed out of band update). The existing tree isn't affected by either the update or the failure so there's not even a problem running it while the user is working, insofar as binaries and libraries aren't being yanked out from under running processes, they're all out of band changes. Something like this exists, but it's not package based, rather it's "git like", and also has no optimizations for Btrfs. It's updates are out of band, and always atomic, not matter the file system. OSTree. https://github.com/ostreedev/ostree https://ostree.readthedocs.io/en/latest/ -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC PATCH] Btrfs: handle unaligned tail of data ranges more efficient
On 25.08.2017 00:12, Timofey Titovets wrote:
> At now while switch page bits in data ranges
> we always hande +1 page, for cover case
> where end of data range is not page aligned
>
> Let's handle that case more obvious and efficient
> Check end aligment directly and touch +1 page
> only then needed
>
> Signed-off-by: Timofey Titovets
> ---
> fs/btrfs/extent_io.c | 12 ++--
> fs/btrfs/inode.c | 6 +-
> 2 files changed, 15 insertions(+), 3 deletions(-)
>
> diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
> index 98d85d2009f8..ccb66c1485e2 100644
> --- a/fs/btrfs/extent_io.c
> +++ b/fs/btrfs/extent_io.c
> @@ -1360,7 +1360,11 @@ void extent_range_clear_dirty_for_io(struct inode
> *inode, u64 start, u64 end)
> unsigned long end_index = end >> PAGE_SHIFT;
> struct page *page;
>
> - while (index <= end_index) {
> + /* Don't miss unaligned end */
> + if (end%PAGE_SIZE > 0)
> + end_index++;
if (!IS_ALIGNED(end, PAGE_SIZE)).
> +
> + while (index < end_index) {
> page = find_get_page(inode->i_mapping, index);
> BUG_ON(!page); /* Pages should be in the extent_io_tree */
> clear_page_dirty_for_io(page);
> @@ -1375,7 +1379,11 @@ void extent_range_redirty_for_io(struct inode *inode,
> u64 start, u64 end)
> unsigned long end_index = end >> PAGE_SHIFT;
> struct page *page;
>
> - while (index <= end_index) {
> + /* Don't miss unaligned end */
> + if (end%PAGE_SIZE > 0)
> + end_index++;
ditto
> +
> + while (index < end_index) {
> page = find_get_page(inode->i_mapping, index);
> BUG_ON(!page); /* Pages should be in the extent_io_tree */
> __set_page_dirty_nobuffers(page);
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index cb7779b08aaf..79feaea0fede 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -10745,7 +10745,11 @@ void btrfs_set_range_writeback(void *private_data,
> u64 start, u64 end)
> unsigned long end_index = end >> PAGE_SHIFT;
> struct page *page;
>
> - while (index <= end_index) {
> + /* Don't miss unaligned end */
> + if (end%PAGE_SIZE > 0)
> + end_index++;
Ditto
> +
> + while (index < end_index) {
> page = find_get_page(inode->i_mapping, index);
> ASSERT(page); /* Pages should be in the extent_io_tree */
> set_page_writeback(page);
> --
> 2.14.1
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
