[PATCH V8] Btrfs: enhance raid1/10 balance heuristic

From: Timofey Titovets Currently btrfs raid1/10 balancer bаlance requests to mirrors, based on pid % num of mirrors. Make logic understood: - if one of underline devices are non rotational - Queue length to underline devices By default try use pid % num_mirrors guessing, but: - If one

Re: [PATCH v2] btrfs: add zstd compression level support

вт, 13 нояб. 2018 г. в 04:52, Nick Terrell : > > > > > On Nov 12, 2018, at 4:33 PM, David Sterba wrote: > > > > On Wed, Oct 31, 2018 at 11:11:08AM -0700, Nick Terrell wrote: > >> From: Jennifer Liu > >> > >> Adds zstd compression level support to btrfs. Zstd requires > >> different amounts of

[PATCH V7] Btrfs: enhance raid1/10 balance heuristic

From: Timofey Titovets Currently btrfs raid1/10 balancer bаlance requests to mirrors, based on pid % num of mirrors. Make logic understood: - if one of underline devices are non rotational - Queue length to underline devices By default try use pid % num_mirrors guessing, but: - If one

Re: [PATCH V6] Btrfs: enhance raid1/10 balance heuristic

пн, 12 нояб. 2018 г. в 10:28, Nikolay Borisov : > > > > On 25.09.18 г. 21:38 ч., Timofey Titovets wrote: > > Currently btrfs raid1/10 balancer bаlance requests to mirrors, > > based on pid % num of mirrors. > > > > Make logic understood: > > - if o

Re: [PATCH V6] Btrfs: enhance raid1/10 balance heuristic

Gentle ping. вт, 25 сент. 2018 г. в 21:38, Timofey Titovets : > > Currently btrfs raid1/10 balancer bаlance requests to mirrors, > based on pid % num of mirrors. > > Make logic understood: > - if one of underline devices are non rotational > - Queue length to underline devic

Re: [PATCH v15.1 03/13] btrfs: dedupe: Introduce function to add hash into in-memory tree

__inmem_del(dedupe_info, last); > + } > +out: > + mutex_unlock(_info->lock); > + return 0; > +} > + > +int btrfs_dedupe_add(struct btrfs_fs_info *fs_info, > +struct btrfs_dedupe_hash *hash) > +{ > + struct btrfs_dedupe_in

Re: [PATCH v15.1 02/13] btrfs: dedupe: Introduce function to initialize dedupe info

; +static inline int btrfs_dedupe_hash_size(u16 algo) > +{ > + if (WARN_ON(algo >= ARRAY_SIZE(btrfs_hash_sizes))) > + return -EINVAL; > + return sizeof(struct btrfs_dedupe_hash) + btrfs_hash_sizes[algo]; > +} > + > +static inline struct btrfs_dedupe_hash *btrfs_dedupe_alloc_hash(u16 algo) > +{ > + return kzalloc(btrfs_dedupe_hash_size(algo), GFP_NOFS); > +} > + > /* > * Initial inband dedupe info > * Called at dedupe enable time. > diff --git a/include/uapi/linux/btrfs.h b/include/uapi/linux/btrfs.h > index 9cd15d2a40aa..ba879ac931f2 100644 > --- a/include/uapi/linux/btrfs.h > +++ b/include/uapi/linux/btrfs.h > @@ -683,6 +683,9 @@ struct btrfs_ioctl_get_dev_stats { > /* Hash algorithm, only support SHA256 yet */ > #define BTRFS_DEDUPE_HASH_SHA256 0 > > +/* Default dedupe limit on number of hash */ > +#define BTRFS_DEDUPE_LIMIT_NR_DEFAULT (32 * 1024) > + > /* > * This structure is used for dedupe enable/disable/configure > * and status ioctl. > -- > 2.19.1 > > > Reviewed-by: Timofey Titovets Thanks. -- Have a nice day, Timofey.

Re: [PATCH v15.1 01/13] btrfs: dedupe: Introduce dedupe framework and its header

+ * and status ioctl. > + * Reserved range should be set to 0xff. > + */ > +struct btrfs_ioctl_dedupe_args { > + __u16 cmd; /* In: command */ > + __u64 blocksize; /* In/Out: blocksize */ > + __u64 limit_nr; /* In/Out: limit nr for inmem backend */ > + __u64 limit_mem;/* In/Out: limit mem for inmem backend */ > + __u64 current_nr; /* Out: current hash nr */ > + __u16 backend; /* In/Out: current backend */ > + __u16 hash_algo;/* In/Out: hash algorithm */ > + u8 status; /* Out: enabled or disabled */ > + u8 flags; /* In: special flags for ioctl */ > + u8 __unused[472]; /* Pad to 512 bytes */ > +}; > + > #define BTRFS_QUOTA_CTL_ENABLE 1 > #define BTRFS_QUOTA_CTL_DISABLE2 > #define BTRFS_QUOTA_CTL_RESCAN__NOTUSED3 > -- > 2.19.1 Reviewed-by: Timofey Titovets Thanks. -- Have a nice day, Timofey.

Re: [PATCH v2] btrfs: add zstd compression level support

+static struct list_head *zstd_alloc_workspace(unsigned int level) > { > ZSTD_parameters params = > - zstd_get_btrfs_parameters(ZSTD_BTRFS_MAX_INPUT); > + zstd_get_btrfs_parameters(ZSTD_BTRFS_MAX_INPUT, > level); > struct workspace *workspace; > > workspace = kzalloc(sizeof(*workspace), GFP_KERNEL); > @@ -69,6 +109,7 @@ static struct list_head *zstd_alloc_workspace(void) > goto fail; > > INIT_LIST_HEAD(>list); > + zstd_set_level(>list, level); > > return >list; > fail: > @@ -95,7 +136,8 @@ static int zstd_compress_pages(struct list_head *ws, > unsigned long len = *total_out; > const unsigned long nr_dest_pages = *out_pages; > unsigned long max_out = nr_dest_pages * PAGE_SIZE; > - ZSTD_parameters params = zstd_get_btrfs_parameters(len); > + ZSTD_parameters params = zstd_get_btrfs_parameters(len, > + workspace->level); > > *out_pages = 0; > *total_out = 0; > @@ -419,15 +461,13 @@ static int zstd_decompress(struct list_head *ws, > unsigned char *data_in, > return ret; > } > > -static void zstd_set_level(struct list_head *ws, unsigned int type) > -{ > -} > - > const struct btrfs_compress_op btrfs_zstd_compress = { > - .alloc_workspace = zstd_alloc_workspace, > - .free_workspace = zstd_free_workspace, > - .compress_pages = zstd_compress_pages, > - .decompress_bio = zstd_decompress_bio, > - .decompress = zstd_decompress, > - .set_level = zstd_set_level, > + .alloc_workspace= zstd_alloc_workspace, > + .free_workspace = zstd_free_workspace, > + .compress_pages = zstd_compress_pages, > + .decompress_bio = zstd_decompress_bio, > + .decompress = zstd_decompress, > + .set_level = zstd_set_level, > + .max_level = BTRFS_ZSTD_MAX_LEVEL, > + .default_level = BTRFS_ZSTD_DEFAULT_LEVEL, > }; > -- > 2.17.1 Reviewed-by: Timofey Titovets You didn't mention, so: Did you test compression ratio/performance with compress-force or just compress? Thanks. -- Have a nice day, Timofey.

Re: [PATCH RESEND] Btrfs: make should_defrag_range() understood compressed extents

вт, 18 сент. 2018 г. в 13:09, Timofey Titovets : > > From: Timofey Titovets > > Both, defrag ioctl and autodefrag - call btrfs_defrag_file() > for file defragmentation. > > Kernel default target extent size - 256KiB. > Btrfs progs default - 32MiB. > >

[PATCH V6] Btrfs: enhance raid1/10 balance heuristic

-> v4: - Rebased on latest misc-next v4 -> v5: - Rebased on latest misc-next v5 -> v6: - Fix spelling - Include bench results Signed-off-by: Timofey Titovets Tested-by: Dmitrii Tcvetkov Reviewed-by: Dmitrii Tcvetkov --- block/genhd.c | 1 + fs/btrfs/

Re: [PATCH V5 RESEND] Btrfs: enchanse raid1/10 balance heuristic

чт, 20 сент. 2018 г. в 12:05, Peter Becker : > > i like the idea. > do you have any benchmarks for this change? > > the general logic looks good for me. https://patchwork.kernel.org/patch/10137909/ > > Tested-by: Dmitrii Tcvetkov > > Benchmark summary (arithmetic mean of 3 runs): > Mainline

[PATCH V5 RESEND] Btrfs: enchanse raid1/10 balance heuristic

From: Timofey Titovets Currently btrfs raid1/10 balancer bаlance requests to mirrors, based on pid % num of mirrors. Make logic understood: - if one of underline devices are non rotational - Queue leght to underline devices By default try use pid % num_mirrors guessing, but: - If one

[PATCH RESEND] Btrfs: make should_defrag_range() understood compressed extents

From: Timofey Titovets Both, defrag ioctl and autodefrag - call btrfs_defrag_file() for file defragmentation. Kernel default target extent size - 256KiB. Btrfs progs default - 32MiB. Both bigger then maximum size of compressed extent - 128KiB. That lead to rewrite all compressed data

Re: [PATCH V5] Btrfs: enchanse raid1/10 balance heuristic

сб, 7 июл. 2018 г. в 18:24, Timofey Titovets : > > From: Timofey Titovets > > Currently btrfs raid1/10 balancer bаlance requests to mirrors, > based on pid % num of mirrors. > > Make logic understood: > - if one of underline devices are non rotational > - Queue

Re: dduper - Offline btrfs deduplication tool

пт, 24 авг. 2018 г. в 7:41, Lakshmipathi.G : > > Hi - > > dduper is an offline dedupe tool. Instead of reading whole file blocks and > computing checksum, It works by fetching checksum from BTRFS csum tree. This > hugely improves the performance. > > dduper works like: > - Read csum for

[PATCH V5] Btrfs: enchanse raid1/10 balance heuristic

From: Timofey Titovets Currently btrfs raid1/10 balancer bаlance requests to mirrors, based on pid % num of mirrors. Make logic understood: - if one of underline devices are non rotational - Queue leght to underline devices By default try use pid % num_mirrors guessing, but: - If one

[PATCH RESEND V4] Btrfs: enchanse raid1/10 balance heuristic

From: Timofey Titovets Currently btrfs raid1/10 balancer bаlance requests to mirrors, based on pid % num of mirrors. Make logic understood: - if one of underline devices are non rotational - Queue leght to underline devices By default try use pid % num_mirrors guessing, but: - If one

Re: [PATCH 2/4] Btrfs: make should_defrag_range() understood compressed extents

вт, 19 дек. 2017 г. в 13:02, Timofey Titovets : > Both, defrag ioctl and autodefrag - call btrfs_defrag_file() > for file defragmentation. > Kernel default target extent size - 256KiB. > Btrfs progs default - 32MiB. > Both bigger then maximum size of compressed

Re: Any chance to get snapshot-aware defragmentation?

пн, 21 мая 2018 г. в 16:16, Austin S. Hemmelgarn : > On 2018-05-19 04:54, Niccolò Belli wrote: > > On venerdì 18 maggio 2018 20:33:53 CEST, Austin S. Hemmelgarn wrote: > >> With a bit of work, it's possible to handle things sanely. You can > >> deduplicate data from

Re: [PATCH] btrfs: inode: Don't compress if NODATASUM or NODATACOW set

пн, 14 мая 2018 г. в 20:32, David Sterba : > On Mon, May 14, 2018 at 03:02:10PM +0800, Qu Wenruo wrote: > > As btrfs(5) specified: > > > > Note > > If nodatacow or nodatasum are enabled, compression is disabled. > > > > If NODATASUM or NODATACOW set, we should not

Re: [PATCH] btrfs: use kvzalloc for EXTENT_SAME temporary data

пт, 11 мая 2018 г. в 20:32, Omar Sandoval : > On Fri, May 11, 2018 at 06:49:16PM +0200, David Sterba wrote: > > On Fri, May 11, 2018 at 05:25:50PM +0100, Filipe Manana wrote: > > > On Fri, May 11, 2018 at 4:57 PM, David Sterba wrote: > > > > The dedupe

[RFC PATCH] raid6_pq: Add module options to prefer algorithm

[1.246684] raid6: using algorithm avx2x4 gen() 28153 MB/s [1.246684] raid6: xor() 19440 MB/s, rmw enabled [1.246685] raid6: using avx2x2 recovery algorithm Signed-off-by: Timofey Titovets <nefelim...@gmail.com> CC: linux-btrfs@vger.kernel.org --- lib/raid6/algos.

[PATCH V3 3/3] Btrfs: btrfs_extent_same() reuse cmp workspace

We support big dedup requests by split range to several smaller, and call dedup logic over each of them. Instead of alloc/dealloc on each, let's reuse allocated memory. Changes: v3: - Splited from one to 3 patches Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs

[PATCH V3 2/3] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

anything for request less then 16MiB. Changes: v3: - Splited from one to 3 patches Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/ioctl.c | 25 +++-- 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c

[PATCH V3 1/3] Btrfs: split btrfs_extent_same() for simplification

Split btrfs_extent_same() for simplification and preparation for call several times over target files Move most logic to __btrfs_extent_same() And leave in btrfs_extent_same() things which must happens only once Changes: v3: - Splited from one to 3 patches Signed-off-by: Timofey Titovets

[PATCH V3 0/3] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

anything for request less then 16MiB. Changes: v1 -> v2: - Refactor btrfs_cmp_data_prepare and btrfs_extent_same - Store memory of pages array between iterations - Lock inodes once, not on each iteration - Small inplace cleanups v2 -> v3: - Split to several patches T

Re: [PATCH 4/4] [RESEND] Btrfs: reduce size of struct btrfs_inode

чт, 26 апр. 2018 г. в 16:44, David Sterba <dste...@suse.cz>: > On Wed, Apr 25, 2018 at 02:37:17AM +0300, Timofey Titovets wrote: > > Currently btrfs_inode have size equal 1136 bytes. (On x86_64). > > > > struct btrfs_inode store several vars releated to compression code

Re: [PATCH 1/4] [RESEND] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

чт, 26 апр. 2018 г. в 17:05, David Sterba <dste...@suse.cz>: > On Wed, Apr 25, 2018 at 02:37:14AM +0300, Timofey Titovets wrote: > > At now btrfs_dedupe_file_range() restricted to 16MiB range for > > limit locking time and memory requirement for dedup ioctl() > > >

Re: [PATCH V4] Btrfs: enchanse raid1/10 balance heuristic

2018-04-25 10:54 GMT+03:00 Misono Tomohiro <misono.tomoh...@jp.fujitsu.com>: > On 2018/04/25 9:20, Timofey Titovets wrote: >> Currently btrfs raid1/10 balancer bаlance requests to mirrors, >> based on pid % num of mirrors. >> >> Make logic understood: >>

[PATCH V4] Btrfs: enchanse raid1/10 balance heuristic

nstead of num_stripes v3 -> v4: - Rebased on latest misc-next Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- block/genhd.c | 1 + fs/btrfs/volumes.c | 111 - 2 files changed, 110 insertions(+), 2 deletions(-) diff --git a/block/

[PATCH 2/4] [RESEND] Btrfs: make should_defrag_range() understood compressed extents

-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/ioctl.c | 28 +--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 45a47d0891fc..b29ea1f0f621 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/i

[PATCH 4/4] [RESEND] Btrfs: reduce size of struct btrfs_inode

Currently btrfs_inode have size equal 1136 bytes. (On x86_64). struct btrfs_inode store several vars releated to compression code, all states use 1 or 2 bits. Lets declare bitfields for compression releated vars, to reduce sizeof btrfs_inode to 1128 bytes. Signed-off-by: Timofey Titovets

[PATCH 1/4] [RESEND] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

anything for request less then 16MiB. Changes: v1 -> v2: - Refactor btrfs_cmp_data_prepare and btrfs_extent_same - Store memory of pages array between iterations - Lock inodes once, not on each iteration - Small inplace cleanups Signed-off-by: Timofey Titovets <nefelim...@gma

[PATCH 0/4] [RESEND] Btrfs: just bunch of patches to ioctl.c

of 2^32-1. So, set a bitfields on that vars, and reduce size of btrfs_inode: 1136 -> 1128. Timofey Titovets (4): Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction Btrfs: make should_defrag_range() understood compressed extents Btrfs: allow btrfs_defrag_file() uncompr

[PATCH 3/4] [RESEND] Btrfs: allow btrfs_defrag_file() uncompress files on defragmentation

Currently defrag ioctl only support recompress files with specified compression type. Allow set compression type to none, while call defrag, and use BTRFS_DEFRAG_RANGE_COMPRESS as flag, that user request change of compression type. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> -

Re: Recovery from full metadata with all device space consumed?

2018-04-20 1:08 GMT+03:00 Drew Bloechl : > I've got a btrfs filesystem that I can't seem to get back to a useful > state. The symptom I started with is that rename() operations started > dying with ENOSPC, and it looks like the metadata allocation on the > filesystem is full: >

Re: [PATCH V3] Btrfs: enchanse raid1/10 balance heuristic

Gentle ping. 2018-01-03 0:23 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > 2018-01-02 21:31 GMT+03:00 Liu Bo <bo.li@oracle.com>: >> On Sat, Dec 30, 2017 at 11:32:04PM +0300, Timofey Titovets wrote: >>> Currently btrfs raid1/10 balancer bаlance request

Re: [PATCH 0/4] Btrfs: just bunch of patches to ioctl.c

Gentle ping 2018-01-09 13:53 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > Gentle ping > > 2017-12-19 13:02 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: >> 1st patch, remove 16MiB restriction from extent_same ioctl(), >> by doing iterations over passe

Re: invalid files names, btrfs check can't repair it

2018-01-13 0:04 GMT+03:00 Sebastian Andrzej Siewior : > Hi, > > so I had bad memory and before I realized it and removed it btrfs took some > damage. Now I have this: > > |ls -lh crap/ > |ls: cannot access 'crap/2f3f379b2a3d7499471edb74869efe-1948311.d': No such > file or

Re: btrfs subvolume mount with different options

2018-01-12 20:49 GMT+03:00 Konstantin V. Gavrilenko : > Hi list, > > just wondering whether it is possible to mount two subvolumes with different > mount options, i.e. > > | > |- /a defaults,compress-force=lza > | > |- /b defaults,nodatacow > > > since, when both

Re: Recommendations for balancing as part of regular maintenance?

2018-01-10 21:33 GMT+03:00 Tom Worster : > On 10 Jan 2018, at 12:01, Austin S. Hemmelgarn wrote: > >> On 2018-01-10 11:30, Tom Worster wrote: >> >> Also, for future reference, the term we typically use is ENOSPC, as that's >> the symbolic name for the error code you get when this

Re: [PATCH 0/4] Btrfs: just bunch of patches to ioctl.c

Gentle ping 2017-12-19 13:02 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > 1st patch, remove 16MiB restriction from extent_same ioctl(), > by doing iterations over passed range. > > I did not see much difference in performance, so it's just remove > logic restri

Re: [PATCH] generic/015: Change the test filesystem size to 101mb

2018-01-08 15:54 GMT+03:00 Qu Wenruo : > > > On 2018年01月08日 16:43, Nikolay Borisov wrote: >> This test has been failing for btrfs for quite some time, >> at least since 4.7. There are 2 implementation details of btrfs that >> it exposes: >> >> 1. Currently btrfs filesystem

Re: [PATCH 0/2] Remove custom crc32c init code from btrfs

sertions(+), 126 deletions(-) > delete mode 100644 fs/btrfs/hash.c > delete mode 100644 fs/btrfs/hash.h > > -- > 2.7.4 > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > Mor

Re: [PATCH 1/4] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

2017-12-20 0:23 GMT+03:00 Darrick J. Wong <darrick.w...@oracle.com>: > On Tue, Dec 19, 2017 at 01:02:44PM +0300, Timofey Titovets wrote: >> At now btrfs_dedupe_file_range() restricted to 16MiB range for >> limit locking time and memory requirement for dedup ioctl() >>

Re: [PATCH] Btrfs: replace raid56 stripe bubble sort with insert sort

2018-01-03 14:40 GMT+03:00 Filipe Manana <fdman...@gmail.com>: > On Thu, Dec 28, 2017 at 3:28 PM, Timofey Titovets <nefelim...@gmail.com> > wrote: >> Insert sort are generaly perform better then bubble sort, >> by have less iterations on avarage. >> That ver

Re: [PATCH V3] Btrfs: enchanse raid1/10 balance heuristic

2018-01-02 21:31 GMT+03:00 Liu Bo <bo.li@oracle.com>: > On Sat, Dec 30, 2017 at 11:32:04PM +0300, Timofey Titovets wrote: >> Currently btrfs raid1/10 balancer bаlance requests to mirrors, >> based on pid % num of mirrors. >> >> Make logic understood: >>

Re: [PATCH 1/2] Btrfs: heuristic: replace workspace managment code by mempool API

2017-12-24 7:55 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > Currently compression code have custom workspace/memory cache > for guarantee forward progress on high memory pressure. > > That api can be replaced with mempool API, which can guarantee the same. > Main goa

[PATCH V3] Btrfs: enchanse raid1/10 balance heuristic

nstead of num_stripes Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- block/genhd.c | 1 + fs/btrfs/volumes.c | 115 - 2 files changed, 114 insertions(+), 2 deletions(-) diff --git a/block/genhd.c b/block/genhd.c index 96a66f671720..

Re: [PATCH v2] Btrfs: enchanse raid1/10 balance heuristic

2017-12-30 11:14 GMT+03:00 Dmitrii Tcvetkov <demfl...@demfloro.ru>: > On Sat, 30 Dec 2017 03:15:20 +0300 > Timofey Titovets <nefelim...@gmail.com> wrote: > >> 2017-12-29 22:14 GMT+03:00 Dmitrii Tcvetkov <demfl...@demfloro.ru>: >> > On Fri, 29 Dec 2017 2

Re: [PATCH v2] Btrfs: enchanse raid1/10 balance heuristic

2017-12-29 22:14 GMT+03:00 Dmitrii Tcvetkov : > On Fri, 29 Dec 2017 21:44:19 +0300 > Dmitrii Tcvetkov wrote: >> > +/** >> > + * guess_optimal - return guessed optimal mirror >> > + * >> > + * Optimal expected to be pid % num_stripes >> > + * >> > + *

[PATCH v2] Btrfs: enchanse raid1/10 balance heuristic

rom genhd.c to get queue lenght - Move guess code to guess_optimal() - Change balancer logic, try use pid % mirror by default Make balancing on spinning rust if one of underline devices are overloaded Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- bloc

Re: [PATCH] Btrfs: enchanse raid1/10 balance heuristic for non rotating devices

2017-12-28 11:06 GMT+03:00 Dmitrii Tcvetkov <demfl...@demfloro.ru>: > On Thu, 28 Dec 2017 01:39:31 +0300 > Timofey Titovets <nefelim...@gmail.com> wrote: > >> Currently btrfs raid1/10 balancer blance requests to mirrors, >> based on pid % num of mirrors. >>

[PATCH] Btrfs: replace raid56 stripe bubble sort with insert sort

of a great justice. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/volumes.c | 29 - 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index 98bc2433a920..7195fc8c49b1 100644 --- a/fs

[PATCH 2/2] Btrfs: compression: replace workspace managment code by mempool API

. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/compression.c | 213 + fs/btrfs/compression.h | 12 +-- fs/btrfs/lzo.c | 64 +-- fs/btrfs/zlib.c| 56 +++-- fs/btrfs/zstd.c

[PATCH 0/2] Btrfs: heuristic/compression convert workspace memory cache

, heuristic memory managment. Define usage pattern and mempool_alloc_wrap() - handle pool resize and pool init errors. Second move zlib/lzo/zstd to new mempool API Timofey Titovets (2): Btrfs: heuristic: replace workspace managment code by mempool API Btrfs: compression: replace workspace managment

[PATCH 1/2] Btrfs: heuristic: replace workspace managment code by mempool API

in call chain. - mempool_create return pointer to mampool or NULL, no error, so macros like IS_ERR(ptr) can't be used. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/compression.c | 197 ++--- 1 file changed, 106 inse

[RFC PATCH] Btrfs: replace custom heuristic ws allocation logic with mempool API

in __init. 2. Convert per compression ws to mempool. Thanks. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> Cc: David Sterba <dste...@suse.com> --- fs/btrfs/compression.c | 123 - 1 file changed, 39 insertions(+), 84 deleti

Btrfs allow compression on NoDataCow files? (AFAIK Not, but it does)

How reproduce: touch test_file chattr +C test_file dd if=/dev/zero of=test_file bs=1M count=1 btrfs fi def -vrczlib test_file filefrag -v test_file test_file Filesystem type is: 9123683e File size of test_file is 1048576 (256 blocks of 4096 bytes) ext: logical_offset:physical_offset:

[PATCH 1/4] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

anything for request less then 16MiB. Changes: v1 -> v2: - Refactor btrfs_cmp_data_prepare and btrfs_extent_same - Store memory of pages array between iterations - Lock inodes once, not on each iteration - Small inplace cleanups Signed-off-by: Timofey Titovets <nefelim...@gma

[PATCH 3/4] Btrfs: allow btrfs_defrag_file() uncompress files on defragmentation

Currently defrag ioctl only support recompress files with specified compression type. Allow set compression type to none, while call defrag, and use BTRFS_DEFRAG_RANGE_COMPRESS as flag, that user request change of compression type. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> -

[PATCH 2/4] Btrfs: make should_defrag_range() understood compressed extents

-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/ioctl.c | 28 +--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 45a47d0891fc..b29ea1f0f621 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/i

[PATCH 0/4] Btrfs: just bunch of patches to ioctl.c

of 2^32-1. So, set a bitfields on that vars, and reduce size of btrfs_inode: 1136 -> 1128. Timofey Titovets (4): Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction Btrfs: make should_defrag_range() understood compressed extents Btrfs: allow btrfs_defrag_file() uncompr

[PATCH 4/4] Btrfs: reduce size of struct btrfs_inode

Currently btrfs_inode have size equal 1136 bytes. (On x86_64). struct btrfs_inode store several vars releated to compression code, all states use 1 or 2 bits. Lets declare bitfields for compression releated vars, to reduce sizeof btrfs_inode to 1128 bytes. Signed-off-by: Timofey Titovets

[PATCH] Btrfs: allow btrfs_defrag_file() uncompress files on defragmentation

Currently defrag ioctl only support compress files with specified compression type. Allow set compression type to none, while call defrag. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/btrfs_inode.h | 1 + fs/btrfs/inode.c | 4 ++-- fs/btrfs/ioctl.c

Re: [PATCH] Btrfs: make should_defrag_range() understood compressed extents

inode_need_compression() logic. In theory that also must be fixed. 2017-12-14 16:37 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > Compile tested and "battle" tested > > 2017-12-14 16:35 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: >> Both, defrag ioctl an

Re: [BUG?] Defrag on compressed FS do massive data rewrites

Titovets <nefelim...@gmail.com>: > 2017-12-14 8:58 GMT+03:00 Duncan <1i5t5.dun...@cox.net>: >> Timofey Titovets posted on Thu, 14 Dec 2017 02:05:35 +0300 as excerpted: >> >>> Also, same problem exist for autodefrag case i.e.: >>> write 4KiB at start of

Re: [PATCH] Btrfs: make should_defrag_range() understood compressed extents

Compile tested and "battle" tested 2017-12-14 16:35 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > Both, defrag ioctl and autodefrag - call btrfs_defrag_file() > for file defragmentation. > > Kernel target extent size default is 256KiB > Btrfs progs by defa

[PATCH] Btrfs: make should_defrag_range() understood compressed extents

that and also make should_defrag_range() understood if requested target compression are same as current extent compression type. To avoid useless recompression of compressed extents. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/ioctl.c | 28 +---

Re: [RFC PATCH] Btrfs: btrfs_defrag_file() force use target extent size SZ_128KiB for compressed data

Ignore that patch please, i will send another 2017-12-14 2:25 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > Defrag heuristic use extent lengh as threshold, > kernel autodefrag use SZ_256KiB and btrfs-progs use SZ_32MiB as > target extent lengh. > > Problem: > Compre

Re: [BUG?] Defrag on compressed FS do massive data rewrites

2017-12-14 8:58 GMT+03:00 Duncan <1i5t5.dun...@cox.net>: > Timofey Titovets posted on Thu, 14 Dec 2017 02:05:35 +0300 as excerpted: > >> Also, same problem exist for autodefrag case i.e.: >> write 4KiB at start of compressed file autodefrag code add that file to &

[RFC PATCH] Btrfs: btrfs_defrag_file() force use target extent size SZ_128KiB for compressed data

fix that by force set target extent size to BTRFS_MAX_COMPRESSED, if file allowed to be compressed. Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/ioctl.c | 23 +++ 1 file changed, 23 insertions(+) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index be5

Re: [BUG?] Defrag on compressed FS do massive data rewrites

2017-12-14 1:09 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > Hi, i see massive data rewrites of defragmented files when work with > btrfs fi def . > Before, i just thought it's a design problem - i.e. defrag always > rewrite data to new place. > > At now, i rea

[BUG?] Defrag on compressed FS do massive data rewrites

Hi, i see massive data rewrites of defragmented files when work with btrfs fi def . Before, i just thought it's a design problem - i.e. defrag always rewrite data to new place. At now, i read the code and see 2 bad cases: 1. With -c all extents of data will be rewriten, always. 2. btrfs use "bad"

Re: [PATCH v2] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

Compile tested && battle tested by btrfs-extent-same from duperemove. At performance, i see a negligible difference. Thanks 2017-12-13 3:45 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > At now btrfs_dedupe_file_range() restricted to 16MiB range for > limit loc

Re: [PATCH v3 0/5] define BTRFS_DEV_STATE

2017-12-13 5:26 GMT+03:00 David Sterba : > On Wed, Dec 13, 2017 at 06:38:12AM +0800, Anand Jain wrote: >> >> >> On 12/13/2017 01:42 AM, David Sterba wrote: >> > On Sun, Dec 10, 2017 at 05:15:17PM +0800, Anand Jain wrote: >> >> As of now device properties and states are being

[PATCH v2] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

anything for request less then 16MiB. Changes: v1 -> v2: - Refactor btrfs_cmp_data_prepare and btrfs_extent_same - Store memory of pages array between iterations - Lock inodes once, not on each iteration - Small inplace cleanups Signed-off-by: Timofey Titovets <nefelim...@gma

Re: [PATCH 0/3] Minor compression heuristic cleanups

> 1 file changed, 11 insertions(+), 31 deletions(-) > > -- > 2.15.1 > Thanks! On whole series: Reviewed-by: Timofey Titovets -- Have a nice day, Timofey. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord..

Re: defragmenting best practice?

2017-12-11 8:18 GMT+03:00 Dave : > On Tue, Oct 31, 2017 someone wrote: >> >> >> > 2. Put $HOME/.cache on a separate BTRFS subvolume that is mounted >> > nocow -- it will NOT be snapshotted > > I did exactly this. It servers the purpose of avoiding snapshots. > However,

Just curious, whats happened with btrfs & rcu skiplist in 2013?

Subj, I found that https://lwn.net/Articles/554885/, https://lwn.net/Articles/553047/ and some others messages like judy rcu & etc. But nothing after june 2013, just curious, may be some one know why that has been stalled? or just what happened in the end? (i.e. ex, RT guys, just said that too

[PATCH v3] Btrfs: heuristic replace heap sort with radix sort

restrict input array size to u32 Assume that kernel will never try sorting arrays > 2^32 - Drop max_cell arg (precheck - correctly find max value by it self) Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/compression.c | 135

Re: [PATCH v2] Btrfs: heuristic replace heap sort with radix sort

2017-12-05 0:24 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > 2017-12-04 23:47 GMT+03:00 David Sterba <dste...@suse.cz>: >> On Mon, Dec 04, 2017 at 12:30:33AM +0300, Timofey Titovets wrote: >>> Slowest part of heuristic for now is kernel heap sort() >>

Re: [PATCH v2] Btrfs: heuristic replace heap sort with radix sort

2017-12-04 23:47 GMT+03:00 David Sterba <dste...@suse.cz>: > On Mon, Dec 04, 2017 at 12:30:33AM +0300, Timofey Titovets wrote: >> Slowest part of heuristic for now is kernel heap sort() >> It's can take up to 55% of runtime on sorting bucket items. >> >> As sort

Re: [PATCH v4] Btrfs: compress_file_range() change page dirty status once

Gentle ping 2017-10-24 1:29 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > We need to call extent_range_clear_dirty_for_io() > on compression range to prevent application from changing > page content, while pages compressing. > > extent_range_clear_dirty_for_io() run o

[PATCH v2] Btrfs: heuristic replace heap sort with radix sort

- average <-> random data: ~6000 MiB/s - radix sort Changes: v1 -> v2: - Tested on Big Endian - Drop most of multiply operations - Separately allocate sort buffer Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/b

Re: How about adding an ioctl to convert a directory to a subvolume?

2017-11-28 21:48 GMT+03:00 David Sterba : > On Mon, Nov 27, 2017 at 05:41:56PM +0800, Lu Fengqi wrote: >> As we all know, under certain circumstances, it is more appropriate to >> create some subvolumes rather than keep everything in the same >> subvolume. As the condition of

Re: zstd compression

2017-11-16 19:32 GMT+03:00 Austin S. Hemmelgarn : > On 2017-11-16 08:43, Duncan wrote: >> >> Austin S. Hemmelgarn posted on Thu, 16 Nov 2017 07:30:47 -0500 as >> excerpted: >> >>> On 2017-11-15 16:31, Duncan wrote: Austin S. Hemmelgarn posted on Wed, 15 Nov 2017

Re: [PATCH 4/4] Btrfs: btrfs_dedupe_file_range() ioctl, remove 16MiB restriction

4PM +0300, Timofey Titovets wrote: >> At now btrfs_dedupe_file_range() restricted to 16MiB range for >> limit locking time and memory requirement for dedup ioctl() >> >> For too big input rage code silently set range to 16MiB >> >> Let's remove that restriction by do i

Re: raid1: can't readd removed dev while the fs is mounted

2017-10-28 1:40 GMT+03:00 Julien Muchembled : > Hello, > > I have 2 disks in RAID1, each one having 2 partitions: > - 1 for / (BtrFS) > - 1 for /home (MD/XFS) > > For some reasons, 1 disk was removed and readded. I had no issue at readding > it to the MD array, but for BtrFS, I

[PATCH v4] Btrfs: compress_file_range() change page dirty status once

_bytes - Update change log - Add comments v3 -> v4: - Rebased on: kdave for-next - To avoid dirty bit clear/set behaviour change call clear_bit once, istead of per compression range Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/inode.c | 6 -- 1 file ch

Re: [PATCH v8 0/6] Btrfs: populate heuristic with code

2017-10-22 16:44 GMT+03:00 Timofey Titovets <nefelim...@gmail.com>: > 2017-10-20 16:45 GMT+03:00 David Sterba <dste...@suse.cz>: >> On Fri, Oct 20, 2017 at 01:48:01AM +0300, Timofey Titovets wrote: >>> 2017-10-19 18:39 GMT+03:00 David Sterba <dste...@suse.cz>:

Re: [PATCH v8 0/6] Btrfs: populate heuristic with code

2017-10-20 16:45 GMT+03:00 David Sterba <dste...@suse.cz>: > On Fri, Oct 20, 2017 at 01:48:01AM +0300, Timofey Titovets wrote: >> 2017-10-19 18:39 GMT+03:00 David Sterba <dste...@suse.cz>: >> > On Fri, Sep 29, 2017 at 06:22:00PM +0200, David Sterba wrote: >> &

Re: [PATCH v8 0/6] Btrfs: populate heuristic with code

2017-10-19 18:39 GMT+03:00 David Sterba <dste...@suse.cz>: > On Fri, Sep 29, 2017 at 06:22:00PM +0200, David Sterba wrote: >> On Thu, Sep 28, 2017 at 05:33:35PM +0300, Timofey Titovets wrote: >> > Compile tested, hand tested on live system >> > >> &g

Re: [PATCH 3/4] Btrfs: handle unaligned tail of data ranges more efficient

assume that everybody who look at kernel code, must understood that basic things? Thanks 2017-10-10 19:37 GMT+03:00 David Sterba <dste...@suse.cz>: > On Tue, Oct 03, 2017 at 06:06:03PM +0300, Timofey Titovets wrote: >> At now while switch page bits in data ranges >> we

Re: [PATCH 2/4] Btrfs: clear_dirty only on pages only in compression range

2017-10-10 19:22 GMT+03:00 David Sterba <dste...@suse.cz>: > On Tue, Oct 03, 2017 at 06:06:02PM +0300, Timofey Titovets wrote: >> We need to call extent_range_clear_dirty_for_io() >> on compression range to prevent application from changing >> page content, while pages c

[RFC PATCH] Btrfs: heuristic replace heap sort with radix sort

~3500 MiB/s - heap sort - average <-> random data: ~6000 MiB/s +71% - radix sort Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/compression.c | 153 + 1 file changed, 141 insertions(+), 12 deletions(-)

Re: [RFC] Btrfs: compression heuristic performance

Just info update. I did some more benchmark, optimization and testing. (Write kernel generalized version of radix sort (that can be ported to sort.c)) (Some memory allocating changes & etc) (Also i'm stupid and make a mistake with inversion of percentage numbers) New, more clean numbers (I

[RFC] Btrfs: compression heuristic performance

Hi David, (and list folks in CC of course). TL;DR Sorting is a slowest part of heuristic for now (in worst case ~55% of run time). Kernel use heap sort, i.e. heuristic also use that. Radix sort speedup heuristic a lot (~+30-40%). But radix have some tradeoffs (i.e. need more memory, but that can

[PATCH v2] Btrfs: heuristic add shannon entropy calculation

L_HIGH 85 -> 80 - Drop usage of u64 from shannon calculation Signed-off-by: Timofey Titovets <nefelim...@gmail.com> --- fs/btrfs/compression.c | 83 +- 1 file changed, 82 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/compressi

  1   2   3   4   >