First of all.
The "crossing stripe boundary" error message itself is *HARMLESS* for
recent kernels.
It only means, that metadata extent won't be checked by scrub on recent
kernels.
Because scrub by its codes, has a limitation that, it can only check
tree blocks which are inside a 64K block.
On Wed, Mar 30, 2016 at 03:00:19PM +0800, Qu Wenruo wrote:
> Marc Haber wrote on 2016/03/29 08:43 +0200:
> >On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
> >>Did you convert this filesystem from ext4 (or ext3)?
> >
> >No.
> >
> >>You hadn't mentioned what version of btrfs-pr
Now on-disk backend should be able to search hash now.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedupe.c | 134 +++---
fs/btrfs/dedupe.h | 1 +
2 files changed, 118 insertions(+), 17 deletions(-)
diff --git a/fs/btrfs
From: Wang Xiaoguang
Add ordered-extent support for dedupe.
Note, current ordered-extent support only supports non-compressed source
extent.
Support for compressed source extent will be added later.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/ordered-data.c | 44 +
From: Wang Xiaoguang
Introduce BTRFS_INODE_NODEDUP flag, then we can explicitly disable
online data dedupelication for specified files.
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/ctree.h | 1 +
fs/btrfs/ioctl.c | 6 +-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/
From: Wang Xiaoguang
Introduce static function inmem_add() to add hash into in-memory tree.
And now we can implement the btrfs_dedupe_add() interface.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedupe.c | 151 ++
1 f
For in-band dedupe, btrfs needs to increase data ref with delayed_ref
locked, so add a new function btrfs_add_delayed_data_ref_lock() to
increase extent ref with delayed_refs already locked.
Signed-off-by: Qu Wenruo
---
fs/btrfs/delayed-ref.c | 30 +++---
fs/btrfs/delayed
From: Wang Xiaoguang
Introduce inode_need_dedupe() to implement per-file online dedupe control.
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/inode.c | 15 ++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 96790d0..c80fd74 1006
This patchset can be fetched from github:
https://github.com/adam900710/linux.git wang_dedupe_20160330
This March 30th patchset update mostly addresses the patchset structure
comment from David:
1) Change the patchset sequence
Not If only apply the first 14 patches, it can provide the full
b
Since we will introduce a new on-disk based dedupe method, introduce new
interfaces to resume previous dedupe setup.
And since we introduce a new tree for status, also add disable handler
for it.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedupe.c | 197 ++
Introduce a new tree, dedupe tree to record on-disk dedupe hash.
As a persist hash storage instead of in-memeory only implement.
Unlike Liu Bo's implement, in this version we won't do hack for
bytenr -> hash search, but add a new type, DEDUP_BYTENR_ITEM for such
search case, just like in-memory ba
Now on-disk backend can delete hash now.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedupe.c | 100 ++
1 file changed, 100 insertions(+)
diff --git a/fs/btrfs/dedupe.c b/fs/btrfs/dedupe.c
index f2c2dde..19fe5ee 100644
Core implement for inband de-duplication.
It reuse the async_cow_start() facility to do the calculate dedupe hash.
And use dedupe hash to do inband de-duplication at extent level.
The work flow is as below:
1) Run delalloc range for an inode
2) Calculate hash for the delalloc range at the unit of
From: Wang Xiaoguang
We use btrfs extended attribute "btrfs.dedupe" to record per-file online
dedupe status, so add a dedupe property handler.
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/props.c | 41 +
1 file changed, 41 insertions(+)
diff --git a/fs/bt
From: Wang Xiaoguang
Introduce static function inmem_del() to remove hash from in-memory
dedupe tree.
And implement btrfs_dedupe_del() and btrfs_dedup_destroy() interfaces.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedupe.c | 105 +
From: Wang Xiaoguang
Add ioctl interface for inband dedupelication, which includes:
1) enable
2) disable
3) status
And a pseudo RO compat flag, to imply that btrfs now supports inband
dedup.
However we don't add any ondisk format change, it's just a pseudo RO
compat flag.
All these ioctl interf
From: Wang Xiaoguang
Introduce static function inmem_search() to handle the job for in-memory
hash tree.
The trick is, we must ensure the delayed ref head is not being run at
the time we search the for the hash.
With inmem_search(), we can implement the btrfs_dedupe_search()
interface.
Signed-
From: Wang Xiaoguang
Introduce the header for btrfs online(write time) de-duplication
framework and needed header.
The new de-duplication framework is going to support 2 different dedupe
methods and 1 dedupe hash.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/ctree.h
From: Wang Xiaoguang
Unlike in-memory or on-disk dedupe method, only SHA256 hash method is
supported yet, so implement btrfs_dedupe_calc_hash() interface using
SHA256.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedupe.c | 49 +++
Now on-disk backend can add hash now.
Since all needed on-disk backend functions are added, also allow on-disk
backend to be used, by changing DEDUPE_BACKEND_COUNT from 1(inmemory
only) to 2 (inmemory + ondisk).
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedupe.c | 83
Marc Haber wrote on 2016/03/30 09:18 +0200:
On Wed, Mar 30, 2016 at 03:00:19PM +0800, Qu Wenruo wrote:
Marc Haber wrote on 2016/03/29 08:43 +0200:
On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
Did you convert this filesystem from ext4 (or ext3)?
No.
You hadn't men
From: Wang Xiaoguang
Add generic function to initialize dedupe info.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/dedupe.c | 154 ++
fs/btrfs/dedupe.h | 16 +-
3 files changed, 169 inse
For dedupe to work with compression, new members recording compression
algorithm and on-disk extent length are needed.
Add them for later compress-dedupe co-work.
Signed-off-by: Qu Wenruo
---
fs/btrfs/ctree.h| 11 -
fs/btrfs/dedupe.c | 64 ++
On Wed, Mar 30, 2016 at 04:03:17PM +0800, Qu Wenruo wrote:
> Did your btrfs have enough *unallocated* space?
87 Gig out of a total 200 Gig Device size. I guess that should be
enough for a rebalance of 2,8 Gig Metadata.
Greetings
Ma "please excuse my cynism" rc
--
---
subscribe linux-btrfs
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Filipe Manana
If we rename an inode A (be it a file or a directory), create a new
inode B with the old name of inode A and under the same parent directory,
at log tree replay time we end up removing inode A completely. If inode A
is a directory then all its files are gone too.
This is repr
From: Filipe Manana
Test that if we rename a file, create a new file that has the old name
of the other file and is a child of the same parent directory, fsync the
new inode, power fail and mount the filesystem, we do not lose the first
file and that file has the name it was renamed to.
This tes
From: Filipe Manana
Test that if we rename a directory, create a new file or directory that
has the old name of our former directory and is a child of the same
parent directory, fsync the new inode, power fail and mount the
filesystem, we see our first directory with the new name and no files
und
Hi Qu,
[auto build test ERROR on btrfs/next]
[also build test ERROR on v4.6-rc1 next-20160330]
[if your patch is applied to the wrong git tree, please drop us a note to help
improving the system]
url:
https://github.com/0day-ci/linux/commits/Qu-Wenruo/Btrfs-dedupe-framework/20160330-160940
On Thu, Mar 24, 2016 at 5:06 PM, wrote:
> From: Filipe Manana
>
> If we delete a snapshot, delete its parent directory, create a new
> directory with the same name as that parent and then fsync either that
> new directory or some file inside it, we end up with a log tree that
> is not possible t
On Wed, Mar 30, 2016 at 10:39 AM, wrote:
> From: Filipe Manana
>
> Test that if we rename a file, create a new file that has the old name
> of the other file and is a child of the same parent directory, fsync the
> new inode, power fail and mount the filesystem, we do not lose the first
> file a
Hi Wang,
[auto build test ERROR on btrfs/next]
[also build test ERROR on v4.6-rc1 next-20160330]
[if your patch is applied to the wrong git tree, please drop us a note to help
improving the system]
url:
https://github.com/0day-ci/linux/commits/Qu-Wenruo/Btrfs-dedupe-framework/20160330
Hi,
You are missing the patch set which includes
https://patchwork.kernel.org/patch/8659651/
btrfs: refactor btrfs_dev_replace_start for reuse
Thanks, Anand
On 03/29/2016 10:45 PM, kbuild test robot wrote:
Hi Anand,
[auto build test ERROR on btrfs/next]
[also build test ERROR on v4.
Thanks for your comments, Qu.
Alex.
On Wed, Mar 30, 2016 at 2:34 AM, Qu Wenruo wrote:
>
>
> Alex Lyakas wrote on 2016/03/29 19:22 +0200:
>>
>> Greetings Qu Wenruo,
>>
>> I have reviewed the dedup patchset found in the github account you
>> mentioned. I have several questions. Please note that b
On 2016-03-29 16:26, Chris Murphy wrote:
On Tue, Mar 29, 2016 at 1:59 PM, Austin S. Hemmelgarn
wrote:
On 2016-03-29 15:24, Yauhen Kharuzhy wrote:
On Tue, Mar 29, 2016 at 10:41:36PM +0800, Anand Jain wrote:
No. No. No please don't do that, it would lead to trouble in handing
slow devi
On Tue, Mar 29, 2016 at 07:04:10PM -1000, Jean-Denis Girard wrote:
> Hi list,
>
> I just started to use send / receive for backups to another drive.
> That's a great feature, but unfortunately I'm getting page allocation
> failure, see below.
>
> My backup script does something like this for 11 s
Hi,
I could find very limited documentation related to on disk layout of btrfs
and how all trees are related to each other. Except wiki which has very
specific top level details I couldn't able to find more details on btrfs.
FSs such as zfs, ext3/4 and XFS there are documents which explains ond
On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo wrote:
> First of all.
>
> The "crossing stripe boundary" error message itself is *HARMLESS* for recent
> kernels.
>
> It only means, that metadata extent won't be checked by scrub on recent
> kernels.
> Because scrub by its codes, has a limitation that,
The allocation of node could fail if the memory is too fragmented for a
given node size, practically observed with 64k.
http://article.gmane.org/gmane.comp.file-systems.btrfs/54689
Reported-by: Jean-Denis Girard
Signed-off-by: David Sterba
---
fs/btrfs/ctree.c | 12
1 file changed
Sorry, I had about 3.5MB if xterm buffer, including my test to see if
I would get a panic with the old kernel i had left in grub - I grabbed
the wrong panic.
running 4.4.6 ( which deb packages as 4.4.0 for some reason - I was
confused) I am able to capture this on a mount attempt before my ssh
con
On Fri, Mar 25, 2016 at 10:55:34AM +0900, Satoru Takeuchi wrote:
> To accept DUP on multidev fs, in addition to the following
> commit, we need to mark DUP as an allowed data/metadata
> profile.
>
> commit 42f1279bf8e9 ("btrfs-progs: mkfs: allow DUP on multidev fs, only warn")
>
> * actual result
Hi,
We are hitting the attached lockup on a somewhat regular basis during
nightly tests. Looks like a bunch of CPUs spin in find_free_extent()
on btrfs_free_cluster::lock, which is held by writer, who seems to be
stuck in an endless loop in btrfs_find_space_cluster(), trying to
cleanup bitmaps li
Hi David,
Le 30/03/2016 03:50, David Sterba a écrit :
> On Tue, Mar 29, 2016 at 07:04:10PM -1000, Jean-Denis Girard wrote:
>>
>> [ 3734.651439] btrfs: page allocation failure: order:4, mode:0x2404040
>
> Order 4 is 64k, and most probably it's the allocation of a nodesize, the
> IP offset in the
On Wed, Mar 30, 2016 at 04:05:43PM +0200, David Sterba wrote:
> The allocation of node could fail if the memory is too fragmented for a
> given node size, practically observed with 64k.
It's not a critical path. Why not use vmalloc directly?
Thanks,
-liubo
>
> http://article.gmane.org/gmane.c
On Wed, Mar 30, 2016 at 05:24:04PM +0200, Ilya Dryomov wrote:
> Hi,
>
> We are hitting the attached lockup on a somewhat regular basis during
> nightly tests. Looks like a bunch of CPUs spin in find_free_extent()
> on btrfs_free_cluster::lock, which is held by writer, who seems to be
> stuck in a
On Wed, Mar 30, 2016 at 01:58:03PM +, sri wrote:
> Hi,
>
> I could find very limited documentation related to on disk layout of btrfs
> and how all trees are related to each other. Except wiki which has very
> specific top level details I couldn't able to find more details on btrfs.
>
> FSs
On Wed, Mar 30, 2016 at 10:10:45AM -0700, Liu Bo wrote:
> On Wed, Mar 30, 2016 at 04:05:43PM +0200, David Sterba wrote:
> > The allocation of node could fail if the memory is too fragmented for a
> > given node size, practically observed with 64k.
>
> It's not a critical path. Why not use vmalloc
On Wed, Mar 30, 2016 at 7:25 PM, Liu Bo wrote:
> On Wed, Mar 30, 2016 at 05:24:04PM +0200, Ilya Dryomov wrote:
>> Hi,
>>
>> We are hitting the attached lockup on a somewhat regular basis during
>> nightly tests. Looks like a bunch of CPUs spin in find_free_extent()
>> on btrfs_free_cluster::lock,
Hi David,
Le 30/03/2016 04:05, David Sterba a écrit :
> The allocation of node could fail if the memory is too fragmented for a
> given node size, practically observed with 64k.
>
> http://article.gmane.org/gmane.comp.file-systems.btrfs/54689
>
> Reported-by: Jean-Denis Girard
> Signed-off-by:
Quoting Liu Bo :
On Wed, Mar 30, 2016 at 01:58:03PM +, sri wrote:
Hi,
I could find very limited documentation related to on disk layout of btrfs
and how all trees are related to each other. Except wiki which has very
specific top level details I couldn't able to find more details on btrfs.
Hi all,
Christoph and I have been working on adding reflink and CoW support to
XFS recently. Since the purpose of (mode 0) fallocate is to make sure
that future file writes cannot ENOSPC, I extended the XFS fallocate
handler to unshare any shared blocks via the copy on write mechanism I
built for
On Wed, Mar 30, 2016 at 01:58:03PM +, sri wrote:
> I could find very limited documentation related to on disk layout of btrfs
> and how all trees are related to each other. Except wiki which has very
> specific top level details I couldn't able to find more details on btrfs.
>
> FSs such as
On 2016-03-30 14:27, Darrick J. Wong wrote:
Hi all,
Christoph and I have been working on adding reflink and CoW support to
XFS recently. Since the purpose of (mode 0) fallocate is to make sure
that future file writes cannot ENOSPC, I extended the XFS fallocate
handler to unshare any shared bloc
Since mixed block groups accounting isn't byte-accurate and f_bree is an
unsigned integer, it could overflow. Avoid this.
Signed-off-by: Luis de Bethencourt
Suggested-by: David Sterba
---
fs/btrfs/super.c | 5 +
1 file changed, 5 insertions(+)
diff --git a/fs/btrfs/super.c b/fs/btrfs/super
Metadata for mixed block is already accounted in total data and should not
be counted as part of the free metadata space.
Signed-off-by: Luis de Bethencourt
Link: https://bugzilla.kernel.org/show_bug.cgi?id=114281
---
fs/btrfs/super.c | 16 +---
1 file changed, 13 insertions(+), 3 de
On Wed, Mar 30, 2016 at 9:53 PM, Luis de Bethencourt
wrote:
> Since mixed block groups accounting isn't byte-accurate and f_bree is an
> unsigned integer, it could overflow. Avoid this.
>
> Signed-off-by: Luis de Bethencourt
> Suggested-by: David Sterba
> ---
> fs/btrfs/super.c | 5 +
> 1 f
On 30/03/16 22:48, Filipe Manana wrote:
> On Wed, Mar 30, 2016 at 9:53 PM, Luis de Bethencourt
> wrote:
>> Since mixed block groups accounting isn't byte-accurate and f_bree is an
>> unsigned integer, it could overflow. Avoid this.
>>
>> Signed-off-by: Luis de Bethencourt
>> Suggested-by: David S
On Tue, Mar 29, 2016 at 10:40:40PM +0300, Yauhen Kharuzhy wrote:
> Hi.
>
> I am testing hotspare v2 on kernel v4.4.5 (I will try latest Chris' tree
> later)
> now with lockdep debugging enabled. At starting of replacement, lockdep
> warning is displayed,
> because kstrdup() is called with GFP_NO
Since mixed block groups accounting isn't byte-accurate and f_bree is an
unsigned integer, it could overflow. Avoid this.
Signed-off-by: Luis de Bethencourt
Suggested-by: David Sterba
---
Hi,
Thanks to Filipe Manana for spotting a mistake in the first version of this
patch.
Luis
fs/btrfs/sup
We are introducing a give loans at a rate below 2%, and the time of repayment
of the loan is 20 years. Contact us if you are interested so we can provide
more details.
E-mail:jlo...@aol.com
Regarding
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a
Henk Slager wrote on 2016/03/30 16:03 +0200:
On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo wrote:
First of all.
The "crossing stripe boundary" error message itself is *HARMLESS* for recent
kernels.
It only means, that metadata extent won't be checked by scrub on recent
kernels.
Because scrub b
On Wed, Mar 30, 2016 at 11:27:55AM -0700, Darrick J. Wong wrote:
> Hi all,
>
> Christoph and I have been working on adding reflink and CoW support to
> XFS recently. Since the purpose of (mode 0) fallocate is to make sure
> that future file writes cannot ENOSPC, I extended the XFS fallocate
> han
create_pending_snapshot() will go readonly on _any_ error return from
btrfs_qgroup_inherit(). If qgroups are enabled, a user can crash their fs by
just making a snapshot and asking it to inherit from an invalid qgroup. For
example:
$ btrfs sub snap -i 1/10 /btrfs/ /btrfs/foo
Will cause a transact
On Wed, Mar 30, 2016 at 11:27:55AM -0700, Darrick J. Wong wrote:
> Or is it ok that fallocate could block, potentially for a long time as
> we stream cows through the page cache (or however unshare works
> internally)? Those same programs might not be expecting fallocate to
> take a long time.
Ye
Kai Krakow wrote on 2016/03/28 12:02 +0200:
Changing subject to reflect the current topic...
Am Sun, 27 Mar 2016 21:55:40 +0800
schrieb Qu Wenruo :
I finally got copy&paste data:
# before mounting let's check the FS:
$ sudo btrfsck /dev/disk/by-label/usb-backup
Checking filesystem on /dev/
On Wed, Mar 30, 2016 at 06:13:43PM +0800, Anand Jain wrote:
>
>
> Hi,
>
> You are missing the patch set which includes
>https://patchwork.kernel.org/patch/8659651/
>
> btrfs: refactor btrfs_dev_replace_start for reuse
Sorry that comes in another patchset and the robot currently is not
sm
Mark Fasheh wrote on 2016/03/30 17:57 -0700:
create_pending_snapshot() will go readonly on _any_ error return from
btrfs_qgroup_inherit(). If qgroups are enabled, a user can crash their fs by
just making a snapshot and asking it to inherit from an invalid qgroup. For
example:
$ btrfs sub snap
At least 2 user from mail list reported btrfsck reported false alert of
"bad metadata [,) crossing stripe boundary".
While the reported number are all inside the same 64K boundary.
After some check, all the false alert have the same bytenr feature,
which can be divided by stripe size (64K)
Henk Slager wrote on 2016/03/30 16:03 +0200:
On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo wrote:
First of all.
The "crossing stripe boundary" error message itself is *HARMLESS* for recent
kernels.
It only means, that metadata extent won't be checked by scrub on recent
kernels.
Because scrub b
Qu Wenruo wrote on 2016/03/31 09:33 +0800:
Kai Krakow wrote on 2016/03/28 12:02 +0200:
Changing subject to reflect the current topic...
Am Sun, 27 Mar 2016 21:55:40 +0800
schrieb Qu Wenruo :
I finally got copy&paste data:
# before mounting let's check the FS:
$ sudo btrfsck /dev/disk/by
On Wed, Mar 30, 2016 at 10:38:42AM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana
>
> Test that if we rename a directory, create a new file or directory that
> has the old name of our former directory and is a child of the same
> parent directory, fsync the new inode, power fail and moun
On Wed, Mar 30, 2016 at 10:39:10AM +0100, fdman...@kernel.org wrote:
> From: Filipe Manana
>
> Test that if we rename a file, create a new file that has the old name
> of the other file and is a child of the same parent directory, fsync the
> new inode, power fail and mount the filesystem, we do
Hi,
one of my problem btrfs instances went into a hung process state
while blancing metadata. This process is recorded in the file system
somehow and the balance restarts immediately after mounting the
filesystem with no chance to issue a btrfs balance cancel command
before the system hangs again.
73 matches
Mail list logo
