Introduce a new tree, dedup tree to record on-disk dedup hash.
As a persist hash storage instead of in-memeory only implement.
Unlike Liu Bo's implement, in this version we won't do hack for
bytenr -> hash search, but add a new type, DEDUP_BYTENR_ITEM for such
search case, just like in-memory back
From: Wang Xiaoguang
In btrfs_delalloc_reserve_metadata(), the number of metadata bytes we try
to reserve is calculated by the difference between outstanding_extents and
reserved_extents.
When reserve_metadata_bytes() fails to reserve desited metadata space,
it has already done some reclaim work
From: Wang Xiaoguang
Introduce BTRFS_INODE_NODEDUP flag, then we can explicitly disable
online data deduplication for specified files.
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/ctree.h | 1 +
fs/btrfs/ioctl.c | 6 +-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/c
From: Wang Xiaoguang
Introduce static function inmem_add() to add hash into in-memory tree.
And now we can implement the btrfs_dedup_add() interface.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedup.c | 157 +++
1 fi
The patchset can also be fetched from github:
https://github.com/adam900710/linux.git wang_dedup
This updated version of inband de-duplication has the following features:
1) ONE unified dedup framework.
Most of its code is hidden quietly in dedup.c and export the minimal
interfaces for its c
Since we will introduce a new on-disk based dedup method, introduce new
interfaces to resume previous dedup setup.
And since we introduce a new tree for status, also add disable handler
for it.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedup.c | 277
Now on-disk backend can delete hash now.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedup.c | 100 +++
1 file changed, 100 insertions(+)
diff --git a/fs/btrfs/dedup.c b/fs/btrfs/dedup.c
index c2b6a03..309cbc3 100644
-
From: Wang Xiaoguang
Introduce static function inmem_del() to remove hash from in-memory
dedup tree.
And implement btrfs_dedup_del() and btrfs_dedup_destroy() interfaces.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedup.c | 88 +
From: Wang Xiaoguang
Add ioctl interface for inband deduplication, which includes:
1) enable
2) disable
3) status
We will later add ioctl to disable inband dedup for given file/dir.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedup.c | 52 +++
From: Wang Xiaoguang
We use btrfs extended attribute "btrfs.dedup" to record per-file online
dedup status, so add a dedup property handler.
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/props.c | 40
1 file changed, 40 insertions(+)
diff --git a/fs/btrfs/
From: Wang Xiaoguang
Introduce the header for btrfs online(write time) de-duplication
framework and needed header.
The new de-duplication framework is going to support 2 different dedup
method and 1 dedup hash.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/ctree.h |
From: Wang Xiaoguang
Unlike in-memory or on-disk dedup method, only SHA256 hash method is
supported yet, so implement btrfs_dedup_calc_hash() interface using
SHA256.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/dedup.c | 43 +++
1
From: Wang Xiaoguang
Introduce static function inmem_search() to handle the job for in-memory
hash tree.
The trick is, we must ensure the delayed ref head is not being run at
the time we search the for the hash.
With inmem_search(), we can implement the btrfs_dedup_search()
interface.
Signed-o
Core implement for inband de-duplication.
It reuse the async_cow_start() facility to do the calculate dedup hash.
And use dedup hash to do inband de-duplication at extent level.
The work flow is as below:
1) Run delalloc range for an inode
2) Calculate hash for the delalloc range at the unit of de
From: Wang Xiaoguang
Add ordered-extent support for dedup.
Note, current ordered-extent support only supports non-compressed source
extent.
Support for compressed source extent will be added later.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/ordered-data.c | 36 ++
Now on-disk backend should be able to search hash now.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedup.c | 133 ---
fs/btrfs/dedup.h | 1 +
2 files changed, 118 insertions(+), 16 deletions(-)
diff --git a/fs/btrfs/
From: Wang Xiaoguang
Add generic function to initialize dedup info.
Signed-off-by: Qu Wenruo
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/Makefile | 2 +-
fs/btrfs/dedup.c | 100 ++
fs/btrfs/dedup.h | 17 --
3 files changed, 116 i
For in-band dedup, btrfs needs to increase data ref with delayed_ref
locked, so add a new function btrfs_add_delayed_data_ref_lock() to
increase extent ref with delayed_refs already locked.
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedup.c | 1 +
fs/btrfs/delayed-ref.c | 30 ++
From: Wang Xiaoguang
Introduce inode_need_dedup() to implement per-file online dedup control.
Signed-off-by: Wang Xiaoguang
---
fs/btrfs/inode.c | 15 ++-
1 file changed, 14 insertions(+), 1 deletion(-)
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 1e27a71..1973f86 100644
Now on-disk backend can add hash now.
Signed-off-by: Wang Xiaoguang
Signed-off-by: Qu Wenruo
---
fs/btrfs/dedup.c | 83
1 file changed, 83 insertions(+)
diff --git a/fs/btrfs/dedup.c b/fs/btrfs/dedup.c
index 309cbc3..073726d 100644
--- a
Add dedup tree support for btrfs-debug-tree.
Signed-off-by: Qu Wenruo
---
v2:
Add support to print hex objectid/offset for dedup hash.
Add support to print hex hash.
---
btrfs-debug-tree.c | 4 +++
ctree.h| 7
print-tree.c | 94 +++
Add disable subcommand for dedup command group.
Signed-off-by: Qu Wenruo
---
Documentation/btrfs-dedup.asciidoc | 5 +
cmds-dedup.c | 42 ++
2 files changed, 47 insertions(+)
diff --git a/Documentation/btrfs-dedup.asciidoc
b/Docume
Add enable subcommand for dedup commmand group.
Signed-off-by: Qu Wenruo
---
Documentation/btrfs-dedup.asciidoc | 79 -
cmds-dedup.c | 138 +
ioctl.h| 2 +
3 files changed, 218 insertions
Now btrfs-show-super can handle DEDUP ro compat flag.
Signed-off-by: Qu Wenruo
---
btrfs-show-super.c | 17 +
1 file changed, 17 insertions(+)
diff --git a/btrfs-show-super.c b/btrfs-show-super.c
index 051bd11..0bc0b1f 100644
--- a/btrfs-show-super.c
+++ b/btrfs-show-super.c
@@
Add status subcommand for dedup command group.
Signed-off-by: Qu Wenruo
---
Documentation/btrfs-dedup.asciidoc | 3 ++
cmds-dedup.c | 76 ++
2 files changed, 79 insertions(+)
diff --git a/Documentation/btrfs-dedup.asciidoc
b/Documentat
Add new DEDUP ro compat flag and corresponding mkfs/convert flag
'dedup'.
Since dedup tree is completely isolated from fs tree, so even old kernel
could do read mount.
So add it to RO compat flag instead of common incompat flags
Signed-off-by: Qu Wenruo
---
Documentation/mkfs.btrfs.asciidoc |
From: Wang Xiaoguang
Normally if we enable online dedup for a fs, it's filesystem wide
de-duplication. With this property, we can explicitly disable data
de-duplication for specified files.
Signed-off-by: Wang Xiaoguang
---
Documentation/btrfs-property.asciidoc | 2 +
props.c
Add basic ioctl header and command group framework for later use.
Alone with basic man page doc.
Signed-off-by: Qu Wenruo
---
Documentation/Makefile.in | 1 +
Documentation/btrfs-dedup.asciidoc | 39 +++
Documentation/btrfs.asciidoc | 4
Makefile
Preparation patchset for in-coming (aimed for 4.6) kernel in-band
de-duplication patchset.
New kernel dedup will has 2 different dedup backends and a ioctl
interface to enable/disable dedup.
The ioctl interface and on-disk format (mostly) is determined, so submit
this patchset first before de-dup
You're right, I misunderstood the mail and get a little emotional.
And sorry for that inappropriate reply.
BTW, the new patchset will be sent to btrfs mail list in coming hour, if
you're interested in inband de-dup, you can try the following github
repos to build and test in-band de-dup:
Ke
Btrfs has its sysfs interface showing what features current kernel/btrfs
module support.
Add _require_btrfs_kernel_feature() to check such interface.
Also rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion.
Signed-off-by: Qu Wenruo
---
common/rc | 12 +++-
This test will check per inode dedup flag.
Signed-off-by: Qu Wenruo
---
tests/btrfs/202 | 117
tests/btrfs/202.out | 15 +++
tests/btrfs/group | 1 +
3 files changed, 133 insertions(+)
create mode 100755 tests/btrfs/202
create m
Basic test for btrfs in-band de-duplication.
The comprehensive test should be enable dedup for every existing test
case (except some ENOSPC test, which need life time to fill it).
Dave is implementing the feature allowing us to execute such command
at mount time, but it's not implemented now.
Qu
Add basic test for btrfs in-band de-duplication, including:
1) Enable
2) Re-enable
3) On disk extents are refering to same bytenr
4) Disable
Signed-off-by: Qu Wenruo
---
common/defrag | 8
tests/btrfs/200 | 112
tests/btrfs/20
Add test case to check btrfs dedup enable/disable race.
Signed-off-by: Qu Wenruo
---
tests/btrfs/201 | 97 +
tests/btrfs/201.out | 1 +
tests/btrfs/group | 1 +
3 files changed, 99 insertions(+)
create mode 100755 tests/btrfs/201
crea
Hi,
On Wed, Jan 20, 2016 at 01:49:24PM -0800, Mark Fasheh wrote:
> A git tree of the patches can be found here:
>
> https://github.com/markfasheh/btrfs-progs-patches/tree/du
what changed since the previous posting?
Otherwise code looks ok, I saw use of uint64_t, the units help can be
extended t
On Mon, Feb 1, 2016 at 2:31 PM, Hendrik Friedel wrote:
> Hello,
>
> I am running CentOS from a btrfs root.
> This worked fine until I added a device to that pool:
> btrfs device add /dev/sda3 /
> reboot
>
> This now causes the errors:
> BTRFS: failed to read chunk tree on sdb3
> BTRFS: open_ctree
On Fri, Jan 29, 2016 at 11:17:48AM +0800, Qu Wenruo wrote:
> Ping.
>
> Any comment?
> Or will it be merged into current 4.5?
Apparently not for 4.5, I'm going to revisit that for 4.6.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@v
On Mon, Feb 01, 2016 at 11:02:42PM +, Duncan wrote:
> Hugo Mills posted on Mon, 01 Feb 2016 22:11:20 + as excerpted:
>
> > On Mon, Feb 01, 2016 at 10:31:47PM +0100, Hendrik Friedel wrote:
> >> Hello,
> >>
> >> I am running CentOS from a btrfs root.
> >> This worked fine until I added a de
Hugo Mills posted on Mon, 01 Feb 2016 22:11:20 + as excerpted:
> On Mon, Feb 01, 2016 at 10:31:47PM +0100, Hendrik Friedel wrote:
>> Hello,
>>
>> I am running CentOS from a btrfs root.
>> This worked fine until I added a device to that pool:
>> btrfs device add /dev/sda3 /
>> reboot
>>
>> Th
On Mon, Feb 01, 2016 at 10:31:47PM +0100, Hendrik Friedel wrote:
> Hello,
>
> I am running CentOS from a btrfs root.
> This worked fine until I added a device to that pool:
> btrfs device add /dev/sda3 /
> reboot
>
> This now causes the errors:
> BTRFS: failed to read chunk tree on sdb3
> BTRFS:
Hello,
I am running CentOS from a btrfs root.
This worked fine until I added a device to that pool:
btrfs device add /dev/sda3 /
reboot
This now causes the errors:
BTRFS: failed to read chunk tree on sdb3
BTRFS: open_ctree failed
Here I am stuck in a recovery prompt.
btrfs fi show displays th
On Mon, Feb 01, 2016 at 03:37:39PM -0500, Austin S. Hemmelgarn wrote:
> On 2016-02-01 15:21, Hugo Mills wrote:
> >On Mon, Feb 01, 2016 at 02:44:24PM -0500, Austin S. Hemmelgarn wrote:
> >>In the process of trying to debug issues I'm having on one of my
> >>systems with a new kernel version, I decid
On Mon, Feb 1, 2016 at 7:10 AM, Christian Rohmann
wrote:
> Hey Chris,
>
>
> sorry for the late reply.
>
>
> On 01/27/2016 10:53 PM, Chris Murphy wrote:
>> I can't exactly reproduce this. I'm using +C qcow2 on Btrfs on one SSD
>> to back the drives in the VM.
>>
>> 2x btrfs raid1 with files totalli
On 2016-02-01 15:21, Hugo Mills wrote:
On Mon, Feb 01, 2016 at 02:44:24PM -0500, Austin S. Hemmelgarn wrote:
In the process of trying to debug issues I'm having on one of my
systems with a new kernel version, I decided to do a dry run check on
the root filesystem. 'btrfs check' returned a bunch
On 2016-02-01 15:27, Chris Murphy wrote:
On Mon, Feb 1, 2016 at 12:44 PM, Austin S. Hemmelgarn
wrote:
In the process of trying to debug issues I'm having on one of my
systems with a new kernel version, I decided to do a dry run check on
the root filesystem. 'btrfs check' returned a bunch of li
On Mon, Feb 1, 2016 at 12:44 PM, Austin S. Hemmelgarn
wrote:
> In the process of trying to debug issues I'm having on one of my
> systems with a new kernel version, I decided to do a dry run check on
> the root filesystem. 'btrfs check' returned a bunch of lines like:
>
> root 257 inode XX er
On Mon, Feb 01, 2016 at 02:44:24PM -0500, Austin S. Hemmelgarn wrote:
> In the process of trying to debug issues I'm having on one of my
> systems with a new kernel version, I decided to do a dry run check on
> the root filesystem. 'btrfs check' returned a bunch of lines like:
>
> root 257 inode
In the process of trying to debug issues I'm having on one of my
systems with a new kernel version, I decided to do a dry run check on
the root filesystem. 'btrfs check' returned a bunch of lines like:
root 257 inode XX errors 2000, link count wrong
unresolved ref dir Y index 53 n
On 2016-02-01 09:03, Christian Rohmann wrote:
Hey Chris,
On 01/28/2016 12:47 AM, Chris Murphy wrote:
Might be a bug, but more likely might be a lack of optimization. If it
eventually mounts without errors that's a pretty good plus. Lots of
file systems can't handle power failures well at all.
Hey Chris,
sorry for the late reply.
On 01/27/2016 10:53 PM, Chris Murphy wrote:
> I can't exactly reproduce this. I'm using +C qcow2 on Btrfs on one SSD
> to back the drives in the VM.
>
> 2x btrfs raid1 with files totalling 5G consistently takes ~1 minute
> [1] to balance (no filters)
>
>
Hey Chris,
On 01/28/2016 12:47 AM, Chris Murphy wrote:
> Might be a bug, but more likely might be a lack of optimization. If it
> eventually mounts without errors that's a pretty good plus. Lots of
> file systems can't handle power failures well at all.
So what and how should I go about profiling
From: Filipe Manana
Hi Chris,
Please consider the following fixes for the extent_same ioctl. They fix
issues related to missing proper page and range locking, which lead to
invalid memory accesses and other unexpected failures (such as data
checksum validation failures, on perfectly healthy hard
On 2016-01-29 17:06, Henk Slager wrote:
On Fri, Jan 29, 2016 at 9:40 PM, Austin S. Hemmelgarn
wrote:
On 2016-01-29 15:27, Henk Slager wrote:
On Fri, Jan 29, 2016 at 1:14 PM, Austin S. Hemmelgarn
wrote:
On 2016-01-28 18:01, Chris Murphy wrote:
On Thu, Jan 28, 2016 at 1:44 PM, Austin S. H
On Mon, Feb 1, 2016 at 11:53 AM, Ben Marsh wrote:
> This patch fixes a pointer issue from passing a struct as a parameter
> to the function btrfs_root_dec_send_in_progress.
>
> Signed-off-by: Ben Marsh
> ---
> fs/btrfs/send.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git
This patch fixes a pointer issue from passing a struct as a parameter
to the function btrfs_root_dec_send_in_progress.
Signed-off-by: Ben Marsh
---
fs/btrfs/send.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index 63a6152..47ee63e 100644
On 30 January 2016 at 12:59, Christian Pernegger wrote:
>
> This is on a 1-month-old Debian stable (jessie) install and yes, I
> know that means the kernel and btrfs-progs are ancient
apt-get install -t jessie-backports linux-image-4.3.0-0.bpo.1-amd64
Or something like that for the image name. U
I didn't know that xfstests covers more than xfs. Will look into their
repo for btrfs specific cases. thanks for the response.
Cheers,
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo inf
58 matches
Mail list logo
