Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Zygo Blaxell
On Tue, Nov 22, 2016 at 06:44:19PM -0800, Darrick J. Wong wrote: > On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > > 3. Both XFS and Btrfs cap each dedupe operation to 16MB, but the > > >implicit EOF gets aroun

Re: Inconsistent free space with false ENOSPC

2016-11-23 Thread Duncan
Martin Raiber posted on Wed, 23 Nov 2016 16:22:29 + as excerpted: > On 23.11.2016 07:09 Duncan wrote: >> Yes, you're in a *serious* metadata bind. >> Any time global reserve has anything above zero usage, it means the >> filesystem is in dire straits, and well over half of your global >> reser

bees v0.1 - Best-Effort Extent-Same, a btrfs deduplication daemon

2016-11-23 Thread Zygo Blaxell
I made a thing! Bees ("Best-Effort Extent-Same") is a dedup daemon for btrfs. Bees is a block-oriented userspace dedup designed to avoid scalability problems on large filesystems. Bees is designed to degrade gracefully when underprovisioned with RAM. Bees does not use more RAM or storage as file

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Darrick J. Wong
On Thu, Nov 24, 2016 at 10:53:24AM +1100, Dave Chinner wrote: > On Wed, Nov 23, 2016 at 06:14:47PM -0500, Zygo Blaxell wrote: > > On Thu, Nov 24, 2016 at 09:13:28AM +1100, Dave Chinner wrote: > > > On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote: > > > > On Wed, Nov 23, 2016 at 03:26:3

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Dave Chinner
On Wed, Nov 23, 2016 at 06:14:47PM -0500, Zygo Blaxell wrote: > On Thu, Nov 24, 2016 at 09:13:28AM +1100, Dave Chinner wrote: > > On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote: > > > On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote: > > > > On Tue, Nov 22, 2016 at 09:02:1

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Zygo Blaxell
On Thu, Nov 24, 2016 at 09:13:28AM +1100, Dave Chinner wrote: > On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote: > > On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote: > > > On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > > > > On Thu, Nov 17, 2016 at 04:07:4

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Dave Chinner
On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote: > On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote: > > On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > > > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > > > 3. Both XFS and Btrfs cap each

[PATCH v2] Btrfs: fix BUG_ON in btrfs_mark_buffer_dirty

2016-11-23 Thread Liu Bo
This can only happen with CONFIG_BTRFS_FS_CHECK_INTEGRITY=y. Commit 1ba98d0 ("Btrfs: detect corruption when non-root leaf has zero item") assumes that a leaf is its root when leaf->bytenr == btrfs_root_bytenr(root), however, we should not use btrfs_root_bytenr(root) since it's mainly got updated d

Re: [PATCH] Btrfs: fix BUG_ON in btrfs_mark_buffer_dirty

2016-11-23 Thread Liu Bo
On Wed, Nov 23, 2016 at 05:48:45PM +, Filipe Manana wrote: > On Wed, Nov 23, 2016 at 1:15 PM, Filipe Manana wrote: > > On Mon, Oct 17, 2016 at 4:44 PM, Liu Bo wrote: > >> On Mon, Oct 17, 2016 at 03:00:25PM +0200, David Sterba wrote: > >>> On Thu, Oct 13, 2016 at 09:47:11AM +0100, Filipe Manan

Re: [PATCH] Btrfs: fix emptiness check for dirtied extent buffers at check_leaf()

2016-11-23 Thread Liu Bo
On Wed, Nov 23, 2016 at 05:47:15PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > We can not simply use the owner field from an extent buffer's header to > get the id of the respective tree when the extent buffer is from a > relocation tree. When we create the root for a relocation t

Re: resend: Re: Btrfs: adjust len of writes if following a preallocated extent

2016-11-23 Thread Liu Bo
Hi, On Wed, Nov 23, 2016 at 06:21:35PM +0100, Stefan Priebe - Profihost AG wrote: > Hi, > > sorry last mail was from the wrong box. > > Am 04.11.2016 um 20:20 schrieb Liu Bo: > > If we have > > > > |0--hole--4095||4096--preallocate--12287| > > > > instead of using preallocated space, a 8K dire

Re: bio linked list corruption.

2016-11-23 Thread Dave Jones
On Wed, Nov 23, 2016 at 02:34:19PM -0500, Dave Jones wrote: > [ 317.689216] BUG: Bad page state in process kworker/u8:8 pfn:4d8fd4 > trace from just before this happened. Does this shed any light ? > > https://codemonkey.org.uk/junk/trace.txt crap, I just noticed the timestamps in the trac

Re: bio linked list corruption.

2016-11-23 Thread Dave Jones
On Mon, Oct 31, 2016 at 01:44:55PM -0600, Chris Mason wrote: > On Mon, Oct 31, 2016 at 12:35:16PM -0700, Linus Torvalds wrote: > >On Mon, Oct 31, 2016 at 11:55 AM, Dave Jones > >wrote: > >> > >> BUG: Bad page state in process kworker/u8:12 pfn:4e0e39 > >> page:ea0013838e40 count:0 mapc

Re: resend: Re: Btrfs: adjust len of writes if following a preallocated extent

2016-11-23 Thread Stefan Priebe - Profihost AG
Am 23.11.2016 um 19:23 schrieb Holger Hoffstätte: > On 11/23/16 18:21, Stefan Priebe - Profihost AG wrote: >> Am 04.11.2016 um 20:20 schrieb Liu Bo: >>> If we have >>> >>> |0--hole--4095||4096--preallocate--12287| >>> >>> instead of using preallocated space, a 8K direct write will just >>> create

Re: btrfs-progs-4.8.3 libbtrfs missing symbols?

2016-11-23 Thread Mike Gilbert
On Wed, Nov 23, 2016 at 1:43 PM, Mike Gilbert wrote: > On Wed, Nov 23, 2016 at 4:45 AM, David Sterba wrote: >> On Tue, Nov 22, 2016 at 01:49:15PM -0500, Mike Gilbert wrote: >>> Hi, >>> >>> I received a bug report about a build failure in a package (snapper) >>> that utilizes libbtrfs. >>> >>> htt

Re: btrfs-progs-4.8.3 libbtrfs missing symbols?

2016-11-23 Thread Mike Gilbert
On Wed, Nov 23, 2016 at 4:45 AM, David Sterba wrote: > On Tue, Nov 22, 2016 at 01:49:15PM -0500, Mike Gilbert wrote: >> Hi, >> >> I received a bug report about a build failure in a package (snapper) >> that utilizes libbtrfs. >> >> https://bugs.gentoo.org/show_bug.cgi?id=600078 >> >> To summarize

Re: Inconsistent free space with false ENOSPC

2016-11-23 Thread Martin Raiber
On 23.11.2016 07:09 Duncan wrote: > Yes, you're in a *serious* metadata bind. > Any time global reserve has anything above zero usage, it means the > filesystem is in dire straits, and well over half of your global reserve > is used, a state that is quite rare as btrfs really tries hard not to us

Re: resend: Re: Btrfs: adjust len of writes if following a preallocated extent

2016-11-23 Thread Holger Hoffstätte
On 11/23/16 18:21, Stefan Priebe - Profihost AG wrote: > Am 04.11.2016 um 20:20 schrieb Liu Bo: >> If we have >> >> |0--hole--4095||4096--preallocate--12287| >> >> instead of using preallocated space, a 8K direct write will just >> create a new 8K extent and it'll end up with >> >> |0--new extent--

Re: [PATCH] Btrfs: fix BUG_ON in btrfs_mark_buffer_dirty

2016-11-23 Thread Filipe Manana
On Wed, Nov 23, 2016 at 1:15 PM, Filipe Manana wrote: > On Mon, Oct 17, 2016 at 4:44 PM, Liu Bo wrote: >> On Mon, Oct 17, 2016 at 03:00:25PM +0200, David Sterba wrote: >>> On Thu, Oct 13, 2016 at 09:47:11AM +0100, Filipe Manana wrote: >>> > > Since the crash is similar to the call chains from Jef

[PATCH] Btrfs: fix emptiness check for dirtied extent buffers at check_leaf()

2016-11-23 Thread fdmanana
From: Filipe Manana We can not simply use the owner field from an extent buffer's header to get the id of the respective tree when the extent buffer is from a relocation tree. When we create the root for a relocation tree we leave (on purpose) the owner field with the same value as the subvolume'

resend: Re: Btrfs: adjust len of writes if following a preallocated extent

2016-11-23 Thread Stefan Priebe - Profihost AG
Hi, sorry last mail was from the wrong box. Am 04.11.2016 um 20:20 schrieb Liu Bo: > If we have > > |0--hole--4095||4096--preallocate--12287| > > instead of using preallocated space, a 8K direct write will just > create a new 8K extent and it'll end up with > > |0--new extent--8191||8192--prea

Re: [PATCH] Btrfs: fix truncate down when no_holes feature is enabled

2016-11-23 Thread David Sterba
On Tue, Nov 22, 2016 at 12:08:47PM -0800, Liu Bo wrote: > On Tue, Nov 22, 2016 at 02:13:21PM -0500, Chris Mason wrote: > > On 11/11/2016 05:27 PM, Liu Bo wrote: > > > For such a file mapping, > > > > > > [0-4k][hole][8k-12k] > > > > > > In NO_HOLES mode, we don't have the [hole] extent any more.

Re: Btrfs: adjust len of writes if following a preallocated extent

2016-11-23 Thread NOC - Profihost AG
Hi, Am 04.11.2016 um 20:20 schrieb Liu Bo: > If we have > > |0--hole--4095||4096--preallocate--12287| > > instead of using preallocated space, a 8K direct write will just > create a new 8K extent and it'll end up with > > |0--new extent--8191||8192--preallocate--12287| > > It's because we fin

Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

2016-11-23 Thread Zygo Blaxell
On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote: > On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote: > > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote: > > > 3. Both XFS and Btrfs cap each dedupe operation to 16MB, but the > > >implicit EOF gets around t

Re: [PATCH] Btrfs: fix BUG_ON in btrfs_mark_buffer_dirty

2016-11-23 Thread Filipe Manana
On Mon, Oct 17, 2016 at 4:44 PM, Liu Bo wrote: > On Mon, Oct 17, 2016 at 03:00:25PM +0200, David Sterba wrote: >> On Thu, Oct 13, 2016 at 09:47:11AM +0100, Filipe Manana wrote: >> > > Since the crash is similar to the call chains from Jeff's report, >> > > ie. >> > > btrfs_del_csums >> > > -> bt

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

2016-11-23 Thread David Sterba
On Wed, Nov 23, 2016 at 11:00:49AM +0800, Anand Jain wrote: > > Preparatory bits are in patches: > > > > btrfs-progs: mkfs: add names of matching sysfs feature names > > btrfs-progs: mkfs: enhance feature table > > btrfs-progs: mkfs: extend mkfs features with compat, safe and default > > ver

[GIT PULL] Btrfs corruption fix and a couple cleanups

2016-11-23 Thread fdmanana
From: Filipe Manana Hi Chris, Here follows two cleanups and a fix for an issue that leads to a logical corruption in the extent tree, where we end up with file extent items in subvolume trees that don't have a matching extent item and back reference in the extent tree after a balance when qgroup

Re: btrfs-progs-4.8.3 libbtrfs missing symbols?

2016-11-23 Thread David Sterba
On Tue, Nov 22, 2016 at 01:49:15PM -0500, Mike Gilbert wrote: > Hi, > > I received a bug report about a build failure in a package (snapper) > that utilizes libbtrfs. > > https://bugs.gentoo.org/show_bug.cgi?id=600078 > > To summarize the issue, the package has a configure test that executes > t

[v2 PATCH] btrfs: raid56: Use correct stolen pages to calculate P/Q

2016-11-23 Thread Qu Wenruo
In the following situation, scrub will calculate wrong parity to overwrite correct one: RAID5 full stripe: Before | Dev 1 | Dev 2 | Dev 3 | | Data stripe 1 | Data stripe 2 | Parity Stripe | --- 0 | 0x (Bad) | 0x