Re: [PATCH v4 13/15] btrfs: Optimize unallocated chunks discard

2019-04-02 Thread Nikolay Borisov
On 1.04.19 г. 21:44 ч., David Sterba wrote: > On Wed, Mar 27, 2019 at 02:24:16PM +0200, Nikolay Borisov wrote: >> Currently unallocated chunks are always trimmed. For example >> 2 consecutive trims on large storage would trim freespace twice >> irrespective of whether the space was actually allo

[PATCH v2] fstests: btrfs: try use forget to unregister device

2019-04-02 Thread Anand Jain
Some btrfs test cases use btrfs module-reload to unregister devices in the btrfs kernel. The problem with the module-reload approach is, if test system contains btrfs as rootfs, then you can't run these test cases. Patches [1] introduced btrfs forget feature which can unregister devices without th

[PATCH] btrfs: trace: Introduce trace events for sleepable tree lock

2019-04-02 Thread Qu Wenruo
There are two tree lock events which can sleep: - btrfs_tree_read_lock() - btrfs_tree_lock() Sometimes we may need to look into the concurrency picture of the fs. For that case, we need the execution time of above two functions and the owner of @eb. Here we introduce a trace events for user space

[PATCH v2] btrfs: trace: Introduce trace events for sleepable tree lock

2019-04-02 Thread Qu Wenruo
There are two tree lock events which can sleep: - btrfs_tree_read_lock() - btrfs_tree_lock() Sometimes we may need to look into the concurrency picture of the fs. For that case, we need the execution time of above two functions and the owner of @eb. Here we introduce a trace events for user space

[PATCH] fstests: btrfs/048: amend property validation cases

2019-04-02 Thread Anand Jain
Add more property validation cases which are fixed by the patches [1] [1] btrfs: fix property validate fail should not increment generation btrfs: open code btrfs_set_prop in inherit_prop btrfs: fix vanished compression property after failed set btrfs: fix zstd compression parameter Signe

[PATCH 0/4 RESEND] btrfs: fix property bugs

2019-04-02 Thread Anand Jain
1/4 Fixes a bug that we were applying an incomplete string zstd. 2/4 Fixes a bug that the compression property gets reset to NULL after a failed attempt to change the compression parameter on the object. 3/4 preparatory patch so that btrfs_set_prop_trans() can be merged with btrfs_set_prop(

[PATCH 1/4 RESEND] btrfs: fix zstd compression parameter

2019-04-02 Thread Anand Jain
We let to pass zstd compression parameter even if its not fully written. For example: btrfs prop set /btrfs compression zst btrfs prop get /btrfs compression compression=zst zlib and lzo are fine. Fix it by using the expected number of char in strncmp(). Signed-off-by: Anand Jain Revie

[PATCH 2/4 RESEND] btrfs: fix vanished compression property after failed set

2019-04-02 Thread Anand Jain
The compression property resets to NULL, instead of the old value if we fail to set the new compression parameter. btrfs prop get /btrfs compression compression=lzo btrfs prop set /btrfs compression zli ERROR: failed to set compression for /btrfs: Invalid argument btrfs prop get /btrfs compres

[PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread Anand Jain
When an inode inherits property from its parent, we call btrfs_set_prop(). btrfs_set_prop() does an elaborate checks, which is not required in the context of inheriting a property. Instead just open-code only the required items from btrfs_set_prop() and then call btrfs_setxattr() directly. So now t

[PATCH 4/4 RESEND] btrfs: fix property validate fail should not increment generation

2019-04-02 Thread Anand Jain
When the property fails to pass the prop_handlers::validate() check, the thread should exit with no changes in the kernel, but as we are starting the transaction too early, we have just updated the generation even if there is no change. For example: btrfs prop get /btrfs compression compression=l

Re: btrfs and write barriers

2019-04-02 Thread Austin S. Hemmelgarn
On 2019-04-01 15:22, Hendrik Friedel wrote: Dear btrfs-team, I am aware, that barriers are essential for btrfs [1]. I have some questions on that topic: 1) I am not aware how to determine, whether barriers are supported, except for searching dmesg for a message that barriers are disabled. Is t

Re: [PATCH] fstests: btrfs/048: amend property validation cases

2019-04-02 Thread Nikolay Borisov
On 2.04.19 г. 13:07 ч., Anand Jain wrote: > Add more property validation cases which are fixed by the patches [1] > [1] > btrfs: fix property validate fail should not increment generation > btrfs: open code btrfs_set_prop in inherit_prop > btrfs: fix vanished compression property after fa

Re: [PATCH 2/4 RESEND] btrfs: fix vanished compression property after failed set

2019-04-02 Thread Nikolay Borisov
On 2.04.19 г. 13:07 ч., Anand Jain wrote: > The compression property resets to NULL, instead of the old value if we > fail to set the new compression parameter. > > btrfs prop get /btrfs compression > compression=lzo > btrfs prop set /btrfs compression zli > ERROR: failed to set compression

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Nik.
2019-04-02 02:24, Qu Wenruo: On 2019/4/1 上午2:44, bt...@avgustinov.eu wrote: Dear all, I am a big fan of btrfs, and I am using it since 2013 - in the meantime on at least four different computers. During this time, I suffered at least four bad btrfs-failures leading to unmountable, unreadabl

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Qu Wenruo
On 2019/4/2 下午9:06, Nik. wrote: > > 2019-04-02 02:24, Qu Wenruo: >> >> On 2019/4/1 上午2:44, bt...@avgustinov.eu wrote: >>> Dear all, >>> >>> >>> I am a big fan of btrfs, and I am using it since 2013 - in the meantime >>> on at least four different computers. During this time, I suffered at >>> le

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Hugo Mills
On Tue, Apr 02, 2019 at 09:24:03PM +0800, Qu Wenruo wrote: > > > On 2019/4/2 下午9:06, Nik. wrote: [snip] > > On the larger file system only "btrfs check --repair --readonly ..." was > > attempted (without success; most command executions were documented, so > > the results can be made available),

Re: [PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread Nikolay Borisov
On 2.04.19 г. 13:07 ч., Anand Jain wrote: > When an inode inherits property from its parent, we call btrfs_set_prop(). > btrfs_set_prop() does an elaborate checks, which is not required in the > context of inheriting a property. Instead just open-code only the required > items from btrfs_set_pro

Re: [PATCH 4/4 RESEND] btrfs: fix property validate fail should not increment generation

2019-04-02 Thread Nikolay Borisov
On 2.04.19 г. 13:07 ч., Anand Jain wrote: > When the property fails to pass the prop_handlers::validate() check, the > thread should exit with no changes in the kernel, but as we are starting > the transaction too early, we have just updated the generation even if > there is no change. > > For

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Nik.
2019-04-02 15:24, Qu Wenruo: On 2019/4/2 下午9:06, Nik. wrote: 2019-04-02 02:24, Qu Wenruo: On 2019/4/1 上午2:44, bt...@avgustinov.eu wrote: Dear all, I am a big fan of btrfs, and I am using it since 2013 - in the meantime on at least four different computers. During this time, I suffered

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Nik.
of course it complains, it was a typo from me, sorry. The real command was "btrfs check --readonly ...", just to ensure that no writing takes place. Nik. -- 2019-04-02 15:29, Hugo Mills: On Tue, Apr 02, 2019 at 09:24:03PM +0800, Qu Wenruo wrote: On 2019/4/2 下午9:06, Nik. wrote: [sni

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Qu Wenruo
On 2019/4/2 下午9:59, Nik. wrote: > > > 2019-04-02 15:24, Qu Wenruo: >> >> >> On 2019/4/2 下午9:06, Nik. wrote: >>> >>> 2019-04-02 02:24, Qu Wenruo: On 2019/4/1 上午2:44, bt...@avgustinov.eu wrote: > Dear all, > > > I am a big fan of btrfs, and I am using it since 2013 - in

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Hans van Kranenburg
On 4/2/19 4:12 PM, Qu Wenruo wrote: > > > On 2019/4/2 下午9:59, Nik. wrote: >> >> >> 2019-04-02 15:24, Qu Wenruo: >>> >>> >>> On 2019/4/2 下午9:06, Nik. wrote: >>> If the larger fs really doesn't get any write (btrfs check --repair >>> failed to open the fs, thus have the output "cannot open fil

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Nik.
2019-04-02 16:19, Hans van Kranenburg: On 4/2/19 4:12 PM, Qu Wenruo wrote: On 2019/4/2 下午9:59, Nik. wrote: 2019-04-02 15:24, Qu Wenruo: On 2019/4/2 下午9:06, Nik. wrote: If the larger fs really doesn't get any write (btrfs check --repair failed to open the fs, thus have the output "c

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Hans van Kranenburg
On 4/2/19 5:04 PM, Nik. wrote: > > > 2019-04-02 16:19, Hans van Kranenburg: >> On 4/2/19 4:12 PM, Qu Wenruo wrote: >>> >>> >>> On 2019/4/2 下午9:59, Nik. wrote: 2019-04-02 15:24, Qu Wenruo: > > > On 2019/4/2 下午9:06, Nik. wrote: >> > If the larger fs really doesn't

Re: [PATCH v4 03/15] btrfs: Handle pending/pinned chunks before blockgroup relocation during device shrink

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 08:55:32AM +0300, Nikolay Borisov wrote: > > > On 1.04.19 г. 21:26 ч., David Sterba wrote: > > On Wed, Mar 27, 2019 at 02:24:06PM +0200, Nikolay Borisov wrote: > >> During device shrink pinned/pending chunks (i.e those which have been > >> deleted/created respectively, in

Re: [PATCH v4 08/15] btrfs: Stop using call_rcu for device freeing

2019-04-02 Thread David Sterba
On Mon, Apr 01, 2019 at 08:20:56PM +0300, Nikolay Borisov wrote: > > > On 1.04.19 г. 20:07 ч., David Sterba wrote: > > On Wed, Mar 27, 2019 at 02:24:11PM +0200, Nikolay Borisov wrote: > >> btrfs_device structs are freed from RCU context since device iteration > >> is protected by RCU. Currently t

[PATCH 2/2] btrfs-progs: check: fixup_extent_flags needs to deal with non-skinny metadata

2019-04-02 Thread jeffm
From: Jeff Mahoney When repairing a file system created by a very old kernel, I ran into issues fixing up the extent flags since fixup_extent_flags assumed that a METADATA_ITEM would be present if the record was for metadata. Since METADATA_ITEMs don't exist without skinny metadata, we need to f

[PATCH 1/2] btrfs-progs: check: run delayed refs after writing out dirty block groups

2019-04-02 Thread jeffm
From: Jeff Mahoney When repairing the extent tree, it's possible for delayed extents to be created when running btrfs_write_dirty_block_groups. We run delayed refs one last time in the kernel but that is missing in the userspace tools. That results in delayed refs getting dropped on the floor,

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Chris Murphy
On Tue, Apr 2, 2019 at 7:24 AM Qu Wenruo wrote: > On 2019/4/2 下午9:06, Nik. wrote: > > On the larger file system only "btrfs check --repair --readonly ..." was > > attempted (without success; most command executions were documented, so > > the results can be made available), no writing commands we

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Hugo Mills
On Tue, Apr 02, 2019 at 12:28:12PM -0600, Chris Murphy wrote: > On Tue, Apr 2, 2019 at 7:24 AM Qu Wenruo wrote: > > On 2019/4/2 下午9:06, Nik. wrote: > > > > On the larger file system only "btrfs check --repair --readonly ..." was > > > attempted (without success; most command executions were docum

Re: [PATCH 1/2] btrfs-progs: check: run delayed refs after writing out dirty block groups

2019-04-02 Thread Filipe Manana
On Tue, Apr 2, 2019 at 7:29 PM wrote: > > From: Jeff Mahoney > > When repairing the extent tree, it's possible for delayed extents to > be created when running btrfs_write_dirty_block_groups. We run > delayed refs one last time in the kernel but that is missing in > the userspace tools. > > That

Re: [PATCH 2/2] btrfs-progs: check: fixup_extent_flags needs to deal with non-skinny metadata

2019-04-02 Thread Filipe Manana
On Tue, Apr 2, 2019 at 7:31 PM wrote: > > From: Jeff Mahoney > > When repairing a file system created by a very old kernel, I ran into > issues fixing up the extent flags since fixup_extent_flags assumed > that a METADATA_ITEM would be present if the record was for metadata. > > Since METADATA_IT

Re: [PATCH 4/4 RESEND] btrfs: fix property validate fail should not increment generation

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 04:45:23PM +0300, Nikolay Borisov wrote: > > > On 2.04.19 г. 13:07 ч., Anand Jain wrote: > > When the property fails to pass the prop_handlers::validate() check, the > > thread should exit with no changes in the kernel, but as we are starting > > the transaction too early,

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Nik.
2019-04-02 16:12, Qu Wenruo: On 2019/4/2 下午9:59, Nik. wrote: 2019-04-02 15:24, Qu Wenruo: On 2019/4/2 下午9:06, Nik. wrote: 2019-04-02 02:24, Qu Wenruo: On 2019/4/1 上午2:44, bt...@avgustinov.eu wrote: Dear all, I am a big fan of btrfs, and I am using it since 2013 - in the meantime

Re: [PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 04:35:12PM +0300, Nikolay Borisov wrote: > On 2.04.19 г. 13:07 ч., Anand Jain wrote: > > When an inode inherits property from its parent, we call btrfs_set_prop(). > > btrfs_set_prop() does an elaborate checks, which is not required in the > > context of inheriting a propert

Re: [PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 06:07:41PM +0800, Anand Jain wrote: > When an inode inherits property from its parent, we call btrfs_set_prop(). > btrfs_set_prop() does an elaborate checks, which is not required in the > context of inheriting a property. Instead just open-code only the required > items fro

Re: [PATCH 4/4 RESEND] btrfs: fix property validate fail should not increment generation

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 06:07:42PM +0800, Anand Jain wrote: > When the property fails to pass the prop_handlers::validate() check, the > thread should exit with no changes in the kernel, but as we are starting > the transaction too early, we have just updated the generation even if > there is no ch

Re: [PATCH 1/4 RESEND] btrfs: fix zstd compression parameter

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 06:07:38PM +0800, Anand Jain wrote: > We let to pass zstd compression parameter even if its not fully written. > For example: > btrfs prop set /btrfs compression zst > btrfs prop get /btrfs compression > compression=zst > > zlib and lzo are fine. > > Fix it by usi

Re: [PATCH 2/4 RESEND] btrfs: fix vanished compression property after failed set

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 06:07:40PM +0800, Anand Jain wrote: > The compression property resets to NULL, instead of the old value if we > fail to set the new compression parameter. > > btrfs prop get /btrfs compression > compression=lzo > btrfs prop set /btrfs compression zli > ERROR: failed to

Re: [PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread David Sterba
On Tue, Apr 02, 2019 at 04:35:12PM +0300, Nikolay Borisov wrote: > > @@ -367,20 +367,35 @@ static int inherit_props(struct btrfs_trans_handle > > *trans, > > if (!value) > > continue; > > > > + /* may be removed */ > > This comment brings no value. Eith

Re: [PATCH 0/2] Preamture OOM fixes

2019-04-02 Thread David Sterba
On Mon, Apr 01, 2019 at 11:29:56AM +0300, Nikolay Borisov wrote: > David reported that following the async cow path refactoring that got into > misc-next he started seeing premature OOM kills due to large page allocations > (order 4). This was due to now the whole async context (alongisde ancilla

Re: [PATCH] fstests: btrfs/048: amend property validation cases

2019-04-02 Thread Anand Jain
On 2/4/19 8:51 PM, Nikolay Borisov wrote: On 2.04.19 г. 13:07 ч., Anand Jain wrote: Add more property validation cases which are fixed by the patches [1] [1] btrfs: fix property validate fail should not increment generation btrfs: open code btrfs_set_prop in inherit_prop btrfs: f

Re: [PATCH 3/4 RESEND] btrfs: open code btrfs_set_prop in inherit_prop

2019-04-02 Thread Anand Jain
On 3/4/19 7:24 AM, David Sterba wrote: On Tue, Apr 02, 2019 at 04:35:12PM +0300, Nikolay Borisov wrote: @@ -367,20 +367,35 @@ static int inherit_props(struct btrfs_trans_handle *trans, if (!value) continue; + /* may be removed */ This comment brin

Re: [PATCH 4/4 RESEND] btrfs: fix property validate fail should not increment generation

2019-04-02 Thread Anand Jain
On 3/4/19 6:04 AM, David Sterba wrote: On Tue, Apr 02, 2019 at 06:07:42PM +0800, Anand Jain wrote: When the property fails to pass the prop_handlers::validate() check, the thread should exit with no changes in the kernel, but as we are starting the transaction too early, we have just updated

Re: interest in post-mortem examination of a BTRFS system and improving the btrfs-code?

2019-04-02 Thread Qu Wenruo
On 2019/4/3 上午5:22, Nik. wrote: > > 2019-04-02 16:12, Qu Wenruo: >> >> >> On 2019/4/2 下午9:59, Nik. wrote: >>> >>> >>> 2019-04-02 15:24, Qu Wenruo: On 2019/4/2 下午9:06, Nik. wrote: > > 2019-04-02 02:24, Qu Wenruo: >> >> On 2019/4/1 上午2:44, bt...@avgustinov.eu wrote:

Re: [PATCH 04/15] dax: Introduce IOMAP_F_COW for copy-on-write

2019-04-02 Thread Goldwyn Rodrigues
On 10:06 02/04, Dave Chinner wrote: > On Mon, Apr 01, 2019 at 04:41:02PM -0500, Goldwyn Rodrigues wrote: > > On 15:38 01/04, Dave Chinner wrote: > > > On Tue, Mar 26, 2019 at 02:02:50PM -0500, Goldwyn Rodrigues wrote: > > > > From: Goldwyn Rodrigues > > > > > > > > The IOMAP_F_COW is a flag to no

Re: [PATCH v2 1/7] fsstress: allow fsync on directories too

2019-04-02 Thread Dave Chinner
On Mon, Apr 01, 2019 at 01:50:18PM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > Currently the fsync function can only be performed against regular files. > Allow it to operate on directories too, to increase test coverage and > allow for chances of finding bugs in a filesystem's im

Re: [PATCH 04/15] dax: Introduce IOMAP_F_COW for copy-on-write

2019-04-02 Thread Dave Chinner
On Tue, Apr 02, 2019 at 08:56:31PM -0500, Goldwyn Rodrigues wrote: > On 10:06 02/04, Dave Chinner wrote: > > On Mon, Apr 01, 2019 at 04:41:02PM -0500, Goldwyn Rodrigues wrote: > > > After Darrick's suggestion, we can even do away with cow_pos, so > > > only the read address of cow_addr will exist.

life time of backup roots

2019-04-02 Thread Chris Murphy
I'm sometimes seeing the same backup_tree_root used more than once. Below you'll see backup 0 and backup 2 have the same address, different generation. The concern is if this suggests backup 2 is actually stale and not useful? We've previously seen that Btrfs with discard mount option can very qui