Re: Btrfs broken in massive transfar

2018-03-14 Thread MASAKI,
Changing crypt layer didn't solve this probrem... > > On Tue, Mar 13, 2018 at 1:25 PM, MASAKI haruka wrote: > > > journal(Kernel log), 7th try (to be readonly): > > > > > > --- > > > 3月 12 16:25:51 lily kernel: BTRFS info (device dm-6): creating UUID tree > > > 3月 12 16:25:53 lily iscsid[1406]

Re: WARN when unmounting a subvolume that is being synced

2018-03-14 Thread Nikolay Borisov
On 14.03.2018 05:10, Tycho Andersen wrote: > Hi all, > > I'm getting the WARN below. I think (?) what I'm doing when I get it > is that I'm unmounting a subvolume while it's being synced (concurrent > uses of the subvolume, at least, happy to look into it further if the > stack trace is not so u

Re: [PATCH 1/2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread Nikolay Borisov
On 13.03.2018 20:47, fdman...@kernel.org wrote: > From: Filipe Manana > > Under some cases the filesystem checker reports an error when it finds > checksum items for an extent that is referenced by an inode as a prealloc > extent. Such cases are not an error when the extent is actually shared >

[PATCH v2 1/2] btrfs: drop num argument from find_live_mirror()

2018-03-14 Thread Anand Jain
Obtain the stripes info from the map directly and so no need to pass it as an argument. Signed-off-by: Anand Jain --- v1->v2: Accepts David's comment to rename %num to %num_stripes. fs/btrfs/volumes.c | 16 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/fs/btr

[PATCH v2 2/2] btrfs: drop optimal argument from find_live_mirror()

2018-03-14 Thread Anand Jain
Drop optimal argument from the function find_live_mirror() as we can deduce it in the function itself. Also rename optimal to preferred_mirror. Signed-off-by: Anand Jain --- I thought I have sent v2 to the ML. But now I realize I didn't. v1->v2: Accepts David's comment to rename %optimal. IMO,

Re: FS unmountable after RAID/LVM problems

2018-03-14 Thread Dirk Gouders
Qu Wenruo writes: > On 2018年03月13日 22:49, Dirk Gouders wrote: > [snip] >>> >>> # btrfs inspect dump-tree -b 848986112 /dev/loop0p1 >>> # btrfs inspect dump-tree -b 72089600 /dev/loop0p1 >> >> OK. >> >> (This mail gets a bit long but I don't want to snip probably important >> information above.

[PATCH 1/3] fstests: log-writes: Add support to output human readable flags

2018-03-14 Thread Qu Wenruo
Also change the flag numeric output to hex. Signed-off-by: Qu Wenruo --- src/log-writes/log-writes.c | 70 - 1 file changed, 63 insertions(+), 7 deletions(-) diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c index 09391574..a87242

[PATCH RFC 3/3] fstests: generic: Check the fs after each FUA writes

2018-03-14 Thread Qu Wenruo
Basic test case which triggers fsstress with dm-log-writes, and then check the fs after each FUA writes. With needed infrastructure and special handlers for journal based fs. Signed-off-by: Qu Wenruo --- In my test, xfs and btrfs survives while ext4 would report error during fsck. My current big

[PATCH 2/3] fstests: log-writes: Add support for METADATA flag

2018-03-14 Thread Qu Wenruo
Signed-off-by: Qu Wenruo --- src/log-writes/log-writes.c | 3 ++- src/log-writes/log-writes.h | 9 + 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c index a872429d..5dc22c24 100644 --- a/src/log-writes/log-writes.c +

Re: FS unmountable after RAID/LVM problems

2018-03-14 Thread Qu Wenruo
On 2018年03月14日 16:53, Dirk Gouders wrote: > Qu Wenruo writes: > >> On 2018年03月13日 22:49, Dirk Gouders wrote: >> [snip] # btrfs inspect dump-tree -b 848986112 /dev/loop0p1 # btrfs inspect dump-tree -b 72089600 /dev/loop0p1 >>> >>> OK. >>> >>> (This mail gets a bit long but I don't

Re: [PATCH 1/2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread Nikolay Borisov
On 13.03.2018 20:47, fdman...@kernel.org wrote: > From: Filipe Manana > > Under some cases the filesystem checker reports an error when it finds > checksum items for an extent that is referenced by an inode as a prealloc > extent. Such cases are not an error when the extent is actually shared >

Re: [RFC PATCH] btrfs: check for SB checksum when scanned

2018-03-14 Thread Nikolay Borisov
On 13.03.2018 17:06, Anand Jain wrote: > We aren't checking the SB csum when the device scanned, > instead we do that when mounting the device, and if the > csum fails we fail the mount. How if we check the csum > when the device is scanned, I can't see any reason for > why not? any idea? So wha

Re: [PATCH v2 1/2] btrfs: drop num argument from find_live_mirror()

2018-03-14 Thread Nikolay Borisov
On 14.03.2018 10:29, Anand Jain wrote: > Obtain the stripes info from the map directly and so no need > to pass it as an argument. > > Signed-off-by: Anand Jain LGTM Reviewed-by: Nikolay Borisov > --- > v1->v2: > Accepts David's comment to rename %num to %num_stripes. > > fs/btrfs/volum

Re: [PATCH v2 2/2] btrfs: drop optimal argument from find_live_mirror()

2018-03-14 Thread Nikolay Borisov
On 14.03.2018 10:29, Anand Jain wrote: > Drop optimal argument from the function find_live_mirror() > as we can deduce it in the function itself. Also rename > optimal to preferred_mirror. > > Signed-off-by: Anand Jain Reviewed-by: Nikolay Borisov > --- > I thought I have sent v2 to the ML.

Re: FS unmountable after RAID/LVM problems

2018-03-14 Thread Dirk Gouders
Qu Wenruo writes: > On 2018年03月14日 16:53, Dirk Gouders wrote: >> Qu Wenruo writes: >> >>> On 2018年03月13日 22:49, Dirk Gouders wrote: >>> [snip] > > # btrfs inspect dump-tree -b 848986112 /dev/loop0p1 > # btrfs inspect dump-tree -b 72089600 /dev/loop0p1 OK. (This m

Re: FS unmountable after RAID/LVM problems

2018-03-14 Thread Qu Wenruo
On 2018年03月14日 17:36, Dirk Gouders wrote: > Qu Wenruo writes: > >> On 2018年03月14日 16:53, Dirk Gouders wrote: >>> Qu Wenruo writes: >>> On 2018年03月13日 22:49, Dirk Gouders wrote: [snip] >> >> # btrfs inspect dump-tree -b 848986112 /dev/loop0p1 >> # btrfs inspect dump-tree -

Re: [PATCH 1/3] fstests: log-writes: Add support to output human readable flags

2018-03-14 Thread Amir Goldstein
On Wed, Mar 14, 2018 at 11:02 AM, Qu Wenruo wrote: > Also change the flag numeric output to hex. > > Signed-off-by: Qu Wenruo Reviewed-by: Amir Goldstein > --- > src/log-writes/log-writes.c | 70 > - > 1 file changed, 63 insertions(+), 7 deletions(-

Re: [PATCH 2/3] fstests: log-writes: Add support for METADATA flag

2018-03-14 Thread Amir Goldstein
On Wed, Mar 14, 2018 at 11:02 AM, Qu Wenruo wrote: > Signed-off-by: Qu Wenruo Reviewed-by: Amir Goldstein > --- > src/log-writes/log-writes.c | 3 ++- > src/log-writes/log-writes.h | 9 + > 2 files changed, 7 insertions(+), 5 deletions(-) > > diff --git a/src/log-writes/log-writes.c b/

Re: FS unmountable after RAID/LVM problems

2018-03-14 Thread Qu Wenruo
On 2018年03月14日 18:06, Dirk Gouders wrote: > Qu Wenruo writes: > >> On 2018年03月14日 17:36, Dirk Gouders wrote: >>> Qu Wenruo writes: >>> On 2018年03月14日 16:53, Dirk Gouders wrote: > Qu Wenruo writes: > >> On 2018年03月13日 22:49, Dirk Gouders wrote: >> [snip]

Re: [PATCH RFC 3/3] fstests: generic: Check the fs after each FUA writes

2018-03-14 Thread Amir Goldstein
On Wed, Mar 14, 2018 at 11:02 AM, Qu Wenruo wrote: > Basic test case which triggers fsstress with dm-log-writes, and then > check the fs after each FUA writes. > With needed infrastructure and special handlers for journal based fs. > > Signed-off-by: Qu Wenruo > --- > In my test, xfs and btrfs su

Re: [PATCH 1/2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread Filipe Manana
On Wed, Mar 14, 2018 at 1:32 AM, Qu Wenruo wrote: > > > On 2018年03月14日 02:47, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Under some cases the filesystem checker reports an error when it finds >> checksum items for an extent that is referenced by an inode as a prealloc >> extent. Such

Re: [PATCH RFC 3/3] fstests: generic: Check the fs after each FUA writes

2018-03-14 Thread Qu Wenruo
On 2018年03月14日 18:27, Amir Goldstein wrote: > On Wed, Mar 14, 2018 at 11:02 AM, Qu Wenruo wrote: >> Basic test case which triggers fsstress with dm-log-writes, and then >> check the fs after each FUA writes. >> With needed infrastructure and special handlers for journal based fs. >> >> Signed-of

Re: [PATCH 1/2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread Filipe Manana
On Wed, Mar 14, 2018 at 8:19 AM, Nikolay Borisov wrote: > > > On 13.03.2018 20:47, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Under some cases the filesystem checker reports an error when it finds >> checksum items for an extent that is referenced by an inode as a prealloc >> extent.

Re: [PATCH 1/2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread Filipe Manana
On Wed, Mar 14, 2018 at 9:19 AM, Nikolay Borisov wrote: > > > On 13.03.2018 20:47, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Under some cases the filesystem checker reports an error when it finds >> checksum items for an extent that is referenced by an inode as a prealloc >> extent.

Re: [PATCH 1/2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread Qu Wenruo
On 2018年03月14日 18:33, Filipe Manana wrote: > On Wed, Mar 14, 2018 at 1:32 AM, Qu Wenruo wrote: >> >> >> On 2018年03月14日 02:47, fdman...@kernel.org wrote: >>> From: Filipe Manana >>> >>> Under some cases the filesystem checker reports an error when it finds >>> checksum items for an extent that i

RE: [PATCH v3] kernel.h: Skip single-eval logic on literals in min()/max()

2018-03-14 Thread David Laight
From: Kees Cook > Sent: 13 March 2018 22:15 ... > I'll send a "const_max()" which will refuse to work on > non-constant-values (so it doesn't get accidentally used on variables > that could be exposed to double-evaluation), and will work for stack > array declarations (to avoid the overly-sensitive

Re: Ongoing Btrfs stability issues

2018-03-14 Thread Austin S. Hemmelgarn
On 2018-03-13 15:36, Goffredo Baroncelli wrote: On 03/12/2018 10:48 PM, Christoph Anton Mitterer wrote: On Mon, 2018-03-12 at 22:22 +0100, Goffredo Baroncelli wrote: Unfortunately no, the likelihood might be 100%: there are some patterns which trigger this problem quite easily. See The link whi

Re: [RFC PATCH] btrfs: check for SB checksum when scanned

2018-03-14 Thread Austin S. Hemmelgarn
On 2018-03-14 05:20, Nikolay Borisov wrote: On 13.03.2018 17:06, Anand Jain wrote: We aren't checking the SB csum when the device scanned, instead we do that when mounting the device, and if the csum fails we fail the mount. How if we check the csum when the device is scanned, I can't see any re

Re: WARN when unmounting a subvolume that is being synced

2018-03-14 Thread Tycho Andersen
On Wed, Mar 14, 2018 at 09:46:07AM +0200, Nikolay Borisov wrote: > > > On 14.03.2018 05:10, Tycho Andersen wrote: > > Hi all, > > > > I'm getting the WARN below. I think (?) what I'm doing when I get it > > is that I'm unmounting a subvolume while it's being synced (concurrent > > uses of the su

[PATCH] generic/015: Issue sync after deleting the fillup file

2018-03-14 Thread Nikolay Borisov
This test fails on btrfs due to the presence of delayed processing of file deletes if the file is smaller than 32mb. Initially commit 97575acd7495b412435d06229a6d94ed9a814ada tried to fix a similar failure by bumping the size of the filesystem. However that change had a knock-on effect in that the

[PATCH 1/2 v2] Btrfs-progs: check, fix false error reports for shared prealloc extents

2018-03-14 Thread fdmanana
From: Filipe Manana Under some cases the filesystem checker reports an error when it finds checksum items for an extent that is referenced by an inode as a prealloc extent. Such cases are not an error when the extent is actually shared (was cloned/reflinked) with other inodes and was written thro

[PATCH 2/2 v2] Btrfs-progs: add fsck test for filesystem with shared prealloc extents

2018-03-14 Thread fdmanana
From: Filipe Manana Verify that a filesystem check operation (fsck) does not report the following scenario as an error: An extent is shared between two inodes, as a result of clone/reflink operation, and for one of the inodes, lets call it inode A, the extent is referenced through a file extent

[PATCH] btrfs: add missing initialization in btrfs_check_shared

2018-03-14 Thread Edmund Nadolski
btrfs_check_shared calls find_parent_nodes in a loop. In each iteration it passes in a share_check struct to request a check for a shared extent. The share_check::share_count must be re-initialized to zero for each iteration in order to avoid using a stale count value from the previous iteration,

[PATCH] missing initialization in btrfs_check_shared

2018-03-14 Thread Edmund Nadolski
This patch addresses an issue that causes fiemap to falsely report a shared extent. The test case is as follows: # cat do_xfs_io xfs_io -f -d -c "pwrite -b 16k 0 64k" -c "fiemap -v" /media/scratch/file5 sync xfs_io -c "fiemap -v" /media/scratch/file5 which gives the resulting output: # . do_xf

Re: [PATCH] missing initialization in btrfs_check_shared

2018-03-14 Thread Filipe Manana
On Wed, Mar 14, 2018 at 3:03 PM, Edmund Nadolski wrote: > This patch addresses an issue that causes fiemap to falsely > report a shared extent. The test case is as follows: > > # cat do_xfs_io > xfs_io -f -d -c "pwrite -b 16k 0 64k" -c "fiemap -v" /media/scratch/file5 > sync > xfs_io -c "fiemap

Re: [PATCH] btrfs: add missing initialization in btrfs_check_shared

2018-03-14 Thread Nikolay Borisov
On 14.03.2018 17:03, Edmund Nadolski wrote: > btrfs_check_shared calls find_parent_nodes in a loop. In each > iteration it passes in a share_check struct to request a check for > a shared extent. The share_check::share_count must be re-initialized > to zero for each iteration in order to avoid u

Re: [PATCH] btrfs: add missing initialization in btrfs_check_shared

2018-03-14 Thread David Sterba
On Wed, Mar 14, 2018 at 09:03:11AM -0600, Edmund Nadolski wrote: > btrfs_check_shared calls find_parent_nodes in a loop. In each > iteration it passes in a share_check struct to request a check for > a shared extent. The share_check::share_count must be re-initialized > to zero for each iteration

Re: [RFC PATCH] btrfs: check for SB checksum when scanned

2018-03-14 Thread David Sterba
On Tue, Mar 13, 2018 at 11:06:37PM +0800, Anand Jain wrote: > We aren't checking the SB csum when the device scanned, > instead we do that when mounting the device, and if the > csum fails we fail the mount. How if we check the csum > when the device is scanned, I can't see any reason for > why not

Re: [RFC PATCH] btrfs: check for SB checksum when scanned

2018-03-14 Thread David Sterba
On Wed, Mar 14, 2018 at 11:20:41AM +0200, Nikolay Borisov wrote: > > > On 13.03.2018 17:06, Anand Jain wrote: > > We aren't checking the SB csum when the device scanned, > > instead we do that when mounting the device, and if the > > csum fails we fail the mount. How if we check the csum > > when

Re: [PATCH] Btrfs: drop a VLA in btrfs_check_super_csum()

2018-03-14 Thread David Sterba
On Tue, Mar 13, 2018 at 08:50:22PM +0100, Salvatore Mesoraca wrote: > Avoid a VLA[1] by using a real constant expression instead of a variable. > The compiler should be able to optimize the original code and avoid using > an actual VLA. Anyway this change is useful because it will avoid a false > p

Re: [RFC PATCH] btrfs: check for SB checksum when scanned

2018-03-14 Thread David Sterba
On Tue, Mar 13, 2018 at 11:06:37PM +0800, Anand Jain wrote: > We aren't checking the SB csum when the device scanned, > instead we do that when mounting the device, and if the > csum fails we fail the mount. How if we check the csum > when the device is scanned, I can't see any reason for > why not

Re: [PATCH] Btrfs: drop a VLA in btrfs_check_super_csum()

2018-03-14 Thread Salvatore Mesoraca
2018-03-14 16:42 GMT+01:00 David Sterba : > We already have a patch for that from a few days ago, but thanks anyway. > > https://patchwork.kernel.org/patch/10277901/ Oh.. OK Best regards, Salvatore -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message

Re: [PATCH v2] btrfs: Use sizeof directly instead of a constant variable

2018-03-14 Thread David Sterba
On Tue, Mar 13, 2018 at 10:26:06AM +0200, Nikolay Borisov wrote: > The kernel would like to have all stack VLA usage removed[1]. > Unfortunately using an integer constant variable as the size of an > array is still considered a VLA. Instead let's use directly sizeof(var) > which removes the VLA usa

Re: Ongoing Btrfs stability issues

2018-03-14 Thread Goffredo Baroncelli
On 03/14/2018 01:02 PM, Austin S. Hemmelgarn wrote: [...] >> >> In btrfs, a checksum mismatch creates an -EIO error during the reading. In a >> conventional filesystem (or a btrfs filesystem w/o datasum) there is no >> checksum, so this problem doesn't exist. >> >> I am curious how ZFS solves thi

Re: [PATCH] btrfs: Handle error from btrfs_uuid_tree_rem call in _btrfs_ioctl_set_received_subvol

2018-03-14 Thread David Sterba
On Mon, Mar 12, 2018 at 02:48:09PM +0200, Nikolay Borisov wrote: > As with every function which deals with modifying the btree > btrfs_uuid_tree_rem can fail for any number of reasons (ie. EIO/ENOMEM). > Handle return error value from this function gracefully by aborting the > transaction. > > Fix

Re: Ongoing Btrfs stability issues

2018-03-14 Thread Austin S. Hemmelgarn
On 2018-03-14 14:39, Goffredo Baroncelli wrote: On 03/14/2018 01:02 PM, Austin S. Hemmelgarn wrote: [...] In btrfs, a checksum mismatch creates an -EIO error during the reading. In a conventional filesystem (or a btrfs filesystem w/o datasum) there is no checksum, so this problem doesn't exis

Re: zerofree btrfs support?

2018-03-14 Thread David Sterba
On Sat, Mar 10, 2018 at 03:55:25AM +0100, Christoph Anton Mitterer wrote: > Hi. > > Just wondered... was it ever planned (or is there some equivalent) to > get support for btrfs in zerofree? The zerofree was requested, eg. here https://bugzilla.kernel.org/show_bug.cgi?id=69121 and there was a di

Re: [PATCH] btrfs: Document parameters of btrfs_reserve_extent

2018-03-14 Thread David Sterba
On Tue, Mar 13, 2018 at 09:56:23PM +0800, Anand Jain wrote: > > > On 03/13/2018 06:22 PM, Nikolay Borisov wrote: > > This function is the entry to the extent allocator and as such has > > quite a number of parameters. Some of those have subtle effects on the > > allocation algorithm. Document the

Re: [PATCH] btrfs: volumes: Remove the meaningless condition of minimal nr_devs when allocating a chunk

2018-03-14 Thread David Sterba
On Wed, Jan 31, 2018 at 01:56:15PM +0800, Qu Wenruo wrote: > When checking the minimal nr_devs, there is one dead and meaningless > condition: > > if (ndevs < devs_increment * sub_stripes || ndevs < devs_min) { > > > This condition is meaningless, @devs_in

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

2018-03-14 Thread David Sterba
On Tue, Feb 13, 2018 at 09:13:32AM +0800, Qu Wenruo wrote: > There are reports in mail list, even with latest mainline kernel, btrfs > can't survive a power loss. > > Unlike journal based filesystem, btrfs doesn't use journal for such > work. (log tree is an optimization for fsync, not to keep fs

Re: [PATCH 1/2] btrfs: Remove extra run_delayed_items call

2018-03-14 Thread David Sterba
On Tue, Feb 13, 2018 at 04:16:48PM +0200, Nikolay Borisov wrote: > Before commit 581227d0d2b8 ("Btrfs: remove the time check in > btrfs_commit_transaction()") there would be a loop in the transaction > commit path that will go on until all existing writers ended their > transaction handles. This lo

Re: Ongoing Btrfs stability issues

2018-03-14 Thread Goffredo Baroncelli
On 03/14/2018 08:27 PM, Austin S. Hemmelgarn wrote: > On 2018-03-14 14:39, Goffredo Baroncelli wrote: >> On 03/14/2018 01:02 PM, Austin S. Hemmelgarn wrote: >> [...] In btrfs, a checksum mismatch creates an -EIO error during the reading. In a conventional filesystem (or a btrfs file

Re: [PATCH] btrfs: Verify extent allocated by find_free_extent() won't overlap with extents from previous transaction

2018-03-14 Thread Qu Wenruo
On 2018年03月15日 05:09, David Sterba wrote: > On Tue, Feb 13, 2018 at 09:13:32AM +0800, Qu Wenruo wrote: >> There are reports in mail list, even with latest mainline kernel, btrfs >> can't survive a power loss. >> >> Unlike journal based filesystem, btrfs doesn't use journal for such >> work. (log

Re: zerofree btrfs support?

2018-03-14 Thread Christoph Anton Mitterer
Hey. On Wed, 2018-03-14 at 20:38 +0100, David Sterba wrote: > I have a prototype code for that and after the years, seeing the > request > again, I'm not against adding it as long as it's not advertised as a > security feature. I'd expect that anyone in the security area should know that securely

[PATCH] btrfs-progs: print-tree: Enhance warning on tree block level mismatch and error handling

2018-03-14 Thread Qu Wenruo
This patch enhance the tree block level mismatch by the following methods: 1) Merge same warning branches into one We had two branches showing the same message, and their condition is also the same. Merge them 2) Only skip bad slot The old code skipped all the remaining slots, here we ju