Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午2:58, Andrey Ivanov wrote: > On 30.09.2019 9:27, Qu Wenruo wrote: >>> This is the kernel message for /dev/sda4. It is also have some problems, >>> but it is successfully mounted. I can't mount /dev/sdc1. >>> >>> Kernel message for /dev/sdc1: >>> [    6.503265] BTRFS info (device sd

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
On 30.09.2019 10:31, Qu Wenruo wrote: For this one, I could help you by just reverting that bit, and then you may be able to continue mounting the fs or at least run btrfs check on it. Please prepare an environment to compile btrfs-progs (at least btrfs-corrupt-block) if you want to try. Great

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午4:00, Andrey Ivanov wrote: > On 30.09.2019 10:31, Qu Wenruo wrote: >> For this one, I could help you by just reverting that bit, and then you >> may be able to continue mounting the fs or at least run btrfs check on >> it. >> >> Please prepare an environment to compile btrfs-progs

[PATCH] fstests: btrfs: Add regression test to check if btrfs can handle high devid

2019-09-30 Thread Qu Wenruo
Add a regression test to check if btrfs can handle high devid. The test will add and remove devices to a btrfs fs, so that the devid will increase to uncommon but still valid values. The regression is introduced by kernel commit ab4ba2e13346 ("btrfs: tree-checker: Verify dev item"). The fix is ti

Re: [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation

2019-09-30 Thread Nikolay Borisov
On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: > There are at least two bug reports of kernel tree-checker complaining > about invalid inode generation. > > All offending inodes seem to be caused by old kernel around 2014, with > inode generation overflow. > > So add such check and repair ability t

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午4:00, Andrey Ivanov wrote: > On 30.09.2019 10:31, Qu Wenruo wrote: >> For this one, I could help you by just reverting that bit, and then you >> may be able to continue mounting the fs or at least run btrfs check on >> it. >> >> Please prepare an environment to compile btrfs-progs

Re: [PATCH 2/3] btrfs-progs: check/original: Add check and repair for invalid inode generation

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午4:41, Nikolay Borisov wrote: > > > On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: >> There are at least two bug reports of kernel tree-checker complaining >> about invalid inode generation. >> >> All offending inodes seem to be caused by old kernel around 2014, with >> inode generation

[PATCH] Btrfs: fix memory leak due to concurrent append writes with fiemap

2019-09-30 Thread fdmanana
From: Filipe Manana When we have a buffered write that starts at an offset greater than or equals to the file's size happening concurrently with a full ranged fiemap, we can end up leaking an extent state structure. Suppose we have a file with a size of 1Mb, and before the buffered write and fie

RE: BTRFS checksum mismatch - false positives

2019-09-30 Thread hoegge
Hi Christ, Thank you very much for your help and insight. Has helped me a lot understanding the setup with BTRFS on top of SHR. I will rebuild the whole thing from scratch, as you recommend, since it seems only to get worse. The file mapping tool from Synology (synodataverifier) used on the fil

Re: BTRFS Raid5 error during Scrub.

2019-09-30 Thread Robert Krig
I've upgraded to btrfs-progs v5.2.1 Here is the output from btrfs check -p --readonly /dev/sda Opening filesystem to check... Checking filesystem on /dev/sda UUID: f7573191-664f-4540-a830-71ad654d9301 [1/7] checking root items (0:01:17 elapsed, 5138533 items checked) parent t

Re: [BTRFS Raid5 error during Scrub.

2019-09-30 Thread Graham Cobb
On 29/09/2019 22:38, Robert Krig wrote: > I'm running Debian Buster with Kernel 5.2. > Btrfs-progs v4.20.1 I am running Debian testing (bullseye) and have chosen not to install the 5.2 kernel yet because the version of it in bullseye (linux-image-5.2.0-2-amd64) is based on 5.2.9 and (as far as I c

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
On 30.09.2019 11:59, Qu Wenruo wrote: Please prepare an environment to compile btrfs-progs (at least btrfs-corrupt-block) if you want to try. Great, I'm ready to do it. Environment is ready. Here it is. https://github.com/adam900710/btrfs-progs/tree/dirty_fix To use it: $ ./btrfs-corrupt-b

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午6:01, Andrey Ivanov wrote: > On 30.09.2019 11:59, Qu Wenruo wrote: Please prepare an environment to compile btrfs-progs (at least btrfs-corrupt-block) if you want to try. >>> >>> Great, I'm ready to do it. Environment is ready. >> >> Here it is. >> >> https://github.com/a

Re: [PATCH] fstests: btrfs: Add regression test to check if btrfs can handle high devid

2019-09-30 Thread Anand Jain
On 9/30/19 4:37 PM, Qu Wenruo wrote: Add a regression test to check if btrfs can handle high devid. The test will add and remove devices to a btrfs fs, so that the devid will increase to uncommon but still valid values. The regression is introduced by kernel commit ab4ba2e13346 ("btrfs: tree

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
On 30.09.2019 13:10, Qu Wenruo wrote: I had built and run it: # /home/andrey/devel/src/btrfs/btrfs-progs-dirty_fix/btrfs-corrupt-block -X /dev/sdc1 incorrect offsets 15223 15287 Open ctree failed That branch updated to skip the item offset check completely. But please keep in mind that, that

Re: [PATCH] fstests: btrfs: Add regression test to check if btrfs can handle high devid

2019-09-30 Thread Filipe Manana
On Mon, Sep 30, 2019 at 9:39 AM Qu Wenruo wrote: > > Add a regression test to check if btrfs can handle high devid. > > The test will add and remove devices to a btrfs fs, so that the devid > will increase to uncommon but still valid values. > > The regression is introduced by kernel commit ab4ba2

Re: Btrfs partition mount error

2019-09-30 Thread Hans van Kranenburg
On 9/30/19 12:23 PM, Andrey Ivanov wrote: > > # /home/andrey/devel/src/btrfs/btrfs-progs-dirty_fix/btrfs-corrupt-block -X > /dev/sdc1 > key (613019873280 EXTENT_ITEM 1048576)slot end outside of leaf 1073755934 > > 16283 > Open ctree failed > > [...] bin(1073755934) '0b1110111000

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午6:23, Andrey Ivanov wrote: > On 30.09.2019 13:10, Qu Wenruo wrote: >>> I had built and run it: >>> >>> # /home/andrey/devel/src/btrfs/btrfs-progs-dirty_fix/btrfs-corrupt-block >>> -X /dev/sdc1 >>> incorrect offsets 15223 15287 >>> Open ctree failed >> >> That branch updated to skip

Re: [PATCH] fstests: btrfs: Add regression test to check if btrfs can handle high devid

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午6:33, Filipe Manana wrote: > On Mon, Sep 30, 2019 at 9:39 AM Qu Wenruo wrote: >> >> Add a regression test to check if btrfs can handle high devid. >> >> The test will add and remove devices to a btrfs fs, so that the devid >> will increase to uncommon but still valid values. >> >>

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午6:57, Qu Wenruo wrote: > > > On 2019/9/30 下午6:23, Andrey Ivanov wrote: >> On 30.09.2019 13:10, Qu Wenruo wrote: I had built and run it: # /home/andrey/devel/src/btrfs/btrfs-progs-dirty_fix/btrfs-corrupt-block -X /dev/sdc1 incorrect offsets 15223 15287 >>>

[PATCH] btrfs: removed unused return variable

2019-09-30 Thread Aliasgar Surti
From: Aliasgar Surti Removed unused return variable and replaced it with returning the value directly Signed-off-by: Aliasgar Surti --- fs/btrfs/disk-io.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 044981c..c80fa67 100

Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation

2019-09-30 Thread Nikolay Borisov
On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: > There are at least two bug reports of kernel tree-checker complaining > about invalid inode generation. > > All offending inodes seem to be caused by old kernel around 2014, with > inode generation overflow. > > So add such check and repair ability t

Re: [PATCH] btrfs: removed unused return variable

2019-09-30 Thread Nikolay Borisov
On 30.09.19 г. 14:17 ч., Aliasgar Surti wrote: > From: Aliasgar Surti > > Removed unused return variable and replaced it with returning > the value directly > > Signed-off-by: Aliasgar Surti Actually this function should be turned to void and even the declaration at the top of disk-io can b

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
We have another report internally about a similar corruption (multiple bit flips in a single fs), and they are also using VMware, along with VMware guest kernel modules. Would you mind to migrate to KVM based hypervisor to see if the corruption happens again? I had this problem with btrfs afte

Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午7:36, Nikolay Borisov wrote: > > > On 24.09.19 г. 11:11 ч., Qu Wenruo wrote: >> There are at least two bug reports of kernel tree-checker complaining >> about invalid inode generation. >> >> All offending inodes seem to be caused by old kernel around 2014, with >> inode generation

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午8:10, Andrey Ivanov wrote: >> >> We have another report internally about a similar corruption (multiple >> bit flips in a single fs), and they are also using VMware, along with >> VMware guest kernel modules. >> >> Would you mind to migrate to KVM based hypervisor to see if the >>

Re: [PATCH] btrfs: removed unused return variable

2019-09-30 Thread Aliasgar Surti
On Mon, Sep 30, 2019 at 02:53:12PM +0300, Nikolay Borisov wrote: > > > On 30.09.19 г. 14:17 ч., Aliasgar Surti wrote: > > From: Aliasgar Surti > > > > Removed unused return variable and replaced it with returning > > the value directly > > > > Signed-off-by: Aliasgar Surti > > Actually this

[PATCH v2] btrfs: removed unused return variable

2019-09-30 Thread Aliasgar Surti
From: Aliasgar Surti Removed unused return variable and replaced it with returning the value directly Signed-off-by: Aliasgar Surti --- v2: Made btrfs_destroy_delayed_refs() as void and removed its declaration --- fs/btrfs/disk-io.c | 9 ++--- 1 file changed, 2 insertions(+), 7 deletions(-

Re: [PATCH v2] btrfs: removed unused return variable

2019-09-30 Thread Nikolay Borisov
On 30.09.19 г. 16:00 ч., Aliasgar Surti wrote: > From: Aliasgar Surti > > Removed unused return variable and replaced it with returning > the value directly The changelog could be expanded, something like: btrfs_destroy_delayed_refs always returns 0 so it can be converted to void. While at i

Re: [PATCH v2] btrfs: removed unused return variable

2019-09-30 Thread David Sterba
On Mon, Sep 30, 2019 at 06:30:21PM +0530, Aliasgar Surti wrote: > From: Aliasgar Surti > > Removed unused return variable and replaced it with returning > the value directly This change has been sent several times and I give the same answer each time: https://lore.kernel.org/linux-btrfs/2019041

Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation

2019-09-30 Thread Nikolay Borisov
On 30.09.19 г. 15:24 ч., Qu Wenruo wrote: > Yes, the ASSERT() doesn't make much sense by itself. > > However I still believe it won't be a problem. It won't be a problem but it feels wrong to have this assert this deep into the call chain. IMO It should be put where it can trigger at the earli

Re: [PATCH] Btrfs: fix memory leak due to concurrent append writes with fiemap

2019-09-30 Thread Sasha Levin
Hi, [This is an automated email] This commit has been processed because it contains a -stable tag. The stable tag indicates that it's relevant for the following trees: 4.16+ The bot has tested the following trees: v5.3.1, v5.2.17, v4.19.75. v5.3.1: Build OK! v5.2.17: Build OK! v4.19.75: Failed

Re: [PATCH] Btrfs: fix memory leak due to concurrent append writes with fiemap

2019-09-30 Thread Josef Bacik
On Mon, Sep 30, 2019 at 10:20:25AM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > When we have a buffered write that starts at an offset greater than or > equals to the file's size happening concurrently with a full ranged > fiemap, we can end up leaking an extent state structure. >

Re: [PATCH 1/3] btrfs-progs: check/lowmem: Add check and repair for invalid inode generation

2019-09-30 Thread Qu Wenruo
On 2019/9/30 下午9:34, Nikolay Borisov wrote: > > > On 30.09.19 г. 15:24 ч., Qu Wenruo wrote: >> Yes, the ASSERT() doesn't make much sense by itself. >> >> However I still believe it won't be a problem. > > It won't be a problem but it feels wrong to have this assert this deep > into the call chai

[GIT PULL] Btrfs fixes for 5.4-rc2

2019-09-30 Thread David Sterba
Hi, a bunch of fixes that accumulated in recent weeks, mostly material for stable. Summary: - fix for regression from 5.3 that prevents to use balance convert with single profile - qgroup fixes: rescan race, accounting leak with multiple writers, potential leak after io failure recovery -

[PATCH] Setup GitLab-CI for btrfs-progs

2019-09-30 Thread Lakshmipathi.G
Make use of GitLab-CI nested virutal environment to start QEMU instance inside containers and perform btrfs-progs build, execute unit test cases and save the logs. More details can be found at https://github.com/kdave/btrfs-progs/issues/171 Signed-off-by: Lakshmipathi.G --- .gitlab-ci.yml

[PATCH] btrfs: fix incorrect updating of log root tree

2019-09-30 Thread Josef Bacik
We've historically had reports of being unable to mount file systems because the tree log root couldn't be read. Usually this is the "parent transid failure", but could be any of the related errors, including "fsid mismatch" or "bad tree block", depending on which block got allocated. The modific

BTRFS corruption, mounts but comes read-only

2019-09-30 Thread Remi Gauvin
Quick history, the system was rejecting ssh connections, and was not responding to ACPI shutdown, so I had to force power off. On restart, everything looked ok, but when I did a scrub, the system quickly reached a point of forced read-only., (kernel Ubuntu 4.15.0-52) I installed a current Arch fo

Re: [GIT PULL] Btrfs fixes for 5.4-rc2

2019-09-30 Thread pr-tracker-bot
The pull request you sent on Mon, 30 Sep 2019 16:25:08 +0200: > git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-5.4-rc1-tag has been merged into torvalds/linux.git: https://git.kernel.org/torvalds/c/bb48a59135926ece9b1361e8b96b33fc658830bc Thank you! -- Deet-doot-dot, I am a

Re: [PATCH][v3] btrfs: use refcount_inc_not_zero in kill_all_nodes

2019-09-30 Thread David Sterba
On Thu, Sep 26, 2019 at 08:29:32AM -0400, Josef Bacik wrote: > We hit the following warning while running down a different problem > > [ 6197.175850] [ cut here ] > [ 6197.185082] refcount_t: underflow; use-after-free. > [ 6197.194704] WARNING: CPU: 47 PID: 966 at lib/refco

Re: BTRFS corruption, mounts but comes read-only

2019-09-30 Thread Remi Gauvin
Mounting the FS after that btrfs check, kernel 5.3.1, here is the dmesg log: [10524.830640] BTRFS info (device sdf4): using free space tree [10524.830644] BTRFS info (device sdf4): has skinny extents [10549.561143] BTRFS info (device sdf4): checking UUID tree [10606.900264] BTRFS info (device sdf4

Re: [PATCH RFC] btrfs: relocation: Hunt down BUG_ON() in merge_reloc_roots()

2019-09-30 Thread David Sterba
On Thu, Sep 26, 2019 at 02:35:45PM +0800, Qu Wenruo wrote: > [BUG] > There is one BUG_ON() report where a transaction is aborted during > balance, then kernel BUG_ON() in merge_reloc_roots(): Do you have details from the report, eg. what's the error code? > void merge_reloc_roots(struct reloc_c

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
On 30.09.2019 15:27, Qu Wenruo wrote: We have another report internally about a similar corruption (multiple bit flips in a single fs), and they are also using VMware, along with VMware guest kernel modules. Would you mind to migrate to KVM based hypervisor to see if the corruption happens again

Re: [PATCH] Btrfs: fix memory leak due to concurrent append writes with fiemap

2019-09-30 Thread David Sterba
On Mon, Sep 30, 2019 at 10:20:25AM +0100, fdman...@kernel.org wrote: > From: Filipe Manana > > When we have a buffered write that starts at an offset greater than or > equals to the file's size happening concurrently with a full ranged > fiemap, we can end up leaking an extent state structure. >

Re: [PATCH RFC] btrfs: relocation: Hunt down BUG_ON() in merge_reloc_roots()

2019-09-30 Thread Qu Wenruo
On 2019/10/1 上午1:51, David Sterba wrote: > On Thu, Sep 26, 2019 at 02:35:45PM +0800, Qu Wenruo wrote: >> [BUG] >> There is one BUG_ON() report where a transaction is aborted during >> balance, then kernel BUG_ON() in merge_reloc_roots(): > > Do you have details from the report, eg. what's the er

Re: BTRFS corruption, mounts but comes read-only

2019-09-30 Thread Qu Wenruo
On 2019/10/1 上午5:08, Remi Gauvin wrote: > Mounting the FS after that btrfs check, kernel 5.3.1, here is the dmesg log: As that btrfs check shows, your extent tree is corrupted. Quite some backref is lost, thus no wonder some write opeartion would cause problem. In that case, it looks like only

Re: BTRFS corruption, mounts but comes read-only

2019-09-30 Thread Remi Gauvin
On 2019-09-30 9:30 p.m., Qu Wenruo wrote: > > > On 2019/10/1 上午5:08, Remi Gauvin wrote: >> Mounting the FS after that btrfs check, kernel 5.3.1, here is the dmesg log: > > As that btrfs check shows, your extent tree is corrupted. > Quite some backref is lost, thus no wonder some write opeartion

Re: BTRFS corruption, mounts but comes read-only

2019-09-30 Thread Qu Wenruo
On 2019/10/1 上午9:41, Remi Gauvin wrote: > On 2019-09-30 9:30 p.m., Qu Wenruo wrote: >> >> >> On 2019/10/1 上午5:08, Remi Gauvin wrote: >>> Mounting the FS after that btrfs check, kernel 5.3.1, here is the dmesg log: >> >> As that btrfs check shows, your extent tree is corrupted. >> Quite some backr

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
On 30.09.2019 9:27, Qu Wenruo wrote: I recommend to do a "btrfs check" on all fses. I had done "btrfs check" on /dev/sda4: attached btrfs-check-sda4.output There are some errors. How to fix them? [1/7] checking root items [2/7] checking extents incorrect local backref count on 1533538304 root

Re: Btrfs partition mount error

2019-09-30 Thread Andrey Ivanov
On 30.09.2019 10:31, Qu Wenruo wrote: Please provide the following dump: # btrfs ins dump-tree -b 855738744832 /dev/sdc1 attached btrfs-ins-dump-tree-855738744832-sdc1.output OK, tree-checker is again detecting the problem correctly. item 19 key (613039370240 EXTENT_ITEM 1048576) it

Re: Btrfs partition mount error

2019-09-30 Thread Qu Wenruo
On 2019/10/1 下午12:08, Andrey Ivanov wrote: > On 30.09.2019 9:27, Qu Wenruo wrote: >> I recommend to do a "btrfs check" on all fses. > > I had done "btrfs check" on /dev/sda4: > > attached btrfs-check-sda4.output > > There are some errors. How to fix them? It looks like "btrfs check --repair"

Re: [PATCH] Setup GitLab-CI for btrfs-progs

2019-09-30 Thread Philipp Hahn
Hi, I'm not yet a GitLab expert myself, but AFAIK ... Am 30.09.19 um 18:56 schrieb Lakshmipathi.G: > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > new file mode 100644 > index 000..2afde50 > --- /dev/null > +++ b/.gitlab-ci.yml > @@ -0,0 +1,181 @@ > +# This program is free software; you can

[PATCH v3] btrfs: removed unused return variable

2019-09-30 Thread Aliasgar Surti
From: Aliasgar Surti Removed unused return variable and replaced it with returning the value directly. Also removed the unnecessary forward declaration. Signed-off-by: Aliasgar Surti --- v2: Made btrfs_destroy_delayed_refs() as void and removed its declaration v3: Reverted back the change where