bcachefs snapshots design doc - RFC

2021-03-17 Thread Kent Overstreet
Snapshots for bcaches are well under way, and I've written a design doc for them. I'd love to get feedback on anything I might have missed, especially from the btrfs people. The current version of this document lives at https://bcachefs.org/Snapshots/ and the in-progress code lives at https:/

Re: [PATCH] btrfs: Use immediate assignment when referencing cc-option

2021-03-17 Thread David Sterba
On Tue, Mar 16, 2021 at 03:46:10PM -0700, Victor Erminpour wrote: > Calling cc-option will use KBUILD_CFLAGS, which when lazy setting > subdir-ccflags-y produces the following build error: > > scripts/Makefile.lib:10: *** Recursive variable `KBUILD_CFLAGS' \ > references itself (eventually).

[PATCH] btrfs: zoned: remove outdated WARN_ON in direct IO

2021-03-17 Thread Johannes Thumshirn
In btrfs_submit_direct() there's a WAN_ON_ONCE() that will trigger if we're submitting a DIO write on a zoned filesystem but are not using REQ_OP_ZONE_APPEND to submit the IO to the block device. This is a left over from a previous version where btrfs_dio_iomap_begin() didn't use btrfs_use_zone_ap

Re: All files are damaged after btrfs restore

2021-03-17 Thread Sebastian Roller
Am Mi., 17. März 2021 um 03:59 Uhr schrieb Chris Murphy : > > On Tue, Mar 16, 2021 at 7:39 PM Qu Wenruo wrote: > > > Using that restore I was able to restore approx. 7 TB of the > > > originally stored 22 TB under that directory. > > > Unfortunately nearly all the files are damaged. Small text fil

[PATCH 5.4] btrfs: scrub: Don't check free space before marking a block group RO

2021-03-17 Thread David Sterba
From: Qu Wenruo [ Upstream commit b12de52896c0e8213f70e3a168fde9e6eee95909 ] [BUG] When running btrfs/072 with only one online CPU, it has a pretty high chance to fail: btrfs/072 12s ... _check_dmesg: something found in dmesg (see xfstests-dev/results//btrfs/072.dmesg) - output mismatch (s

Re: [RFC] btrfs: Allow read-only mount with corrupted extent tree

2021-03-17 Thread Qu Wenruo
On 2021/3/17 上午9:29, Dāvis Mosāns wrote: trešd., 2021. g. 17. marts, plkst. 03:18 — lietotājs Dāvis Mosāns () rakstīja: Currently if there's any corruption at all in extent tree (eg. even single bit) then mounting will fail with: "failed to read block groups: -5" (-EIO) It happens because we

[PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread Johannes Thumshirn
When a file gets deleted on a zoned file system, the space freed is no returned back into the block group's free space, but is migrated to zone_unusable. As this zone_unusable space is behind the current write pointer it is not possible to use it for new allocations. In the current implementation

[PATCH] btrfs: fix build when using M=fs/btrfs

2021-03-17 Thread David Sterba
There are people building the module with M= that's supposed to be used for external modules. This got broken in e9aa7c285d20 ("btrfs: enable W=1 checks for btrfs"). $ make M=fs/btrfs scripts/Makefile.lib:10: *** Recursive variable 'KBUILD_CFLAGS' references itself (eventually). Stop. make

Re: All files are damaged after btrfs restore

2021-03-17 Thread Sebastian Roller
Am Mi., 17. März 2021 um 02:54 Uhr schrieb Dāvis Mosāns : > > root@hikitty:~$ install/btrfs-progs-5.9/btrfs check --readonly /dev/sdi1 > > Opening filesystem to check... > > checksum verify failed on 99593231630336 found 00B6 wanted > > checksum verify failed on 124762809384960 found

Re: [PATCH] btrfs: zoned: remove outdated WARN_ON in direct IO

2021-03-17 Thread David Sterba
On Wed, Mar 17, 2021 at 05:57:31PM +0900, Johannes Thumshirn wrote: > In btrfs_submit_direct() there's a WAN_ON_ONCE() that will trigger if > we're submitting a DIO write on a zoned filesystem but are not using > REQ_OP_ZONE_APPEND to submit the IO to the block device. > > This is a left over from

Re: [PATCH 1/3] btrfs: init devices always

2021-03-17 Thread David Sterba
On Fri, Mar 12, 2021 at 01:57:32PM +0800, Anand Jain wrote: > > > On 12/3/21 1:52 pm, Anand Jain wrote: > > On 12/3/21 12:23 am, Josef Bacik wrote: > >> Neal reported a panic trying to use -o rescue=all > >> > >> BUG: kernel NULL pointer dereference, address: 0030 > >> PGD 0 P4D 0 > >

Re: [PATCH v8 04/10] btrfs: fix check_data_csum() error message for direct I/O

2021-03-17 Thread Qu Wenruo
On 2021/3/17 上午3:43, Omar Sandoval wrote: From: Omar Sandoval Commit 1dae796aabf6 ("btrfs: inode: sink parameter start and len to check_data_csum()") replaced the start parameter to check_data_csum() with page_offset(), but page_offset() is not meaningful for direct I/O pages. Bring back the

Fwd: btrfs-progs: libbtrfsutil is under LGPL-3.0 and statically liked into btrfs

2021-03-17 Thread Adam Borowski
This is https://bugs.debian.org/985400 - Forwarded message from Claudius Heine - Dear Maintainer, I looked into the licenses of the btrfs-progs project and found that the libbtrfsutils library is licensed under LGPL-3.0-or-later [1] The `copyright` file does not show this this. IANAL,

Re: [PATCH 0/3] Handle bad dev_root properly with rescue=all

2021-03-17 Thread David Sterba
On Thu, Mar 11, 2021 at 11:23:13AM -0500, Josef Bacik wrote: > Hello, > > My recent debugging session with Neal's broken filesystem uncovered a glaring > hole in my rescue=all patches, they don't deal with a NULL dev_root properly. > In testing I only ever tested corrupting the extent tree or the

Re: [PATCH] btrfs: zoned: remove outdated WARN_ON in direct IO

2021-03-17 Thread Johannes Thumshirn
On 17/03/2021 11:54, David Sterba wrote: > On Wed, Mar 17, 2021 at 05:57:31PM +0900, Johannes Thumshirn wrote: >> In btrfs_submit_direct() there's a WAN_ON_ONCE() that will trigger if >> we're submitting a DIO write on a zoned filesystem but are not using >> REQ_OP_ZONE_APPEND to submit the IO to t

Re: [PATCH] btrfs: zoned: remove outdated WARN_ON in direct IO

2021-03-17 Thread Johannes Thumshirn
On 17/03/2021 14:20, Johannes Thumshirn wrote: > On 17/03/2021 11:54, David Sterba wrote: >> On Wed, Mar 17, 2021 at 05:57:31PM +0900, Johannes Thumshirn wrote: >>> In btrfs_submit_direct() there's a WAN_ON_ONCE() that will trigger if >>> we're submitting a DIO write on a zoned filesystem but are n

Re: Fwd: btrfs-progs: libbtrfsutil is under LGPL-3.0 and statically liked into btrfs

2021-03-17 Thread David Sterba
On Wed, Mar 17, 2021 at 12:53:00PM +0100, Adam Borowski wrote: > This is https://bugs.debian.org/985400 > > - Forwarded message from Claudius Heine - > > Dear Maintainer, > > I looked into the licenses of the btrfs-progs project and found that the > libbtrfsutils library is licensed und

Re: [PATCH] btrfs: zoned: remove outdated WARN_ON in direct IO

2021-03-17 Thread David Sterba
On Wed, Mar 17, 2021 at 01:22:11PM +, Johannes Thumshirn wrote: > On 17/03/2021 14:20, Johannes Thumshirn wrote: > > On 17/03/2021 11:54, David Sterba wrote: > >> On Wed, Mar 17, 2021 at 05:57:31PM +0900, Johannes Thumshirn wrote: > >>> In btrfs_submit_direct() there's a WAN_ON_ONCE() that will

Re: [PATCH] btrfs: zoned: remove outdated WARN_ON in direct IO

2021-03-17 Thread Johannes Thumshirn
On 17/03/2021 15:29, David Sterba wrote: > On Wed, Mar 17, 2021 at 01:22:11PM +, Johannes Thumshirn wrote: >> On 17/03/2021 14:20, Johannes Thumshirn wrote: >>> On 17/03/2021 11:54, David Sterba wrote: On Wed, Mar 17, 2021 at 05:57:31PM +0900, Johannes Thumshirn wrote: > In btrfs_submi

Re: [PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread David Sterba
On Wed, Mar 17, 2021 at 07:38:11PM +0900, Johannes Thumshirn wrote: > When a file gets deleted on a zoned file system, the space freed is no > returned back into the block group's free space, but is migrated to > zone_unusable. > > As this zone_unusable space is behind the current write pointer it

Re: [PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread Johannes Thumshirn
On 17/03/2021 15:33, David Sterba wrote: > On Wed, Mar 17, 2021 at 07:38:11PM +0900, Johannes Thumshirn wrote: >> When a file gets deleted on a zoned file system, the space freed is no >> returned back into the block group's free space, but is migrated to >> zone_unusable. >> >> As this zone_unusab

Re: [PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread Johannes Thumshirn
On 17/03/2021 16:26, Filipe Manana wrote: >> +void btrfs_reclaim_bgs(struct btrfs_fs_info *fs_info) >> +{ >> + struct btrfs_block_group *bg; >> + struct btrfs_space_info *space_info; >> + int ret = 0; >> + >> + if (!test_bit(BTRFS_FS_OPEN, &fs_info->flags)) >> +

Re: [PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread Filipe Manana
On Wed, Mar 17, 2021 at 3:31 PM Johannes Thumshirn wrote: > > On 17/03/2021 16:26, Filipe Manana wrote: > >> +void btrfs_reclaim_bgs(struct btrfs_fs_info *fs_info) > >> +{ > >> + struct btrfs_block_group *bg; > >> + struct btrfs_space_info *space_info; > >> + int ret = 0; > >> +

Re: [PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread Filipe Manana
On Wed, Mar 17, 2021 at 10:40 AM Johannes Thumshirn wrote: > > When a file gets deleted on a zoned file system, the space freed is no > returned back into the block group's free space, but is migrated to > zone_unusable. > > As this zone_unusable space is behind the current write pointer it is not

Re: [PATCH 0/3] Handle bad dev_root properly with rescue=all

2021-03-17 Thread Josef Bacik
On 3/17/21 8:27 AM, David Sterba wrote: On Thu, Mar 11, 2021 at 11:23:13AM -0500, Josef Bacik wrote: Hello, My recent debugging session with Neal's broken filesystem uncovered a glaring hole in my rescue=all patches, they don't deal with a NULL dev_root properly. In testing I only ever tested c

Re: Linux RAID with btrfs stuck and consume 100 % CPU

2021-03-17 Thread Vojtech Myslivec
Thanks a lot Manuel for your findings and information. It's good to know btrfs is not causing this issue and the common symptom is an MD journal on another RAID device. I have moved journal from logical volume on RAID1 to a plain partition on a SSD and I will monitor the state. Vojtech O

Re: [PATCH] btrfs: zoned: automatically reclaim zones

2021-03-17 Thread Josef Bacik
On 3/17/21 6:38 AM, Johannes Thumshirn wrote: When a file gets deleted on a zoned file system, the space freed is no returned back into the block group's free space, but is migrated to zone_unusable. As this zone_unusable space is behind the current write pointer it is not possible to use it for

[PATCH v2] btrfs: Use immediate assignment when referencing cc-option

2021-03-17 Thread Victor Erminpour
Calling cc-option will use KBUILD_CFLAGS, which when lazy setting subdir-ccflags-y produces the following build error: scripts/Makefile.lib:10: *** Recursive variable `KBUILD_CFLAGS' \ references itself (eventually). Stop. Use single := assignment for subdir-ccflags-y. The cc-option call

Re: [PATCH v2] btrfs: Use immediate assignment when referencing cc-option

2021-03-17 Thread David Sterba
On Wed, Mar 17, 2021 at 09:08:48AM -0700, Victor Erminpour wrote: > Calling cc-option will use KBUILD_CFLAGS, which when lazy setting > subdir-ccflags-y produces the following build error: > > scripts/Makefile.lib:10: *** Recursive variable `KBUILD_CFLAGS' \ > references itself (eventually).

Re: [PATCH v8 01/10] iov_iter: add copy_struct_from_iter()

2021-03-17 Thread Christian Brauner
On Tue, Mar 16, 2021 at 12:42:57PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > This is essentially copy_struct_from_user() but for an iov_iter. > > Suggested-by: Aleksa Sarai > Reviewed-by: Josef Bacik > Signed-off-by: Omar Sandoval > --- > include/linux/uio.h | 2 ++ > lib/iov_it

Re: [PATCH] btrfs: fix subvolume/snapshot deletion not triggered on mount

2021-03-17 Thread David Sterba
On Tue, Mar 16, 2021 at 04:53:46PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > During the mount procedure we are calling btrfs_orphan_cleanup() against > the root tree, which will find all orphans items in this tree. When an > orphan item corresponds to a deleted subvolume/snapsho

Re: [PATCH v8 04/10] btrfs: fix check_data_csum() error message for direct I/O

2021-03-17 Thread Omar Sandoval
On Wed, Mar 17, 2021 at 07:21:46PM +0800, Qu Wenruo wrote: > > > On 2021/3/17 上午3:43, Omar Sandoval wrote: > > From: Omar Sandoval > > > > Commit 1dae796aabf6 ("btrfs: inode: sink parameter start and len to > > check_data_csum()") replaced the start parameter to check_data_csum() > > with page_

Re: [PATCH v2] btrfs-progs: common: make sure that qgroup id is in range

2021-03-17 Thread David Sterba
On Tue, Mar 16, 2021 at 01:27:46PM +, Sidong Yang wrote: > When user assign qgroup with qgroup id that is too big to exceeds > range and invade level value, and it works without any error. but > this action would be make undefined error. this code make sure that > qgroup id doesn't exceed range

Re: [PATCH v8 01/10] iov_iter: add copy_struct_from_iter()

2021-03-17 Thread Omar Sandoval
On Wed, Mar 17, 2021 at 06:56:11PM +0100, Christian Brauner wrote: > On Tue, Mar 16, 2021 at 12:42:57PM -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > This is essentially copy_struct_from_user() but for an iov_iter. > > > > Suggested-by: Aleksa Sarai > > Reviewed-by: Josef Bacik >

[PATCH 0/1] btrfs-progs: libbtrfsutil: Relicense to LGPLv2.1+

2021-03-17 Thread Neal Gompa
This is a patch requesting all substantial copyright owners to sign off on changing the license of the libbtrfsutil code to LGPLv2.1+ in order to resolve various concerns around the mixture of code in btrfs-progs with libbtrfsutil. Each significant (i.e. non-trivial) commit author has been CC'd to

[PATCH 1/1] btrfs-progs: libbtrfsutil: Relicense to LGPLv2.1+

2021-03-17 Thread Neal Gompa
This resolves the licensing conflict with GPLv2 code and makes it suitable for environments that have a policy against GNU v3 licenses. Signed-off-by: Neal Gompa --- libbtrfsutil/COPYING | 1130 - libbtrfsutil/COPYING.LESSER | 165 - libbtrfsut

Re: Fwd: btrfs-progs: libbtrfsutil is under LGPL-3.0 and statically liked into btrfs

2021-03-17 Thread Neal Gompa
On Wed, Mar 17, 2021 at 10:19 AM David Sterba wrote: > > On Wed, Mar 17, 2021 at 12:53:00PM +0100, Adam Borowski wrote: > > This is https://bugs.debian.org/985400 > > > > - Forwarded message from Claudius Heine - > > > > Dear Maintainer, > > > > I looked into the licenses of the btrfs-pro

Re: [RFC] btrfs: Allow read-only mount with corrupted extent tree

2021-03-17 Thread Dāvis Mosāns
trešd., 2021. g. 17. marts, plkst. 12:28 — lietotājs Qu Wenruo () rakstīja: > > > > On 2021/3/17 上午9:29, Dāvis Mosāns wrote: > > trešd., 2021. g. 17. marts, plkst. 03:18 — lietotājs Dāvis Mosāns > > () rakstīja: > >> > >> Currently if there's any corruption at all in extent tree > >> (eg. even sing

Re: [PATCH 0/1] btrfs-progs: libbtrfsutil: Relicense to LGPLv2.1+

2021-03-17 Thread Omar Sandoval
On Wed, Mar 17, 2021 at 04:01:43PM -0400, Neal Gompa wrote: > This is a patch requesting all substantial copyright owners to sign off > on changing the license of the libbtrfsutil code to LGPLv2.1+ in order > to resolve various concerns around the mixture of code in btrfs-progs > with libbtrfsutil.

Re: [PATCH v8 04/10] btrfs: fix check_data_csum() error message for direct I/O

2021-03-17 Thread Qu Wenruo
On 2021/3/18 上午2:33, Omar Sandoval wrote: On Wed, Mar 17, 2021 at 07:21:46PM +0800, Qu Wenruo wrote: On 2021/3/17 上午3:43, Omar Sandoval wrote: From: Omar Sandoval Commit 1dae796aabf6 ("btrfs: inode: sink parameter start and len to check_data_csum()") replaced the start parameter to check

Re: [RFC] btrfs: Allow read-only mount with corrupted extent tree

2021-03-17 Thread Qu Wenruo
On 2021/3/18 上午5:03, Dāvis Mosāns wrote: trešd., 2021. g. 17. marts, plkst. 12:28 — lietotājs Qu Wenruo () rakstīja: On 2021/3/17 上午9:29, Dāvis Mosāns wrote: trešd., 2021. g. 17. marts, plkst. 03:18 — lietotājs Dāvis Mosāns () rakstīja: Currently if there's any corruption at all in exte

RE: [PATCH 0/1] btrfs-progs: libbtrfsutil: Relicense to LGPLv2.1+

2021-03-17 Thread misono.tomoh...@fujitsu.com
> Subject: [PATCH 0/1] btrfs-progs: libbtrfsutil: Relicense to LGPLv2.1+ > > This is a patch requesting all substantial copyright owners to sign off > on changing the license of the libbtrfsutil code to LGPLv2.1+ in order > to resolve various concerns around the mixture of code in btrfs-progs > wi

Re: [PATCH 0/1] btrfs-progs: libbtrfsutil: Relicense to LGPLv2.1+

2021-03-17 Thread Qu Wenruo
On 2021/3/18 上午4:01, Neal Gompa wrote: This is a patch requesting all substantial copyright owners to sign off on changing the license of the libbtrfsutil code to LGPLv2.1+ in order to resolve various concerns around the mixture of code in btrfs-progs with libbtrfsutil. Each significant (i.e.

Re: [PATCH v2] btrfs-progs: common: make sure that qgroup id is in range

2021-03-17 Thread Sidong Yang
On Wed, Mar 17, 2021 at 07:36:47PM +0100, David Sterba wrote: > On Tue, Mar 16, 2021 at 01:27:46PM +, Sidong Yang wrote: > > When user assign qgroup with qgroup id that is too big to exceeds > > range and invade level value, and it works without any error. but > > this action would be make unde

Re: [PATCH v2] btrfs-progs: common: make sure that qgroup id is in range

2021-03-17 Thread Qu Wenruo
On 2021/3/18 上午10:22, Sidong Yang wrote: On Wed, Mar 17, 2021 at 07:36:47PM +0100, David Sterba wrote: On Tue, Mar 16, 2021 at 01:27:46PM +, Sidong Yang wrote: When user assign qgroup with qgroup id that is too big to exceeds range and invade level value, and it works without any error.

Re: [btrfs] 5297199a8b: xfstests.btrfs.220.fail

2021-03-17 Thread Oliver Sang
Hi Nikolay, On Tue, Mar 09, 2021 at 10:36:52AM +0200, Nikolay Borisov wrote: > > > On 9.03.21 г. 10:49 ч., kernel test robot wrote: > > > > > > Greeting, > > > > FYI, we noticed the following commit (built with gcc-9): > > > > commit: 5297199a8bca12b8b96afcbf2341605efb6005de ("btrfs: remove