Re: [PATCH v14 29/42] btrfs: introduce dedicated data write path for ZONED mode

2021-02-04 Thread Naohiro Aota
On Tue, Feb 02, 2021 at 04:00:45PM +0100, David Sterba wrote: On Tue, Jan 26, 2021 at 11:25:07AM +0900, Naohiro Aota wrote: If more than one IO is issued for one file extent, these IO can be written to separate regions on a device. Since we cannot map one file extent to such a separate area, we

Lieber Freund (Assalamu Alaikum),

2021-02-04 Thread AISHA GADDAFI
-- Lieber Freund (Assalamu Alaikum), Ich bin vor einer privaten Suche auf Ihren E-Mail-Kontakt gestoßen Ihre Hilfe. Mein Name ist Aisha Al-Qaddafi, eine alleinerziehende Mutter und eine Witwe mit drei Kindern. Ich bin die einzige leibliche Tochter des Spätlibyschen Präsident (verstorbener Oberst

Re: [PATCH 2/4] btrfs: fix race between writes to swap files and scrub

2021-02-04 Thread Anand Jain
On 2/3/2021 7:17 PM, fdman...@kernel.org wrote: From: Filipe Manana When we active a swap file, at btrfs_swap_activate(), we acquire the exclusive operation lock to prevent the physical location of the swap file extents to be changed by operations such as balance and device replace/resize/remov

Re: [PATCH 2/4] btrfs: fix race between writes to swap files and scrub

2021-02-04 Thread Filipe Manana
On Thu, Feb 4, 2021 at 8:48 AM Anand Jain wrote: > > On 2/3/2021 7:17 PM, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > When we active a swap file, at btrfs_swap_activate(), we acquire the > > exclusive operation lock to prevent the physical location of the swap > > file extents to b

Re: Space cache

2021-02-04 Thread Filipe Manana
On Wed, Feb 3, 2021 at 10:15 PM Martin Raiber wrote: > > Hi, > > I've been looking a bit into the btrfs space cache and came to following > conclusions. Please correct me if I'm wrong: > > 1. The space cache mount option only modifies how the space cache is > persisted and not the in-memory stru

[PATCH v15 01/42] block: add bio_add_zone_append_page

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn Add bio_add_zone_append_page(), a wrapper around bio_add_hw_page() which is intended to be used by file systems that directly add pages to a bio instead of using bio_iov_iter_get_pages(). Reviewed-by: Christoph Hellwig Reviewed-by: Josef Bacik Reviewed-by: Chaitanya Ku

[PATCH v15 03/42] btrfs: zoned: defer loading zone info after opening trees

2021-02-04 Thread Naohiro Aota
This is a preparation patch to implement zone emulation on a regular device. To emulate a zoned filesystem on a regular (non-zoned) device, we need to decide an emulated zone size. Instead of making it a compile-time static value, we'll make it configurable at mkfs time. Since we have one zone ==

[PATCH v15 04/42] btrfs: zoned: use regular super block location on zone emulation

2021-02-04 Thread Naohiro Aota
A zoned btrfs filesystem currently has a superblock at the beginning of the superblock logging zones if the zones are conventional. This difference in superblock position causes a chicken-and-egg problem for filesystems with emulated zones. Since the device is a regular (non-zoned) device, we canno

[PATCH v15 07/42] btrfs: zoned: disallow fitrim on zoned filesystems

2021-02-04 Thread Naohiro Aota
The implementation of fitrim depends on space cache, which is not used and disabled for zoned extent allocator. So the current code does not work with zoned filesystem. In the future, we can implement fitrim for zoned filesystems by enabling space cache (but, only for fitrim) or scanning the exten

[PATCH v15 06/42] btrfs: zoned: do not load fs_info::zoned from incompat flag

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn Don't set the zoned flag in fs_info as soon as we're encountering the incompat filesystem flag for a zoned filesystem on mount. The zoned flag in fs_info is in a union together with the zone_size, so setting it too early will result in setting an incorrect zone_size as we

[PATCH v15 09/42] btrfs: zoned: implement zoned chunk allocator

2021-02-04 Thread Naohiro Aota
Implement a zoned chunk and device extent allocator. One device zone becomes a device extent so that a zone reset affects only this device extent and does not change the state of blocks in the neighbor device extents. To implement the allocator, we need to extend the following functions for a zone

[PATCH v15 12/42] btrfs: zoned: calculate allocation offset for conventional zones

2021-02-04 Thread Naohiro Aota
Conventional zones do not have a write pointer, so we cannot use it to determine the allocation offset for sequential allocation if a block group contains a conventional zone. But instead, we can consider the end of the highest addressed extent in the block group for the allocation offset. For ne

[PATCH v15 11/42] btrfs: zoned: load zone's allocation offset

2021-02-04 Thread Naohiro Aota
A zoned filesystem must allocate blocks at the zones' write pointer. The device's write pointer position can be mapped to a logical address within a block group. To facilitate this, add an "alloc_offset" to the block-group to track the logical addresses of the write pointer. This logical address i

[PATCH v15 05/42] btrfs: release path before calling to btrfs_load_block_group_zone_info

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn Since we have no write pointer in conventional zones, we cannot determine the allocation offset from it. Instead, we set the allocation offset after the highest addressed extent. This is done by reading the extent tree in btrfs_load_block_group_zone_info(). However, this

[PATCH v15 15/42] btrfs: zoned: redirty released extent buffers

2021-02-04 Thread Naohiro Aota
Tree manipulating operations like merging nodes often release once-allocated tree nodes. Such nodes are cleaned so that pages in the node are not uselessly written out. On zoned volumes, however, such optimization blocks the following IOs as the cancellation of the write out of the freed blocks bre

[PATCH v15 16/42] btrfs: zoned: advance allocation pointer after tree log node

2021-02-04 Thread Naohiro Aota
Since the allocation info of a tree log node is not recorded in the extent tree, calculate_alloc_pointer() cannot detect this node, so the pointer can be over a tree node. Replaying the log calls btrfs_remove_free_space() for each node in the log tree. So, advance the pointer after the node to no

[PATCH v15 10/42] btrfs: zoned: verify device extent is aligned to zone

2021-02-04 Thread Naohiro Aota
Add a check in verify_one_dev_extent() to ensure that a device extent on a zoned block device is aligned to the respective zone boundary. If it isn't mark the filesystem as unclean. Reviewed-by: Anand Jain Reviewed-by: Josef Bacik Signed-off-by: Naohiro Aota Reviewed-by: David Sterba --- fs/

[PATCH v15 14/42] btrfs: zoned: implement sequential extent allocation

2021-02-04 Thread Naohiro Aota
Implement a sequential extent allocator for zoned filesystems. This allocator only needs to check if there is enough space in the block group after the allocation pointer to satisfy the extent allocation request. Therefore the allocator never manages bitmaps or clusters. Also, add assertions to the

[PATCH v15 08/42] btrfs: zoned: allow zoned filesystems on non-zoned block devices

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn Run a zoned filesystem on non-zoned devices. This is done by "slicing up" the block device into static sized chunks and fake a conventional zone on each of them. The emulated zone size is determined from the size of device extent. This is mainly aimed at testing parts of

[PATCH v15 13/42] btrfs: zoned: track unusable bytes for zones

2021-02-04 Thread Naohiro Aota
In a zoned filesystem a once written then freed region is not usable until the underlying zone has been reset. So we need to distinguish such unusable space from usable free space. Therefore we need to introduce the "zone_unusable" field to the block group structure, and "bytes_zone_unusable" to t

[PATCH v15 02/42] iomap: support REQ_OP_ZONE_APPEND

2021-02-04 Thread Naohiro Aota
A ZONE_APPEND bio must follow hardware restrictions (e.g. not exceeding max_zone_append_sectors) not to be split. bio_iov_iter_get_pages builds such restricted bio using __bio_iov_append_get_pages if bio_op(bio) == REQ_OP_ZONE_APPEND. To utilize it, we need to set the bio_op before calling bio_iov

[PATCH v15 00/42] btrfs: zoned block device support

2021-02-04 Thread Naohiro Aota
This series adds zoned block device support to btrfs. Some of the patches in the previous series are already merged as preparation patches. This series and related changes to userland tools are also available on github. Kernel https://github.com/naota/linux/tree/btrfs-zoned-v15 Userland https:/

[PATCH v15 17/42] btrfs: zoned: reset zones of unused block groups

2021-02-04 Thread Naohiro Aota
We must reset the zones of a deleted unused block group to rewind the zones' write pointers to the zones' start. To do this, we can use the DISCARD_SYNC code to do the reset when the filesystem is running on zoned devices. Reviewed-by: Josef Bacik Reviewed-by: Anand Jain Signed-off-by: Naohiro

[PATCH v15 19/42] btrfs: zoned: use bio_add_zone_append_page

2021-02-04 Thread Naohiro Aota
A zoned device has its own hardware restrictions e.g. max_zone_append_size when using REQ_OP_ZONE_APPEND. To follow these restrictions, use bio_add_zone_append_page() instead of bio_add_page(). We need target device to use bio_add_zone_append_page(), so this commit reads the chunk information to ca

[PATCH v15 23/42] btrfs: extend btrfs_rmap_block for specifying a device

2021-02-04 Thread Naohiro Aota
btrfs_rmap_block currently reverse-maps the physical addresses on all devices to the corresponding logical addresses. Extend the function to match to a specified device. The old functionality of querying all devices is left intact by specifying NULL as target device. A block_device instead of a b

[PATCH v15 21/42] btrfs: zoned: split ordered extent when bio is sent

2021-02-04 Thread Naohiro Aota
For a zone append write, the device decides the location the data is being written to. Therefore we cannot ensure that two bios are written consecutively on the device. In order to ensure that an ordered extent maps to a contiguous region on disk, we need to maintain a "one bio == one ordered exten

[PATCH v15 24/42] btrfs: zoned: cache if block-group is on a sequential zone

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn On a zoned filesystem, cache if a block-group is on a sequential write only zone. On sequential write only zones, we can use REQ_OP_ZONE_APPEND for writing of data, therefore provide btrfs_use_zone_append() to figure out if I/O is targeting a sequential write only zone a

[PATCH v15 18/42] btrfs: factor out helper adding a page to bio

2021-02-04 Thread Naohiro Aota
Extract adding a page to a bio from submit_extent_page(). The page is added only when bio_flags are the same, contiguous and the added page fits in the same stripe as pages in the bio. Condition checks are reordered to allow early return to avoid possibly heavy btrfs_bio_fits_in_stripe() calling.

[PATCH v15 22/42] btrfs: zoned: check if bio spans across an ordered extent

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn To ensure that an ordered extent maps to a contiguous region on disk, we need to maintain a "one bio == one ordered extent" rule. Ensure that constructing bio does not span more than an ordered extent. Reviewed-by: Josef Bacik Signed-off-by: Johannes Thumshirn Signed-

[PATCH v15 25/42] btrfs: save irq flags when looking up an ordered extent

2021-02-04 Thread Naohiro Aota
From: Johannes Thumshirn A following patch will add another caller of btrfs_lookup_ordered_extent(), but from a bio's endio context. btrfs_lookup_ordered_extent() uses spin_lock_irq() which unconditionally disables interrupts. Change this to spin_lock_irqsave() so interrupts aren't disabled and

[PATCH v15 20/42] btrfs: zoned: handle REQ_OP_ZONE_APPEND as writing

2021-02-04 Thread Naohiro Aota
Zoned filesystems use REQ_OP_ZONE_APPEND bios for writing to actual devices. Let btrfs_end_bio() and btrfs_op be aware of it, by mapping REQ_OP_ZONE_APPEND to BTRFS_MAP_WRITE and using btrfs_op() instead of bio_op(). Reviewed-by: Josef Bacik Signed-off-by: Naohiro Aota --- fs/btrfs/disk-io.c |

[PATCH v15 26/42] btrfs: zoned: use ZONE_APPEND write for zoned btrfs

2021-02-04 Thread Naohiro Aota
This commit enables zone append writing for zoned btrfs. When using zone append, a bio is issued to the start of a target zone and the device decides to place it inside the zone. Upon completion the device reports the actual written position back to the host. Three parts are necessary to enable zo

[PATCH v15 27/42] btrfs: zoned: enable zone append writing for direct IO

2021-02-04 Thread Naohiro Aota
Likewise to buffered IO, enable zone append writing for direct IO when its used on a zoned block device. Reviewed-by: Josef Bacik Signed-off-by: Naohiro Aota --- fs/btrfs/inode.c | 18 ++ 1 file changed, 18 insertions(+) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index 6d

[PATCH v15 28/42] btrfs: zoned: introduce dedicated data write path for zoned filesystems

2021-02-04 Thread Naohiro Aota
If more than one IO is issued for one file extent, these IO can be written to separate regions on a device. Since we cannot map one file extent to such a separate area on a zoned filesystem, we need to follow the "one IO == one ordered extent" rule. The normal buffered, uncompressed and not pre-al

[PATCH v15 30/42] btrfs: zoned: wait for existing extents before truncating

2021-02-04 Thread Naohiro Aota
When truncating a file, file buffers which have already been allocated but not yet written may be truncated. Truncating these buffers could cause breakage of a sequential write pattern in a block group if the truncated blocks are for example followed by blocks allocated to another file. To avoid th

[PATCH v15 29/42] btrfs: zoned: serialize metadata IO

2021-02-04 Thread Naohiro Aota
We cannot use zone append for writing metadata, because the B-tree nodes have references to each other using logical address. Without knowing the address in advance, we cannot construct the tree in the first place. So we need to serialize write IOs for metadata. We cannot add a mutex around alloca

[PATCH v15 31/42] btrfs: zoned: do not use async metadata checksum on zoned filesystems

2021-02-04 Thread Naohiro Aota
On zoned filesystems, btrfs uses per-FS zoned_meta_io_lock to serialize the metadata write IOs. Even with this serialization, write bios sent from btree_write_cache_pages can be reordered by async checksum workers as these workers are per CPU and not per zone. To preserve write BIO ordering, we d

[PATCH v15 32/42] btrfs: zoned: mark block groups to copy for device-replace

2021-02-04 Thread Naohiro Aota
This is the 1/4 patch to support device-replace on zoned filesystems. We have two types of I/Os during the device-replace process. One is an I/O to "copy" (by the scrub functions) all the device extents from the source device to the destination device. The other one is an I/O to "clone" (by handle

[PATCH v15 35/42] btrfs: zoned: support dev-replace in zoned filesystems

2021-02-04 Thread Naohiro Aota
This is 4/4 patch to implement device-replace on zoned filesystems. Even after the copying is done, the write pointers of the source device and the destination device may not be synchronized. For example, when the last allocated extent is freed before device-replace process, the extent is not copi

[PATCH v15 33/42] btrfs: zoned: implement cloning for zoned device-replace

2021-02-04 Thread Naohiro Aota
This is 2/4 patch to implement device-replace for zoned filesystems. In zoned mode, a block group must be either copied (from the source device to the destination device) or cloned (to the both device). This commit implements the cloning part. If a block group targeted by an IO is marked to copy,

[PATCH v15 34/42] btrfs: zoned: implement copying for zoned device-replace

2021-02-04 Thread Naohiro Aota
This is 3/4 patch to implement device-replace on zoned filesystems. This commit implements copying. To do this, it tracks the write pointer during the device replace process. As device-replace's copy process is smart enough to only copy used extents on the source device, we have to fill the gap to

[PATCH v15 36/42] btrfs: zoned: enable relocation on a zoned filesystem

2021-02-04 Thread Naohiro Aota
Currently fallocate() is disabled on a zoned filesystem. Since current relocation process relies on preallocation to move file data extents, it must be handled differently. On a zoned filesystem, we just truncate the inode to the size that we wanted to pre-allocate. Then, we flush dirty pages on t

[PATCH v15 37/42] btrfs: zoned: relocate block group to repair IO failure in zoned filesystems

2021-02-04 Thread Naohiro Aota
When btrfs finds a checksum error and if the file system has a mirror of the damaged data, btrfs read the correct data from the mirror and writes it to damaged blocks. This however, violates the sequential write constraints of a zoned block device. We can consider three methods to repair an IO fai

[PATCH v15 38/42] btrfs: split alloc_log_tree()

2021-02-04 Thread Naohiro Aota
This is a preparation patch for the next patch. Split alloc_log_tree() into two parts. The first one allocating the tree structure, remains in alloc_log_tree() and the second part allocating the tree node, which is moved into btrfs_alloc_log_tree_node(). Also export the latter part is to be used i

[PATCH v15 39/42] btrfs: zoned: extend zoned allocator to use dedicated tree-log block group

2021-02-04 Thread Naohiro Aota
This is the 1/3 patch to enable tree log on zoned filesystems. The tree-log feature does not work on a zoned filesystem as is. Blocks for a tree-log tree are allocated mixed with other metadata blocks and btrfs writes and syncs the tree-log blocks to devices at the time of fsync(), which has a dif

[PATCH v15 41/42] btrfs: zoned: reorder log node allocation on zoned filesystem

2021-02-04 Thread Naohiro Aota
This is the 3/3 patch to enable tree-log on zoned filesystems. The allocation order of nodes of "fs_info->log_root_tree" and nodes of "root->log_root" is not the same as the writing order of them. So, the writing causes unaligned write errors. Reorder the allocation of them by delaying allocation

[PATCH v15 40/42] btrfs: zoned: serialize log transaction on zoned filesystems

2021-02-04 Thread Naohiro Aota
This is the 2/3 patch to enable tree-log on zoned filesystems. Since we can start more than one log transactions per subvolume simultaneously, nodes from multiple transactions can be allocated interleaved. Such mixed allocation results in non-sequential writes at the time of a log transaction comm

[PATCH v15 42/42] btrfs: zoned: enable to mount ZONED incompat flag

2021-02-04 Thread Naohiro Aota
This final patch adds the ZONED incompat flag to BTRFS_FEATURE_INCOMPAT_SUPP and enables btrfs to mount ZONED flagged file system. Reviewed-by: Anand Jain Signed-off-by: Naohiro Aota Reviewed-by: Josef Bacik --- fs/btrfs/ctree.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --g

btrfs becomes read only on removal of folders

2021-02-04 Thread mig...@rozsas.eng.br
Hi there, I am using opensuse tumbleweed on a desktop. More than 1 year ago, I setup a rotating disk to use btrfs and things went nice until yesterday. I was done with Civ VI on steam, and decided to remove the associated files. I've started to remove files and I got the message the file system is

Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

2021-02-04 Thread Andy Smith
Hi Cedric, On Wed, Feb 03, 2021 at 08:33:18PM +0100, wrote: > it's called "dm-integrity", as mentioned in this e-mail: > https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg93037.html If you do this it would be very interesting to see performance figures for the following setups: - btr

Re: [PATCH 6/6] btrfs: do not block inode logging for so long during transaction commit

2021-02-04 Thread Wang Yugui
Hi, > On 4.02.21 г. 5:17 ч., Wang Yugui wrote: > > Hi, > > > > I tried to run btrfs misc-next(5.11-rc6 +81patches) based on linux LTS > > 5.10.12 with the same other kernel components and the same kernel config. > > > > Better dbench(sync open) result on both Throughput and max_latency. > > >

Re: [PATCH v15 40/42] btrfs: zoned: serialize log transaction on zoned filesystems

2021-02-04 Thread Filipe Manana
On Thu, Feb 4, 2021 at 10:23 AM Naohiro Aota wrote: > > This is the 2/3 patch to enable tree-log on zoned filesystems. > > Since we can start more than one log transactions per subvolume > simultaneously, nodes from multiple transactions can be allocated > interleaved. Such mixed allocation result

Re: [PATCH v15 41/42] btrfs: zoned: reorder log node allocation on zoned filesystem

2021-02-04 Thread Filipe Manana
On Thu, Feb 4, 2021 at 10:23 AM Naohiro Aota wrote: > > This is the 3/3 patch to enable tree-log on zoned filesystems. > > The allocation order of nodes of "fs_info->log_root_tree" and nodes of > "root->log_root" is not the same as the writing order of them. So, the > writing causes unaligned writ

Re: [PATCH 6/6] btrfs: do not block inode logging for so long during transaction commit

2021-02-04 Thread Nikolay Borisov
On 4.02.21 г. 13:34 ч., Wang Yugui wrote: > Hi, > >> On 4.02.21 г. 5:17 ч., Wang Yugui wrote: >>> Hi, >>> >>> I tried to run btrfs misc-next(5.11-rc6 +81patches) based on linux LTS >>> 5.10.12 with the same other kernel components and the same kernel config. >>> >>> Better dbench(sync open) res

Re: [PATCH 0/2] btrfs: add proper subpage compress read support

2021-02-04 Thread Anand Jain
Nice. Works well. Thanks, Anand On 2/4/2021 3:03 PM, Qu Wenruo wrote: During the long time subpage development, I forgot to properly check compression code after just one compression read success during early development. It turns out that, with current RO support, the compression read nee

Re: [PATCH v4 1/3] btrfs: add read_policy latency

2021-02-04 Thread Anand Jain
Hi Michal, Did you get any chance to run the evaluation with this patchset? Thanks, Anand On 1/30/2021 9:08 AM, Anand Jain wrote: 500m is really small data size for such measurement I reran the read policy tests with some changes in the fio command options. Mainly to measure IOPS throug

[PATCH] btrfs: fix extent buffer leak on failure to copy root

2021-02-04 Thread fdmanana
From: Filipe Manana At btrfs_copy_root(), if the call to btrfs_inc_ref() fails we end up returning without unlocking and releasing our reference on the extent buffer named "cow" we previously allocated with btrfs_alloc_tree_block(). So fix that by unlocking the extent buffer and dropping our ref

Re: [PATCH v15 41/42] btrfs: zoned: reorder log node allocation on zoned filesystem

2021-02-04 Thread Johannes Thumshirn
On 04/02/2021 12:57, Filipe Manana wrote: > On Thu, Feb 4, 2021 at 10:23 AM Naohiro Aota wrote: >> >> This is the 3/3 patch to enable tree-log on zoned filesystems. >> >> The allocation order of nodes of "fs_info->log_root_tree" and nodes of >> "root->log_root" is not the same as the writing order

Re: [PATCH v15 41/42] btrfs: zoned: reorder log node allocation on zoned filesystem

2021-02-04 Thread Johannes Thumshirn
On 04/02/2021 16:50, David Sterba wrote: > On Thu, Feb 04, 2021 at 02:54:25PM +, Johannes Thumshirn wrote: >> On 04/02/2021 12:57, Filipe Manana wrote: >>> On Thu, Feb 4, 2021 at 10:23 AM Naohiro Aota wrote: --- a/fs/btrfs/tree-log.c +++ b/fs/btrfs/tree-log.c @@ -3159,6 +3159,19

Re: [PATCH v15 41/42] btrfs: zoned: reorder log node allocation on zoned filesystem

2021-02-04 Thread David Sterba
On Thu, Feb 04, 2021 at 02:54:25PM +, Johannes Thumshirn wrote: > On 04/02/2021 12:57, Filipe Manana wrote: > > On Thu, Feb 4, 2021 at 10:23 AM Naohiro Aota wrote: > >> --- a/fs/btrfs/tree-log.c > >> +++ b/fs/btrfs/tree-log.c > >> @@ -3159,6 +3159,19 @@ int btrfs_sync_log(struct btrfs_trans_ha

Re: [PATCH] btrfs: fix extent buffer leak on failure to copy root

2021-02-04 Thread David Sterba
On Thu, Feb 04, 2021 at 02:35:44PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > At btrfs_copy_root(), if the call to btrfs_inc_ref() fails we end up > returning without unlocking and releasing our reference on the extent > buffer named "cow" we previously allocated with btrfs_alloc

Re: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

2021-02-04 Thread
--- Ursprüngliche Nachricht --- Von: Andy Smith Datum: 04.02.2021 11:54:57 An: linux-btrfs@vger.kernel.org Betreff: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called? Hi Cedric, On Wed, Feb 03, 2021 at 08:33:18PM +0100, wrote: > it's called "dm-int

Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

2021-02-04 Thread Goffredo Baroncelli
[...] Hey Andy, I would rather see performance figures for these setups: A) btrfs with 2 (or more) hard drives and one SSD in writeback bcache configuration (unsafe against failure of the ssd): +-+ | btrfs raid 1 /mnt | +--+--+ | /de

Re: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called?

2021-02-04 Thread
--- Ursprüngliche Nachricht --- Von: Goffredo Baroncelli Datum: 04.02.2021 19:13:50 An: cedric.dew...@eclipso.eu, Andy Smith Betreff: Re: put 2 hard drives in mdadm raid 1 and detect bitrot like btrfs does, what's that called? [...] > Hey Andy, > > I would rather see performance figures for

[PATCH 0/2] btrfs-corrupt-block btree data corruption

2021-02-04 Thread Boris Burkov
Add some more generic corruption to btrfs-corrupt-block which allows corrupting the data in metadata items. Motivated by testing fsverity which requires rather specific corruption of the metadata. The first patch adds corrupting arbitrary regions of item data with -I. The second patch adds corrup

[PATCH 1/2] btrfs-progs: corrupt generic item data with btrfs-corrupt-block

2021-02-04 Thread Boris Burkov
btrfs-corrupt-block already has a mix of generic and specific corruption options, but currently lacks the capacity for totally arbitrary corruption in item data. There is already a flag for corruption size (bytes/-b), so add a flag for an offset and a value to memset the item with. Exercise the ne

[PATCH 2/2] btrfs-progs: expand corrupt_file_extent in btrfs-corrupt-block

2021-02-04 Thread Boris Burkov
To corrupt holes/prealloc/inline extents, we need to mess with extent data items. This patch makes it possible to modify disk_bytenr with a specific value (useful for hole corruptions) and to modify the type field (useful for prealloc corruptions) Signed-off-by: Boris Burkov --- btrfs-corrupt-bl

Re: btrfs becomes read only on removal of folders

2021-02-04 Thread Chris Murphy
On Thu, Feb 4, 2021 at 4:04 AM mig...@rozsas.eng.br wrote: > https://susepaste.org/51166386 It's raid1 metadata on the same physical device, so depending on the device, if the metadata writes are concurrent they may end up being deduped by the drive firmware no matter that they're supposed to g

[PATCH 0/5] btrfs: support fsverity

2021-02-04 Thread Boris Burkov
This patchset provides support for fsverity in btrfs. At a high level, we store the verity descriptor and Merkle tree data in the file system btree with the file's inode as the objectid, and direct reads/writes to those items to implement the generic fsverity interface required by fs/verity/. The

[PATCH 4/5] btrfs: fallback to buffered io for verity files

2021-02-04 Thread Boris Burkov
Reading the contents with direct IO would circumvent verity checks, so fallback to buffered reads. For what it's worth, this is how ext4 handles it as well. Signed-off-by: Boris Burkov --- fs/btrfs/file.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c in

[PATCH 1/5] btrfs: add compat_flags to btrfs_inode_item

2021-02-04 Thread Boris Burkov
The tree checker currently rejects unrecognized flags when it reads btrfs_inode_item. Practically, this means that adding a new flag makes the change backwards incompatible if the flag is ever set on a file. Take up one of the 4 reserved u64 fields in the btrfs_inode_item as a new "compat_flags".

[PATCH 3/5] btrfs: check verity for reads of inline extents and holes

2021-02-04 Thread Boris Burkov
The majority of reads receive a verity check after the bio is complete as the page is marked uptodate. However, there is a class of reads which are handled with btrfs logic in readpage, rather than by submitting a bio. Specifically, these are inline extents, preallocated extents, and holes. Tweak r

[PATCH 2/5] btrfs: initial fsverity support

2021-02-04 Thread Boris Burkov
From: Chris Mason Add support for fsverity in btrfs. To support the generic interface in fs/verity, we add two new item types in the fs tree for inodes with verity enabled. One stores the per-file verity descriptor and the other stores the Merkle tree data itself. Verity checking is done at the

[PATCH 5/5] btrfs: add sysfs feature for fsverity

2021-02-04 Thread Boris Burkov
Now that we support fsverity, enable a feature flag for it in sysfs. Signed-off-by: Boris Burkov --- fs/btrfs/sysfs.c | 6 ++ include/uapi/linux/btrfs.h | 1 + 2 files changed, 7 insertions(+) diff --git a/fs/btrfs/sysfs.c b/fs/btrfs/sysfs.c index 19b9fffa2c9c..40e780724c03 100644

[PATCH] btrfs: add a test for btrfs fsverity

2021-02-04 Thread Boris Burkov
There are some btrfs specific fsverity scenarios that don't map neatly onto the tests in generic/574, like holes, inline extents, and preallocated extents. Cover those in a btrfs specific test. That test relies on assumptions about how the Merkle tree is stored by ext4/f2fs which don't apply to bt

Re: [PATCH 5/5] btrfs: add allocator_hint mode

2021-02-04 Thread Zygo Blaxell
On Mon, Feb 01, 2021 at 10:28:20PM +0100, Goffredo Baroncelli wrote: > From: Goffredo Baroncelli > > When this mode is enabled, the chunk allocation policy is modified as follow. > > Each disk may have a different tag: > - BTRFS_DEV_ALLOCATION_PREFERRED_METADATA > - BTRFS_DEV_ALLOCATION_METADATA

a test

2021-02-04 Thread Le Cai
forgive me to send this test email. My previous email could not go through.

Re: btrfs becomes read only on removal of folders

2021-02-04 Thread Qu Wenruo
On 2021/2/4 下午7:02, mig...@rozsas.eng.br wrote: Hi there, I am using opensuse tumbleweed on a desktop. More than 1 year ago, I setup a rotating disk to use btrfs and things went nice until yesterday. I was done with Civ VI on steam, and decided to remove the associated files. I've started to

Re: [PATCH 6/6] btrfs: do not block inode logging for so long during transaction commit

2021-02-04 Thread Wang Yugui
Hi, > On 4.02.21 г. 13:34 ч., Wang Yugui wrote: > > Hi, > > > >> On 4.02.21 г. 5:17 ч., Wang Yugui wrote: > >>> Hi, > >>> > >>> I tried to run btrfs misc-next(5.11-rc6 +81patches) based on linux LTS > >>> 5.10.12 with the same other kernel components and the same kernel config. > >>> > >>> Better

Re: [PATCH 2/5] btrfs: initial fsverity support

2021-02-04 Thread kernel test robot
e' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Boris-Burkov/btrfs-support-fsverity/20210205-072745 base:1048ba83fb1c00cd24172e23e8263972f6b5d9ac config: x86_64-randconfig-a006-20210204 (attached as .config) compiler: cla

Re: [PATCH 2/5] btrfs: initial fsverity support

2021-02-04 Thread kernel test robot
e' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Boris-Burkov/btrfs-support-fsverity/20210205-072745 base:1048ba83fb1c00cd24172e23e8263972f6b5d9ac config: arc-randconfig-r004-20210204 (attached as .config) compiler: arc-el

Re: [PATCH 2/5] btrfs: initial fsverity support

2021-02-04 Thread kernel test robot
'--base' as documented in https://git-scm.com/docs/git-format-patch] url: https://github.com/0day-ci/linux/commits/Boris-Burkov/btrfs-support-fsverity/20210205-072745 base:1048ba83fb1c00cd24172e23e8263972f6b5d9ac config: x86_64-randconfig-a002-20210204 (attached as .config) compi

Re: [PATCH] btrfs: add a test for btrfs fsverity

2021-02-04 Thread Eric Biggers
Thanks for writing a test for this! On Thu, Feb 04, 2021 at 03:24:26PM -0800, Boris Burkov wrote: > There are some btrfs specific fsverity scenarios that don't map > neatly onto the tests in generic/574, like holes, inline extents, > and preallocated extents. Cover those in a btrfs specific test.

Re: [PATCH 0/5] btrfs: support fsverity

2021-02-04 Thread Eric Biggers
On Thu, Feb 04, 2021 at 03:21:36PM -0800, Boris Burkov wrote: > This patchset provides support for fsverity in btrfs. Very interested to see this! It generally looks good, but I have some comments. Also, when you send this out next, can you include linux-fscr...@vger.kernel.org, as per 'get_main

Re: [PATCH] btrfs: add a test for btrfs fsverity

2021-02-04 Thread Boris Burkov
Thu, Feb 04, 2021 at 10:04:07PM -0800, Eric Biggers wrote: > Thanks for writing a test for this! > > On Thu, Feb 04, 2021 at 03:24:26PM -0800, Boris Burkov wrote: > > There are some btrfs specific fsverity scenarios that don't map > > neatly onto the tests in generic/574, like holes, inline extent

Re: [PATCH 2/5] btrfs: initial fsverity support

2021-02-04 Thread Eric Biggers
On Thu, Feb 04, 2021 at 03:21:38PM -0800, Boris Burkov wrote: > +/* > + * drop all the items for this inode with this key_type. Before > + * doing a verity enable we cleanup any existing verity items. > + * > + * This is also used to clean up if a verity enable failed half way > + * through > + */

Re: [PATCH 0/5] btrfs: support fsverity

2021-02-04 Thread Boris Burkov
On Thu, Feb 04, 2021 at 10:13:54PM -0800, Eric Biggers wrote: > On Thu, Feb 04, 2021 at 03:21:36PM -0800, Boris Burkov wrote: > > This patchset provides support for fsverity in btrfs. > > Very interested to see this! It generally looks good, but I have some > comments. > > Also, when you send t

Re: [PATCH v15 40/42] btrfs: zoned: serialize log transaction on zoned filesystems

2021-02-04 Thread Naohiro Aota
On Thu, Feb 04, 2021 at 11:50:45AM +, Filipe Manana wrote: > On Thu, Feb 4, 2021 at 10:23 AM Naohiro Aota wrote: > > > > This is the 2/3 patch to enable tree-log on zoned filesystems. > > > > Since we can start more than one log transactions per subvolume > > simultaneously, nodes from multipl

Re: [PATCH 2/4] btrfs: fix race between writes to swap files and scrub

2021-02-04 Thread Anand Jain
On 2/4/2021 6:11 PM, Filipe Manana wrote: On Thu, Feb 4, 2021 at 8:48 AM Anand Jain wrote: On 2/3/2021 7:17 PM, fdman...@kernel.org wrote: From: Filipe Manana When we active a swap file, at btrfs_swap_activate(), we acquire the exclusive operation lock to prevent the physical location of th