Re: performance recommendations

2021-02-16 Thread Piotr Szymaniak
On Mon, Feb 15, 2021 at 10:50:15PM +0100, Pal, Laszlo wrote: > So, You should start here: https://btrfs.wiki.kernel.org/index.php/Btrfs_mailing_list#What_information_to_provide_when_asking_a_support_question Best regards, Piotr Szymaniak. -- - Chyba nie jest pan jednym z tych roniacych lzy lib

Re: performance recommendations

2021-02-16 Thread Pal, Laszlo
Thank you all for the quick response. The server is running, but as I said the i/o perf. is not as good as it should be. I'm also thinking the fragmentation is the issue but I also would like to optimise my config and if possible keep this server running with acceptable performance, so let me answe

Re: performance recommendations

2021-02-16 Thread Nikolay Borisov
On 16.02.21 г. 10:54 ч., Pal, Laszlo wrote: > Thank you all for the quick response. The server is running, but as I > said the i/o perf. is not as good as it should be. I'm also thinking > the fragmentation is the issue but I also would like to optimise my > config and if possible keep this serv

Re: [PATCH v2 2/2] btrfs-progs: device stats: add json output format

2021-02-16 Thread Filipe Manana
On Wed, Nov 11, 2020 at 4:42 PM Sidong Yang wrote: > > Add supports for json formatting, this patch changes hard coded printing > code to formatted print with output formatter. Json output would be > useful for other programs that parse output of the command. but it > changes the text format. > >

Re: performance recommendations

2021-02-16 Thread Lionel Bouton
Hi, Le 16/02/2021 à 09:54, Pal, Laszlo a écrit : > [...] > So, as far as I see the action plan is the following > - enable v2 space_cache. is this safe/stable enough? > - run defrag on old data, I suppose it will run weeks, but I'm ok with > it if the server can run smoothly during this process >

Re: Btrfs progs release 5.10.1

2021-02-16 Thread Filipe Manana
On Fri, Feb 5, 2021 at 11:33 AM David Sterba wrote: > > Hi, > > btrfs-progs version 5.10.1 have been released. > > The static build got broken due to libmount added in 5.10, this works now. The > minimum libmount version is 2.24 that is not available on some LTS distros > like > CentOS 7. The pla

[PATCH] btrfs: fix stale data exposure after cloning a hole with NO_HOLES enabled

2021-02-16 Thread fdmanana
From: Filipe Manana When using the NO_HOLES feature, if we clone a file range that spans only a hole into a range that is at or beyond the current i_size of the destination file, we end up not setting the full sync runtime flag on the inode. As a result, if we then fsync the destination file and

[PATCH] btrfs: add test for cloning a hole post eof when using NO_HOLES feature

2021-02-16 Thread fdmanana
From: Filipe Manana Test that when using the NO_HOLES feature, if we truncate down a file, clone a file range covering only a hole into an offset beyond the current file size, and then fsync the file, after a power failure we get the expected file content and we do not get stale data correspondin

Re: [PATCH v15 00/42] btrfs: zoned block device support

2021-02-16 Thread David Sterba
On Tue, Feb 16, 2021 at 01:33:28PM +0900, Naohiro Aota wrote: > On Mon, Feb 15, 2021 at 04:58:05PM +, Johannes Thumshirn wrote: > > On 11/02/2021 16:48, David Sterba wrote: > > > On Thu, Feb 11, 2021 at 03:26:04PM +, Johannes Thumshirn wrote: > > >> On 11/02/2021 16:21, David Sterba wrote:

Re: Btrfs progs release 5.10

2021-02-16 Thread Filipe Manana
On Tue, Jan 19, 2021 at 5:26 AM David Sterba wrote: > > Hi, > > btrfs-progs version 5.10 have been released. > > Only minor changes snice -rc1: CI on gitlab disabled, some docs added. > > Changelog: > > * scrub status: > * print percentage of progress > * add size unit options > * fi u

[GIT PULL] Btrfs updates for 5.12

2021-02-16 Thread David Sterba
Hi, this update brings updates of space handling, performance improvements or bug fixes. The subpage block size and zoned mode features have reached state where they're usable but with limitations. The branch merges cleanly on top of current master, there are some minor conflicts reported by linu

Re: [PATCH 4/7] fsdax: Replace mmap entry in case of CoW

2021-02-16 Thread David Sterba
On Mon, Feb 08, 2021 at 01:09:21AM +0800, Shiyang Ruan wrote: > We replace the existing entry to the newly allocated one > in case of CoW. Also, we mark the entry as PAGECACHE_TAG_TOWRITE > so writeback marks this entry as writeprotected. This > helps us snapshots so new write pagefaults after snap

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread David Sterba
On Mon, Feb 15, 2021 at 11:05:33AM +, Filipe Manana wrote: > On Sat, Feb 13, 2021 at 1:07 AM Wang Yugui wrote: > > > This bug only affects 5.10 kernels, and the regression was introduced in > > > 5.10-rc1 by commit 0eb79294dbe328 ("btrfs: dio iomap DSYNC workaround"). > > > The bug does not ex

Re: [PATCH] btrfs: ordered-extent: fix comment for btrfs ordered extent flag bits

2021-02-16 Thread David Sterba
On Thu, Feb 11, 2021 at 04:14:05PM +0800, Qu Wenruo wrote: > There is small error in comment about BTRFS_ORDERED_* flags. > > The 4 types are for ordered extent itself, not for direct io. > Only 3 types support direct io, REGULAR/NOCOW/PREALLOC. > > Fix the comment to reflect that. > > Signed-of

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread Filipe Manana
On Tue, Feb 16, 2021 at 2:25 PM David Sterba wrote: > > On Mon, Feb 15, 2021 at 11:05:33AM +, Filipe Manana wrote: > > On Sat, Feb 13, 2021 at 1:07 AM Wang Yugui wrote: > > > > This bug only affects 5.10 kernels, and the regression was introduced in > > > > 5.10-rc1 by commit 0eb79294dbe328 (

Re: [PATCH 1/2] btrfs-progs: convert: Ensure the data chunks size never exceed device size

2021-02-16 Thread Filipe Manana
On Wed, Jun 24, 2020 at 10:00 PM Qu Wenruo wrote: > > [BUG] > The following script could lead to corrupted btrfs fs after > btrfs-convert: > > fallocate -l 1G test.img > mkfs.ext4 test.img > mount test.img $mnt > fallocate -l 200m $mnt/file1 > fallocate -l 200m $mnt/file2 > fallocate -

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread Greg KH
On Tue, Feb 16, 2021 at 02:40:31PM +, fdman...@kernel.org wrote: > From: Filipe Manana > > Whenever we attempt to do a non-aligned direct IO write with O_DSYNC, we > end up triggering an assertion and crashing. Example reproducer: > > $ cat test.sh > #!/bin/bash > > DEV=/dev/sdj > M

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread Filipe Manana
On 16/02/21 14:50, Greg KH wrote: > On Tue, Feb 16, 2021 at 02:40:31PM +, fdman...@kernel.org wrote: >> From: Filipe Manana >> >> Whenever we attempt to do a non-aligned direct IO write with O_DSYNC, we >> end up triggering an assertion and crashing. Example reproducer: >> >> $ cat test.s

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread David Sterba
On Tue, Feb 16, 2021 at 03:50:36PM +0100, Greg KH wrote: > On Tue, Feb 16, 2021 at 02:40:31PM +, fdman...@kernel.org wrote: > > From: Filipe Manana > > > > Whenever we attempt to do a non-aligned direct IO write with O_DSYNC, we > > end up triggering an assertion and crashing. Example reprodu

Re: Recovering Btrfs from a freak failure of the disk controller

2021-02-16 Thread Josef Bacik
On 2/14/21 3:25 PM, Neal Gompa wrote: Hey all, So one of my main computers recently had a disk controller failure that caused my machine to freeze. After rebooting, Btrfs refuses to mount. I tried to do a mount and the following errors show up in the journal: Feb 14 15:20:49 localhost-live ker

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread Greg KH
On Tue, Feb 16, 2021 at 04:15:46PM +0100, David Sterba wrote: > On Tue, Feb 16, 2021 at 03:50:36PM +0100, Greg KH wrote: > > On Tue, Feb 16, 2021 at 02:40:31PM +, fdman...@kernel.org wrote: > > > From: Filipe Manana > > > > > > Whenever we attempt to do a non-aligned direct IO write with O_DS

Re: [PATCH] btrfs: fix stale data exposure after cloning a hole with NO_HOLES enabled

2021-02-16 Thread Josef Bacik
On 2/16/21 6:09 AM, fdman...@kernel.org wrote: From: Filipe Manana When using the NO_HOLES feature, if we clone a file range that spans only a hole into a range that is at or beyond the current i_size of the destination file, we end up not setting the full sync runtime flag on the inode. As a r

Re: Recovering Btrfs from a freak failure of the disk controller

2021-02-16 Thread Neal Gompa
On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik wrote: > > On 2/14/21 3:25 PM, Neal Gompa wrote: > > Hey all, > > > > So one of my main computers recently had a disk controller failure > > that caused my machine to freeze. After rebooting, Btrfs refuses to > > mount. I tried to do a mount and the fol

[RFC] btrfs-progs: format-output: remove newline in fmt_end text mode

2021-02-16 Thread Sidong Yang
Remove a code that inserting new line in fmt_end() for text mode. Old code made a failure in fstest btrfs/006. Signed-off-by: Sidong Yang --- Hi, I've just read mail that Filipe written that some failure about fstest. I'm worried about this patch makes other problem. So make it RFC. Thanks. ---

Re: performance recommendations

2021-02-16 Thread Pal, Laszlo
Hi, Thank you. So, I've installed a new centos7 with the same configuration, old kernel and using btrfs. Then, upgraded the kernel to 5.11 and all went well, so I thought let's do it on the prod server Unfortunately when I boot on 5.11 sysroot mount times out and I have something like this in log

Re: performance recommendations

2021-02-16 Thread Leonidas Spyropoulos
Hi Laszlo, On 16/02/21, Pal, Laszlo wrote: > Hi, > > Thank you. So, I've installed a new centos7 with the same > configuration, old kernel and using btrfs. Then, upgraded the kernel > to 5.11 and all went well, so I thought let's do it on the prod server > Since this is a VM can you clone the di

Re: [PATCH 5.10.x] btrfs: fix crash after non-aligned direct IO write with O_DSYNC

2021-02-16 Thread David Sterba
On Tue, Feb 16, 2021 at 04:34:27PM +0100, Greg KH wrote: > On Tue, Feb 16, 2021 at 04:15:46PM +0100, David Sterba wrote: > > On Tue, Feb 16, 2021 at 03:50:36PM +0100, Greg KH wrote: > > > On Tue, Feb 16, 2021 at 02:40:31PM +, fdman...@kernel.org wrote: > > > As this is a one-off patch, I need t

Re: performance recommendations

2021-02-16 Thread Pal, Laszlo
Hi, Thank you. If I have to clone, I think I'll just get rid of the machine and recreate with some other file system. I'm aware, this is my fault -lack of research and time pressure-, but I think if I can boot it with the old kernel I'll keep it running as long as it can and I'll use this time to

Re: Recovering Btrfs from a freak failure of the disk controller

2021-02-16 Thread Josef Bacik
On 2/16/21 11:27 AM, Neal Gompa wrote: On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik wrote: On 2/14/21 3:25 PM, Neal Gompa wrote: Hey all, So one of my main computers recently had a disk controller failure that caused my machine to freeze. After rebooting, Btrfs refuses to mount. I tried to d

Re: performance recommendations

2021-02-16 Thread Lionel Bouton
Hi, Le 16/02/2021 à 19:01, Pal, Laszlo a écrit : > Hi, > > Thank you. If I have to clone, I think I'll just get rid of the > machine and recreate with some other file system. I'm aware, this is > my fault -lack of research and time pressure-, but I think if I can > boot it with the old kernel I'll

Re: performance recommendations

2021-02-16 Thread Pal, Laszlo
Thanks. Unfortunately, this machine with this file system can boot only with 3.10 kernel somehow. Now it is started, and I hope it can hold the line while I'm creating another machine with a different FS. This is not a criticism towards BTRFS, this is a criticism to myself to not thinking enough be

Re: Recovering Btrfs from a freak failure of the disk controller

2021-02-16 Thread Neal Gompa
On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik wrote: > > On 2/16/21 11:27 AM, Neal Gompa wrote: > > On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik wrote: > >> > >> On 2/14/21 3:25 PM, Neal Gompa wrote: > >>> Hey all, > >>> > >>> So one of my main computers recently had a disk controller failure > >>>

[PATCH] btrfs: do not error out if the extent ref hash doesn't match

2021-02-16 Thread Josef Bacik
The tree checker checks the extent ref hash at read and write time to make sure we do not corrupt the file system. Generally extent references go inline, but if we have enough of them we need to make an item, which looks like key.objectid= key.type= key.offset = hash(tree, owne

[PATCH] fstests: test a regression with btrfs extent reference collisions

2021-02-16 Thread Josef Bacik
This is a regression test for a problem where we would flip read only if we reflink'ed enough extents to generate key'ed references, and then got a hash collision with those references. This is a test for the fix btrfs: do not error out if the extent ref hash doesn't match and is relativ

Re: Recovering Btrfs from a freak failure of the disk controller

2021-02-16 Thread Josef Bacik
On 2/16/21 3:29 PM, Neal Gompa wrote: On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik wrote: On 2/16/21 11:27 AM, Neal Gompa wrote: On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik wrote: On 2/14/21 3:25 PM, Neal Gompa wrote: Hey all, So one of my main computers recently had a disk controller fa

Re: [RFC][PATCH V6] btrfs: allocation_hint mode

2021-02-16 Thread Josef Bacik
On 2/11/21 1:47 PM, Goffredo Baroncelli wrote: On 2/10/21 5:04 PM, Josef Bacik wrote: On 2/1/21 4:28 PM, Goffredo Baroncelli wrote: Hi all, the previous V5 serie was called "btrfs: preferred_metadata: preferred device for metadata". A few general points up front, first I'd highly recommend

Re: [PATCH 1/2] btrfs-progs: convert: Ensure the data chunks size never exceed device size

2021-02-16 Thread Qu Wenruo
On 2021/2/16 下午10:45, Filipe Manana wrote: On Wed, Jun 24, 2020 at 10:00 PM Qu Wenruo wrote: [BUG] The following script could lead to corrupted btrfs fs after btrfs-convert: fallocate -l 1G test.img mkfs.ext4 test.img mount test.img $mnt fallocate -l 200m $mnt/file1 fallocat

Re: Recovering Btrfs from a freak failure of the disk controller

2021-02-16 Thread Neal Gompa
On Tue, Feb 16, 2021 at 4:24 PM Josef Bacik wrote: > > On 2/16/21 3:29 PM, Neal Gompa wrote: > > On Tue, Feb 16, 2021 at 1:11 PM Josef Bacik wrote: > >> > >> On 2/16/21 11:27 AM, Neal Gompa wrote: > >>> On Tue, Feb 16, 2021 at 10:19 AM Josef Bacik wrote: > > On 2/14/21 3:25 PM, Neal Go

Re: [PATCH 4/7] fsdax: Replace mmap entry in case of CoW

2021-02-16 Thread Ruan Shiyang
On 2021/2/16 下午9:11, David Sterba wrote: On Mon, Feb 08, 2021 at 01:09:21AM +0800, Shiyang Ruan wrote: We replace the existing entry to the newly allocated one in case of CoW. Also, we mark the entry as PAGECACHE_TAG_TOWRITE so writeback marks this entry as writeprotected. This helps us snaps

Re: [PATCH 5/7] fsdax: Dedup file range to use a compare function

2021-02-16 Thread Ruan Shiyang
On 2021/2/10 下午9:19, Christoph Hellwig wrote: On Tue, Feb 09, 2021 at 05:46:13PM +0800, Ruan Shiyang wrote: On 2021/2/9 下午5:34, Christoph Hellwig wrote: On Tue, Feb 09, 2021 at 05:15:13PM +0800, Ruan Shiyang wrote: The dax dedupe comparison need the iomap_ops pointer as argument, so my un

[PATCH] btrfs: prevent potential out of bounds in btrfs_ioctl_snap_create_v2()

2021-02-16 Thread Dan Carpenter
The problem is we're copying "inherit" from user space but we don't necessarily know that we're copying enough data for a 64 byte struct. Then the next problem is that "inherit" has a variable size array at the end, and we have to verify that array is the size we expected. Fixes: 6f72c7e20dba: ("

[PATCH] btrfs: zoned: fix possible deadlock on log sync

2021-02-16 Thread Johannes Thumshirn
Lockdep with fstests test-case btrfs/041 detected a unsafe locking scenario when we allocate the log node on a zoned filesystem. btrfs/041 WARNING: possible recursive locking detected 5.11.0-rc7+ #939 Not tainted