Re: slow btrfs with a single kworker process using 100% CPU

i've backported the free space cache tree to my kerne and hopefully any fixes related to it. The first mount with clear_cache,space_cache=v2 took around 5 hours. Currently i do not see any kworker with 100CPU but i don't see much load at all. btrfs-transaction tooks around 2-4% CPU together

Re: Raid0 rescue

OK this time, also -mraid1 -draid0, and filled it with some more metadata this time, but I then formatted NTFS, then ext4, then xfs. And then wiped those signatures. Brutal, especially ext4 which writes a lot more stuff and zeros a bunch of areas too. # btrfs rescue super -v /dev/mapper/vg-2

Re: Raid0 rescue

I'm testing explicitly for this case: # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert 1 vg Vwi-a-tz-- 10.00g thintastic0.00 2 vg Vwi-a-tz-- 10.00g thintastic0.00 thintastic vg twi-aotz-- 100.00g

Re: btrfs fi du -s gives Inappropriate ioctl for device

On Wed, Aug 16, 2017 at 3:27 AM, Piotr Szymaniak wrote: > On Mon, Aug 14, 2017 at 05:40:30PM -0600, Chris Murphy wrote: >> On Mon, Aug 14, 2017 at 4:57 PM, Piotr Szymaniak wrote: >> >> > >> > and... some issues: >> > ~ # btrfs fi du -s /mnt/red/\@backup/

Re: qcow2 images make scrub believe the filesystem is corrupted.

>>> On Tue, Aug 15, 2017 at 7:12 PM, Paulo Dias wrote: Device Model: Samsung SSD 850 EVO M.2 500GB Serial Number:S33DNX0H812686V LU WWN Device Id: 5 002538 d4130d027 Firmware Version: EMT21B6Q >>> Unfortunately no firmware updates listed with Samsung for this

Re: slow btrfs with a single kworker process using 100% CPU

>>> I've one system where a single kworker process is using 100% >>> CPU sometimes a second process comes up with 100% CPU >>> [btrfs-transacti]. [ ... ] >> [ ... ]1413 Snapshots. I'm deleting 50 of them every night. But >> btrfs-cleaner process isn't running / consuming CPU currently. Reminder

Re: BTRFS error (device sda4): failed to read chunk tree: -5

On Wed, Aug 16, 2017 at 4:25 PM, Zirconium Hacker wrote: > Hi, > This is my first time using a mailing list, and I hope I'm doing this right. > > $ uname -a > Linux thinkpad 4.12.6-1-ARCH #1 SMP PREEMPT Sat Aug 12 09:16:22 CEST > 2017 x86_64 GNU/Linux > $ btrfs --version >

BTRFS error (device sda4): failed to read chunk tree: -5

Hi, This is my first time using a mailing list, and I hope I'm doing this right. $ uname -a Linux thinkpad 4.12.6-1-ARCH #1 SMP PREEMPT Sat Aug 12 09:16:22 CEST 2017 x86_64 GNU/Linux $ btrfs --version btrfs-progs v4.12 $ sudo mount -o ro,recovery /dev/sda4 /mnt mount: /mnt: wrong fs type, bad

Re: [PATCH] btrfs-progs: fix cross-compile build

On 8/15/17 7:17 PM, Qu Wenruo wrote: > > > On 2017年08月16日 02:11, Eric Sandeen wrote: >> The mktables binary needs to be build with the host >> compiler at built time, not the target compiler, because >> it runs at build time to generate the raid tables. >> >> Copy auto-fu from xfsprogs and

[PATCH 4/4 v4] btrfs: add compression trace points

From: Anand Jain This patch adds compression and decompression trace points for the purpose of debugging. Signed-off-by: Anand Jain Reviewed-by: Nikolay Borisov --- v4: Accepts David's review comments . changes from unsigned

Re: CrashMonkey: A Framework to Systematically Test File-System Crash Consistency

Amir, That's a fair response. I certainly did not mean to add more work on your end :) Using dm-log-writes for now is a reasonable approach. Like I mentioned before, I think there is further work involved in getting CrashMonkey to a useful point (where it finds at least known bugs). Once this is

Re: CrashMonkey: A Framework to Systematically Test File-System Crash Consistency

On Wed, Aug 16, 2017 at 10:06 PM, Vijay Chidambaram wrote: > Hi Josef, > > Thank you for the detailed reply -- I think it provides several > pointers for our future work. It sounds like we have a similar vision > for where we want this to go, though we may disagree about how

Re: [PATCH 2/4] btrfs: convert enum btrfs_compression_type to define

On 08/16/2017 09:59 PM, David Sterba wrote: On Sun, Aug 13, 2017 at 12:02:42PM +0800, Anand Jain wrote: There isn't a huge list to manage the types, which can be managed with defines. It helps to easily print the types in tracing as well. We use enums in a lot of places, I'd rather keep it

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On Wed, Aug 16, 2017 at 8:01 AM, Qu Wenruo wrote: > BTW, when Fujitsu tested the postgresql workload on btrfs, the result is > quite interesting. > > For HDD, when number of clients is low, btrfs shows obvious performance > drop. > And the problem seems to be mandatory

Re: CrashMonkey: A Framework to Systematically Test File-System Crash Consistency

Hi Josef, Thank you for the detailed reply -- I think it provides several pointers for our future work. It sounds like we have a similar vision for where we want this to go, though we may disagree about how to implement this :) This is exciting! I agree that we should be building off existing

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On Wed, Aug 16, 2017 at 09:53:57AM -0400, Austin S. Hemmelgarn wrote: > > So apart from some central DBs for the storage management system > > itself, CoW is mostly no issue for us. > > But I've talked to some friend at the local super computing centre and > > they have rather general issues with

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On Thu, Aug 03, 2017 at 08:08:59PM +0200, waxhead wrote: > BTRFS biggest problem is not that there are some bits and pieces that > are thoroughly screwed up (raid5/6 (which just got some fixes by the > way)), but the fact that the documentation is rather dated. > > There is a simple status

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

[ ... ] >>> Snapshots work fine with nodatacow, each block gets CoW'ed >>> once when it's first written to, and then goes back to being >>> NOCOW. >>> The only caveat is that you probably want to defrag either >>> once everything has been rewritten, or right after the >>> snapshot. >> I thought

Re: [PATCH v2 0/7] add sanity check for extent inline ref type

On Wed, Aug 16, 2017 at 04:53:15PM +0200, David Sterba wrote: > On Mon, Aug 07, 2017 at 03:55:24PM -0600, Liu Bo wrote: > > An invalid extent inline ref type could be read from a btrfs image and > > it ends up with a panic[1], this set is to deal with the insane value > > gracefully in patch 1-2

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

[ ... ] > But I've talked to some friend at the local super computing > centre and they have rather general issues with CoW at their > virtualisation cluster. Amazing news! :-) > Like SUSE's snapper making many snapshots leading the storage > images of VMs apparently to explode (in terms of

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

> We use the crcs to catch storage gone wrong, [ ... ] And that's an opportunistically feasible idea given that current CPUs can do that in real-time. > [ ... ] It's possible to protect against all three without COW, > but all solutions have their own tradeoffs and this is the setup > we chose.

Re: [PATCH] btrfs: copy fsid to super_block s_uuid

On Tue, Aug 01, 2017 at 06:35:08PM +0800, Anand Jain wrote: > We didn't copy fsid to struct super_block.s_uuid so Overlay disables > index feature with btrfs as the lower FS. > > kernel: overlayfs: fs on '/lower' does not support file handles, falling back > to index=off. > > Fix this by

[PATCH] btrfs: Remove unused sectorsize variable from struct map_lookup

This variable was added in 1abe9b8a138c ("Btrfs: add initial tracepointi support for btrfs"), yet it never really got used, only assigned to. So let's remove it. Signed-off-by: Nikolay Borisov --- fs/btrfs/volumes.c | 2 -- fs/btrfs/volumes.h | 1 - 2 files changed, 3

[PATCH] btrfs: expose internal free space tree routine only if sanity tests are enabled

The internal free space tree management routines are always exposed for testing purposes. Make them dependent on SANITY_TESTS being on so that they are exposed only when they really have to. Signed-off-by: Nikolay Borisov --- fs/btrfs/free-space-tree.h | 3 ++- 1 file

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On 2017-08-16 10:11, Christoph Anton Mitterer wrote: On Wed, 2017-08-16 at 09:53 -0400, Austin S. Hemmelgarn wrote: Go try BTRFS on top of dm-integrity, or on a system with T10-DIF or T13-EPP support When dm-integrity is used... would that be enough for btrfs to do a proper repair in the

Re: [PATCH v2 0/7] add sanity check for extent inline ref type

On Mon, Aug 07, 2017 at 03:55:24PM -0600, Liu Bo wrote: > An invalid extent inline ref type could be read from a btrfs image and > it ends up with a panic[1], this set is to deal with the insane value > gracefully in patch 1-2 and clean up BUG() in the code in patch 3-6. > > Patch 7 adds one more

Re: [PATCH 4/4 v3] btrfs: add compression trace points

On Sun, Aug 13, 2017 at 12:02:44PM +0800, Anand Jain wrote: > From: Anand Jain > > This patch adds compression and decompression trace points for the > purpose of debugging. > > Signed-off-by: Anand Jain > Reviewed-by: Nikolay Borisov

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On Wed, 2017-08-16 at 09:53 -0400, Austin S. Hemmelgarn wrote: > Go try BTRFS on top of dm-integrity, or on a  > system with T10-DIF or T13-EPP support When dm-integrity is used... would that be enough for btrfs to do a proper repair in the RAID+nodatacow case? I assume it can't do repairs now

Re: [PATCH v2] btrfs: use appropriate define for the fsid

On Sat, Jul 29, 2017 at 05:50:09PM +0800, Anand Jain wrote: > Though BTRFS_FSID_SIZE and BTRFS_UUID_SIZE or of same size, > for the purpose of doing it correctly use BTRFS_FSID_SIZE instead. > > Signed-off-by: Anand Jain Reviewed-by: David Sterba -- To

Re: [PATCH 3/4] btrfs: decode compress type for tracing

On Sun, Aug 13, 2017 at 12:02:43PM +0800, Anand Jain wrote: > So with this now we see the compression type in string. > > Signed-off-by: Anand Jain Reviewed-by: David Sterba -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On 2017年08月16日 21:12, Chris Mason wrote: On Mon, Aug 14, 2017 at 09:54:48PM +0200, Christoph Anton Mitterer wrote: On Mon, 2017-08-14 at 11:53 -0400, Austin S. Hemmelgarn wrote: Quite a few applications actually _do_ have some degree of secondary verification or protection from a crash. Go

Re: [PATCH 2/4] btrfs: convert enum btrfs_compression_type to define

On Sun, Aug 13, 2017 at 12:02:42PM +0800, Anand Jain wrote: > There isn't a huge list to manage the types, which can be managed > with defines. It helps to easily print the types in tracing as well. We use enums in a lot of places, I'd rather keep it as it is. -- To unsubscribe from this list:

Re: [PATCH 1/4] btrfs: remove unused BTRFS_COMPRESS_LAST

On Sun, Aug 13, 2017 at 12:02:41PM +0800, Anand Jain wrote: > We aren't using this define, so removing it. > > Signed-off-by: Anand Jain Reviewed-by: David Sterba -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On 2017-08-16 09:12, Chris Mason wrote: My real goal is to make COW fast enough that we can leave it on for the database applications too. Obviously I haven't quite finished that one yet ;) But I'd rather keep the building block of all the other btrfs features in place than try to do crcs

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On 2017-08-16 09:31, Christoph Anton Mitterer wrote: Just out of curiosity: On Wed, 2017-08-16 at 09:12 -0400, Chris Mason wrote: Btrfs couples the crcs with COW because this (which sounds like you want it to stay coupled that way)... plus It's possible to protect against all three

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

Just out of curiosity: On Wed, 2017-08-16 at 09:12 -0400, Chris Mason wrote: > Btrfs couples the crcs with COW because this (which sounds like you want it to stay coupled that way)... plus > It's possible to protect against all three without COW, but all  > solutions have their own tradeoffs

Re: RedHat 7.4 Release Notes: "Btrfs has been deprecated" - wut?

On Mon, Aug 14, 2017 at 09:54:48PM +0200, Christoph Anton Mitterer wrote: On Mon, 2017-08-14 at 11:53 -0400, Austin S. Hemmelgarn wrote: Quite a few applications actually _do_ have some degree of secondary  verification or protection from a crash.  Go look at almost any database  software.

Re: [PATCH] btrfs: Fix -EOVERFLOW handling in btrfs_ioctl_tree_search_v2

On Fri, Aug 04, 2017 at 02:41:18PM +0300, Nikolay Borisov wrote: > The buffer passed to btrfs_ioctl_tree_search* functions have to be at least > sizeof(struct btrfs_ioctl_search_header). If this is not the case then the > ioctl should return -EOVERFLOW and set the uarg->buf_size to the minimum >

Re: CrashMonkey: A Framework to Systematically Test File-System Crash Consistency

On Tue, Aug 15, 2017 at 08:44:16PM -0500, Vijay Chidambaram wrote: > Hi Amir, > > I neglected to mention this earlier: CrashMonkey does not require > recompiling the kernel (it is a stand-alone kernel module), and has > been tested with the kernel 4.4. It should work with future kernel > versions

Re: slow btrfs with a single kworker process using 100% CPU

Am 16.08.2017 um 14:29 schrieb Konstantin V. Gavrilenko: > Roman, initially I had a single process occupying 100% CPU, when sysrq it was > indicating as "btrfs_find_space_for_alloc" > but that's when I used the autodefrag, compress, forcecompress and commit=10 > mount flags and space_cache was

Re: slow btrfs with a single kworker process using 100% CPU

Am 16.08.2017 um 14:29 schrieb Konstantin V. Gavrilenko: > Roman, initially I had a single process occupying 100% CPU, when sysrq it was > indicating as "btrfs_find_space_for_alloc" > but that's when I used the autodefrag, compress, forcecompress and commit=10 > mount flags and space_cache was

Re: slow btrfs with a single kworker process using 100% CPU

Am 16.08.2017 um 14:29 schrieb Konstantin V. Gavrilenko: > Roman, initially I had a single process occupying 100% CPU, when sysrq it was > indicating as "btrfs_find_space_for_alloc" > but that's when I used the autodefrag, compress, forcecompress and commit=10 > mount flags and space_cache was

Re: slow btrfs with a single kworker process using 100% CPU

Roman, initially I had a single process occupying 100% CPU, when sysrq it was indicating as "btrfs_find_space_for_alloc" but that's when I used the autodefrag, compress, forcecompress and commit=10 mount flags and space_cache was v1 by default. when I switched to

Re: slow btrfs with a single kworker process using 100% CPU

On Wed, 16 Aug 2017 12:48:42 +0100 (BST) "Konstantin V. Gavrilenko" wrote: > I believe the chunk size of 512kb is even worth for performance then the > default settings on my HW RAID of 256kb. It might be, but that does not explain the original problem reported at

Re: slow btrfs with a single kworker process using 100% CPU

I believe the chunk size of 512kb is even worth for performance then the default settings on my HW RAID of 256kb. Peter Grandi explained it earlier on in one of his posts. QTE ++ That runs counter to this simple story: suppose a program is doing 64KiB IO: * For *reads*, there are 4 data

Re: btrfs fi du -s gives Inappropriate ioctl for device

On Mon, Aug 14, 2017 at 05:40:30PM -0600, Chris Murphy wrote: > On Mon, Aug 14, 2017 at 4:57 PM, Piotr Szymaniak wrote: > > > > > and... some issues: > > ~ # btrfs fi du -s /mnt/red/\@backup/ > > Total Exclusive Set shared Filename > > ERROR: cannot check space of

Re: slow btrfs with a single kworker process using 100% CPU

Am 16.08.2017 um 11:02 schrieb Konstantin V. Gavrilenko: > Could be similar issue as what I had recently, with the RAID5 and 256kb chunk > size. > please provide more information about your RAID setup. Hope this helps: # cat /proc/mdstat Personalities : [raid1] [raid6] [raid5] [raid4] [linear]

Re: slow btrfs with a single kworker process using 100% CPU

Could be similar issue as what I had recently, with the RAID5 and 256kb chunk size. please provide more information about your RAID setup. p.s. you can also check the tread "Btrfs + compression = slow performance and high cpu usage" - Original Message - From: "Stefan Priebe -

Re: slow btrfs with a single kworker process using 100% CPU

Am 16.08.2017 um 08:53 schrieb Marat Khalili: >> I've one system where a single kworker process is using 100% CPU >> sometimes a second process comes up with 100% CPU [btrfs-transacti]. Is >> there anything i can do to get the old speed again or find the culprit? > > 1. Do you use quotas

Re: qcow2 images make scrub believe the filesystem is corrupted.

BTW, to determine it's really data corruption, you could check the data checksum by executing "btrfs check --check-data-csum". --check-data-csum has its limitation of skipping remaining mirrors if the first mirror is correct, but since your data is single, such limitation is not a problem at

Re: slow btrfs with a single kworker process using 100% CPU

I've one system where a single kworker process is using 100% CPU sometimes a second process comes up with 100% CPU [btrfs-transacti]. Is there anything i can do to get the old speed again or find the culprit? 1. Do you use quotas (qgroups)? 2. Do you have a lot of snapshots? Have you deleted

slow btrfs with a single kworker process using 100% CPU

Hello, I've one system where a single kworker process is using 100% CPU sometimes a second process comes up with 100% CPU [btrfs-transacti]. Is there anything i can do to get the old speed again or find the culprit? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe