[PATCH v6 00/15] Btrfs-progs offline scrub

For any one who wants to try it, it can be get from my repo: https://github.com/gujx2017/btrfs-progs/tree/offline_scrub In this v6, fix for make error reported by Marco Lorenzo Crociani. Several reports on kernel scrub screwing up good data stripes are in ML for sometime. And since kernel scrub

[PATCH v6 08/15] btrfs-progs: scrub: Introduce function to scrub one data stripe

From: Qu Wenruo Introduce new function, scrub_one_data_stripe(), to check all data and tree blocks inside the data stripe. This function will not try to recovery any error, but only check if any data/tree blocks has mismatch csum. If data missing csum, which is

[PATCH v6 14/15] btrfs-progs: scrub: Introduce function to check a whole block group

From: Qu Wenruo Introduce new function, scrub_one_block_group(), to scrub a block group. For Single/DUP/RAID0/RAID1/RAID10, we use old mirror number based map_block, and check extent by extent. For parity based profile (RAID5/6), we use new map_block_v2() and check

[PATCH v6 15/15] btrfs-progs: scrub: Introduce offline scrub function

From: Qu Wenruo Now, btrfs-progs has a kernel scrub equivalent. A new option, --offline is added to "btrfs scrub start". If --offline is given, btrfs scrub will just act like kernel scrub, to check every copy of extent and do a report on corrupted data and if it's

[PATCH v6 11/15] btrfs-progs: scrub: Introduce function to recover data parity

From: Qu Wenruo Introduce function, recover_from_parities(), to recover data stripes. It just wraps raid56_recov() with extra check functions to scrub_full_stripe structure. Signed-off-by: Qu Wenruo --- scrub.c | 51

[PATCH v6 06/15] btrfs-progs: scrub: Introduce functions to scrub mirror based data blocks

From: Qu Wenruo Introduce new function, check/recover_data_mirror(), to check and recover mirror based data blocks. Unlike tree block, data blocks must be recovered sector by sector, so we introduced corrupted_bitmap for check and recover. Signed-off-by: Qu Wenruo

[PATCH v6 12/15] btrfs-progs: scrub: Introduce helper to write a full stripe

From: Qu Wenruo Introduce a internal helper, write_full_stripe() to calculate P/Q and write the whole full stripe. This is useful to recover RAID56 stripes. Signed-off-by: Qu Wenruo --- scrub.c | 44

[PATCH v6 13/15] btrfs-progs: scrub: Introduce a function to scrub one full stripe

From: Qu Wenruo Introduce a new function, scrub_one_full_stripe(), to check a full stripe. It handles the full stripe scrub in the following steps: 0) Check if we need to check full stripe If full stripe contains no extent, why waste our CPU and IO? 1) Read out full

[PATCH v6 07/15] btrfs-progs: scrub: Introduce function to scrub one mirror-based extent

From: Qu Wenruo Introduce a new function, scrub_one_extent(), as a wrapper to check one mirror-based extent. It will accept a btrfs_path parameter @path, which must point to a META/EXTENT_ITEM. And @start, @len, which must be a subset of META/EXTENT_ITEM.

[PATCH v6 09/15] btrfs-progs: scrub: Introduce function to verify parities

From: Qu Wenruo Introduce new function, verify_parities(), to check whether parities match with full stripe, whose data stripes match with their csum. Caller should fill the scrub_full_stripe structure properly before calling this function. Signed-off-by: Qu Wenruo

[PATCH v6 10/15] btrfs-progs: extent-tree: Introduce function to check if there is any extent in given range.

From: Qu Wenruo Introduce a new function, btrfs_check_extent_exists(), to check if there is any extent in the range specified by user. The parameter can be a large range, and if any extent exists in the range, it will return >0 (in fact it will return 1). Or return 0 if

[PATCH v6 04/15] btrfs-progs: scrub: Introduce structures to support offline scrub for RAID56

From: Qu Wenruo Introuduce new local structures, scrub_full_stripe and scrub_stripe, for incoming offline RAID56 scrub support. For pure stripe/mirror based profiles, like raid0/1/10/dup/single, we will follow the original bytenr and mirror number based iteration, so

[PATCH v6 02/15] btrfs-progs: Allow __btrfs_map_block_v2 to remove unrelated stripes

From: Qu Wenruo For READ, caller normally hopes to get what they request, other than full stripe map. In this case, we should remove unrelated stripe map, just like the following case: 32K 96K |<-request range->| 0

Re: [PATCH 00/15] Btrfs-progs offline scrub

On 20/07/2017 05:39, Qu Wenruo wrote: On 2017年07月20日 00:45, Marco Lorenzo Crociani wrote: On 18/07/2017 08:33, Gu Jinxiang wrote: For any one who wants to try it, it can be get from my repo: https://github.com/gujx2017/btrfs-progs/tree/offline_scrub Hi, I'm getting this error during make:

btrfs cant mount -d single 4 volumes of different sizes hard drives

Hello Sir, my btrfs -d single(4 drives) is failing when one of my hard drive is slowly dying. because of that, i wanted to remove the hard drive thats responsible for it, so i used btrfs device delete /dev/sdf after it mounted. It was successful, so i have 3 drives left and its working. So i

Re: [PATCH 00/15] Btrfs-progs offline scrub

On 2017年07月20日 16:55, Marco Lorenzo Crociani wrote: On 20/07/2017 05:39, Qu Wenruo wrote: On 2017年07月20日 00:45, Marco Lorenzo Crociani wrote: On 18/07/2017 08:33, Gu Jinxiang wrote: For any one who wants to try it, it can be get from my repo:

Re: [PATCH 00/15] Btrfs-progs offline scrub

On 2017年07月20日 17:10, Qu Wenruo wrote: On 2017年07月20日 16:55, Marco Lorenzo Crociani wrote: On 20/07/2017 05:39, Qu Wenruo wrote: On 2017年07月20日 00:45, Marco Lorenzo Crociani wrote: On 18/07/2017 08:33, Gu Jinxiang wrote: For any one who wants to try it, it can be get from my repo:

[PATCH v6 05/15] btrfs-progs: scrub: Introduce functions to scrub mirror based tree block

From: Qu Wenruo Introduce new functions, check/recover_tree_mirror(), to check and recover mirror-based tree blocks (Single/DUP/RAID0/1/10). check_tree_mirror() can also be used on in-memory tree blocks using @data parameter. This is very handy for RAID5/6 case, either

[PATCH v6 01/15] btrfs-progs: Introduce new btrfs_map_block function which returns more unified result.

From: Qu Wenruo Introduce a new function, __btrfs_map_block_v2(). Unlike old btrfs_map_block(), which needs different parameter to handle different RAID profile, this new function uses unified btrfs_map_block structure to handle all RAID profile in a more meaningful

[PATCH v6 03/15] btrfs-progs: csum: Introduce function to read out data csums

From: Qu Wenruo Introduce a new function: btrfs_read_data_csums(), to read out csums for sectors in range. This is quite useful for read out data csum so we don't need to do it using open code. Signed-off-by: Qu Wenruo Signed-off-by: Su Yue

Re: btrfs cant mount -d single 4 volumes of different sizes hard drives

I shoulda mentioned; I did change the sata port back but the same error still occured.. Scanning for Btrfs filesystems ~ >>> sudo btrfs fi show Label: none uuid: 489e95eb-03c5-40e1-bd57-cd6e17584404 Total devices 1 FS bytes used 5.41TiB devid1 size 7.28TiB used 5.74TiB path /dev/sda Label:

btrfs check --clear-space-cache v2- Aborted (core dumped)

Hi, trying to clear the free space tree along with its superblock flag with: btrfs check --clear-space-cache v2 /dev/sda1 # btrfs check --clear-space-cache v2 /dev/sda1 warning, device 2 is missing checksum verify failed on 2933047296 found C638A419 wanted 7BA892F9 checksum verify failed on

Re: [PATCH 2/4] btrfs: separate defrag and property compression

On 07/18/2017 02:46 AM, David Sterba wrote: Add new value for compression to distinguish between defrag and property. Previously, a single variable was used and this caused clashes when the per-file 'compression' was set and a defrag -c was called. How about.. deprecate property

Re: [PATCH 4/4] btrfs: allow defrag compress to override NOCOMPRESS attribute

On 07/18/2017 02:46 AM, David Sterba wrote: Currently, the BTRFS_INODE_NOCOMPRESS will prevent any compression on a given file, except when the mount is force-compress. As users have reported on IRC, this will also prevent compression when requested by defrag (btrfs fi defrag -c file).

Re: [PATCH] btrfs-progs: Use '-t btrfs' mount option in tests.

On 2017年07月13日 05:52, David Sterba wrote: On Wed, Jul 12, 2017 at 01:05:22PM -0700, Adam Buchbinder wrote: Without it, mount (at least from util-linux 2.20.1) tries (and fails) to mount some filesystems as NTFS. Signed-off-by: Adam Buchbinder Applied, thanks.

Re: [PATCH 00/15] Btrfs-progs offline scrub

On 2017年07月20日 17:40, Marco Lorenzo Crociani wrote: On 20/07/2017 11:17, Qu Wenruo wrote: On 2017年07月20日 17:10, Qu Wenruo wrote: On 2017年07月20日 16:55, Marco Lorenzo Crociani wrote: On 20/07/2017 05:39, Qu Wenruo wrote: On 2017年07月20日 00:45, Marco Lorenzo Crociani wrote: On

Re: [PATCH 00/15] Btrfs-progs offline scrub

On 20/07/2017 11:17, Qu Wenruo wrote: On 2017年07月20日 17:10, Qu Wenruo wrote: On 2017年07月20日 16:55, Marco Lorenzo Crociani wrote: On 20/07/2017 05:39, Qu Wenruo wrote: On 2017年07月20日 00:45, Marco Lorenzo Crociani wrote: On 18/07/2017 08:33, Gu Jinxiang wrote: For any one who wants to

Re: strange No space left on device issues

On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote: > Against 4.12 would be best, thanks! okay,.. but that will take a while to compile... in the meantime... do you know whether it's more or less safe to use the 4.9 kernel without any fix, when I change the parameters mentioned before,

Re: strange No space left on device issues

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote: > Could you try 4.12? Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux from Debian experimental, doesn't fix the issue... > If that doesn't work, could you please also try >

Re: strange No space left on device issues

On Thu, Jul 20, 2017 at 07:48:24PM +0200, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote: > > Could you try 4.12? > Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) > x86_64 GNU/Linux > from Debian experimental, doesn't fix the issue...

strange No space left on device issues

Hey. The following happens on Debian stretch systems: # uname -a Linux lcg-lrz-admin 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux What I have are VMs, which run with root fs as ext4 and which I want to migrate to btrfs. So I've added further disk images and then

Re: strange No space left on device issues

Oh and I should add: After such error, cp goes on copying (with other files)... Same issue occurs when I do something like tar -cf - /media | tar -xf Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature

Re: strange No space left on device issues

On Thu, Jul 20, 2017 at 05:20:13PM +0200, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote: > > It would be interesting if lowering the dirty ratio is a viable > > work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl > >

Re: [PATCH] btrfs-progs: Use '-t btrfs' mount option in tests.

On Thu, Jul 20, 2017 at 05:40:00PM +0800, Qu Wenruo wrote: > > > On 2017年07月13日 05:52, David Sterba wrote: > > On Wed, Jul 12, 2017 at 01:05:22PM -0700, Adam Buchbinder wrote: > >> Without it, mount (at least from util-linux 2.20.1) tries (and > >> fails) to mount some filesystems as NTFS. > >>

Re: strange No space left on device issues

On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote: > there are patches on this list/upstream which could fix this ( e.g. > "fix > delalloc accounting leak caused by u32 overflow"/"fix early ENOSPC > due > to delalloc"). mhh... it's a bit problematic to test these on that nodes... > Do you

Re: strange No space left on device issues

On Thu, 2017-07-20 at 15:00 +, Martin Raiber wrote: > It would be interesting if lowering the dirty ratio is a viable > work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl > vm.dirty_bytes=1258291200). > > Regards, > Martin I took away a trailing 0 for each of them... and then

Re: [PATCH] btrfs: test if receive with qgroups corrupts metadata

On Thu, Jul 20, 2017 at 4:44 PM, Qu Wenruo wrote: > On 2017年07月21日 07:03, Justin Maggard wrote: >> >> This test case does some concurrent send/receives with qgroups enabled. >> Currently (4.13-rc1) this usually results in btrfs check errors, and >> often also results in a

Re: [PATCH v3] Btrfs: add skeleton code for compression heuristic

On 07/18/2017 02:30 AM, David Sterba wrote: So it basically looks good, I could not resist and rewrote the changelog and comments. There's one code fix: On Mon, Jul 17, 2017 at 04:52:58PM +0300, Timofey Titovets wrote: -static inline int inode_need_compress(struct inode *inode) +static

Re: kernel btrfs file system wedged -- is it toast?

Thanks for you thoughtful answer, Hans. >> what was the reason for using balance? I had filled up that 8TB btrfs file system a couple of days ago, which made me worry about it, and do what things I thought might ensure it was healthy again (after I removed a bunch of stuff.)I am not using

Re: [PATCH] btrfs: test if receive with qgroups corrupts metadata

On 2017年07月21日 07:03, Justin Maggard wrote: This test case does some concurrent send/receives with qgroups enabled. Currently (4.13-rc1) this usually results in btrfs check errors, and often also results in a WARN_ON in record_root_in_trans(). This is quite interesting. Despite

[PATCH RESEND] Btrfs: fix early ENOSPC due to delalloc

From: Omar Sandoval If a lot of metadata is reserved for outstanding delayed allocations, we rely on shrink_delalloc() to reclaim metadata space in order to fulfill reservation tickets. However, shrink_delalloc() has a shortcut where if it determines that space can be

[PATCH] btrfs: test if receive with qgroups corrupts metadata

This test case does some concurrent send/receives with qgroups enabled. Currently (4.13-rc1) this usually results in btrfs check errors, and often also results in a WARN_ON in record_root_in_trans(). Bisecting points to 6426c7ad697d (btrfs: qgroup: Fix qgroup accounting when creating snapshot) as

kernel btrfs file system wedged -- is it toast?

This may be a repeat, in a slightly variant form, of the problem that Marc MERLIN reports in: https://www.spinics.net/lists/linux-btrfs/msg67393.html 20170716160658.r4uxuqlfe24hg...@merlins.org BTRFS: error (device dm-2) in btrfs_run_delayed_refs:2960: errno=-17 Object already exists

Re: strange No space left on device issues

On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote: > Yes, that's a safe enough workaround. It's a good idea to change the > parameters back after the copy. you mean even without having the fix, right? So AFAIU, the bug doesn't really cause FS corruption, but just "false" ENOSPC and these

Re: kernel btrfs file system wedged -- is it toast?

Hi, On 07/20/2017 08:47 PM, Paul Jackson wrote: > > This may be a repeat, in a slightly variant form, of the problem that > Marc MERLIN reports in: > > https://www.spinics.net/lists/linux-btrfs/msg67393.html > 20170716160658.r4uxuqlfe24hg...@merlins.org > BTRFS: error (device dm-2) in

Re: strange No space left on device issues

On Thu, Jul 20, 2017 at 10:28:15PM +0200, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote: > > Yes, that's a safe enough workaround. It's a good idea to change the > > parameters back after the copy. > you mean even without having the fix, right? Yes, even

[PATCH v3 0/4] Add xxhash and zstd modules

Hi all, This patch set adds xxhash, zstd compression, and zstd decompression modules. It also adds zstd support to BtrFS and SquashFS. Each patch has relevant summaries, benchmarks, and tests. Best, Nick Terrell Changelog: v1 -> v2: - Make pointer in lib/xxhash.c:394 non-const (1/4) - Use

Re: strange No space left on device issues

On Thu, Jul 20, 2017 at 08:06:52PM +0200, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote: > > Against 4.12 would be best, thanks! > okay,.. but that will take a while to compile... > > > in the meantime... do you know whether it's more or less safe to

[PATCH v3 1/4] lib: Add xxhash module

Adds xxhash kernel module with xxh32 and xxh64 hashes. xxhash is an extremely fast non-cryptographic hash algorithm for checksumming. The zstd compression and decompression modules added in the next patch require xxhash. I extracted it out from zstd since it is useful on its own. I copied the code

[PATCH v3 4/4] squashfs: Add zstd support

Add zstd compression and decompression support to SquashFS. zstd is a great fit for SquashFS because it can compress at ratios approaching xz, while decompressing twice as fast as zlib. For SquashFS in particular, it can decompress as fast as lzo and lz4. It also has the flexibility to turn down

[PATCH v3 3/4] btrfs: Add zstd support

Add zstd compression and decompression support to BtrFS. zstd at its fastest level compresses almost as well as zlib, while offering much faster compression and decompression, approaching lzo speeds. I benchmarked btrfs with zstd compression against no compression, lzo compression, and zlib

Re: strange No space left on device issues

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote: > If that doesn't work, could you please also try > https://patchwork.kernel.org/patch/9829593/? Okay, tried the patch now, applied upon: Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux (that is the Debian

Re: strange No space left on device issues

On Thu, Jul 20, 2017 at 11:33:56PM +0200, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote: > > If that doesn't work, could you please also try > > https://patchwork.kernel.org/patch/9829593/? > > Okay, tried the patch now, applied upon: > Linux

Re: strange No space left on device issues

On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote: > Just to be sure, did you explicitly write 0 to these? Nope... that seemed to have been the default value, i.e. I used sysctl(8) in read (and not set) mode here. > These sysctls are > really confusing, see

Re: strange No space left on device issues

On Thu, Jul 20, 2017 at 11:53:16PM +0200, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote: > > Just to be sure, did you explicitly write 0 to these? > Nope... that seemed to have been the default value, i.e. I used > sysctl(8) in read (and not set) mode

Re: strange No space left on device issues

On 07/20/2017 11:53 PM, Christoph Anton Mitterer wrote: > On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote: > [...] > >>> I assume you'll take care to get that patch into stable kernels? >>> Is this patch alone enough to recommend the Debian maintainers to >>> include it into their 4.9 long