Re: Re-mounting removable btrfs on different device

2018-09-06 Thread Remi Gauvin
On 2018-09-06 11:32 PM, Duncan wrote: > Without the mentioned patches, the only way (other than reboot) is to > remove and reinsert the btrfs kernel module (assuming it's a module, not > built-in), thus forcing it to forget state. > > Of course if other critical mounted filesystems (such as

Re: dduper - Offline btrfs deduplication tool

2018-09-06 Thread Lakshmipathi.G
> > One question: > Why not ioctl_fideduperange? > i.e. you kill most of benefits from that ioctl - atomicity. > I plan to add fideduperange as an option too. User can choose between fideduperange and ficlonerange call. If I'm not wrong, with fideduperange, kernel performs comparsion check

Re: Re-mounting removable btrfs on different device

2018-09-06 Thread Duncan
Remi Gauvin posted on Thu, 06 Sep 2018 20:54:17 -0400 as excerpted: > I'm trying to use a BTRFS filesystem on a removable drive. > > The first drive drive was added to the system, it was /dev/sdb > > Files were added and device unmounted without error. > > But when I re-attach the drive, it

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Chris Murphy
On Thu, Sep 6, 2018 at 2:16 PM, Stefan Loewen wrote: > Data,single: Size:695.01GiB, Used:653.69GiB > /dev/sdb1 695.01GiB > Metadata,DUP: Size:4.00GiB, Used:2.25GiB > /dev/sdb1 8.00GiB > System,DUP: Size:40.00MiB, Used:96.00KiB > Does that mean Metadata is duplicated? Yes. Single

Re-mounting removable btrfs on different device

2018-09-06 Thread Remi Gauvin
I'm trying to use a BTRFS filesystem on a removable drive. The first drive drive was added to the system, it was /dev/sdb Files were added and device unmounted without error. But when I re-attach the drive, it becomes /dev/sdg (kernel is fussy about re-using /dev/sdb). btrfs fi show: output:

[PATCH 3/3] btrfs: keep trim from interfering with transaction commits

2018-09-06 Thread jeffm
From: Jeff Mahoney Commit 499f377f49f08 (btrfs: iterate over unused chunk space in FITRIM) fixed free space trimming, but introduced latency when it was running. This is due to it pinning the transaction using both a incremented refcount and holding the commit root sem for the duration of a

[PATCH 1/3] btrfs: use ->devices list instead of ->alloc_list in btrfs_trim_fs

2018-09-06 Thread jeffm
From: Jeff Mahoney btrfs_trim_fs iterates over the fs_devices->alloc_list while holding the device_list_mutex. The problem is that ->alloc_list is protected by the chunk mutex. We don't want to hold the chunk mutex over the trim of the entire file system. Fortunately, the ->dev_list list is

[PATCH 0/3] btrfs: trim latency improvements

2018-09-06 Thread jeffm
From: Jeff Mahoney This patch set fixes a few issues with trim. 1) Fix device list iteration. We're iterating the ->alloc_list while holding the device_list_mutex. The ->alloc_list is protected by the chunk mutex and we don't want to hold it across the entire trim execution.

[PATCH 2/3] btrfs: don't attempt to trim devices that don't support it

2018-09-06 Thread jeffm
From: Jeff Mahoney We check whether any device the file system is using supports discard in the ioctl call, but then we attempt to trim free extents on every device regardless of whether discard is supported. Due to the way we mask off EOPNOTSUPP, we can end up issuing the trim operations on

[PATCH RESEND] btrfs: fix error handling in free_log_tree

2018-09-06 Thread jeffm
From: Jeff Mahoney When we hit an I/O error in free_log_tree->walk_log_tree during file system shutdown we can crash due to there not being a valid transaction handle. Use btrfs_handle_fs_error when there's no transaction handle to use. BUG: unable to handle kernel NULL pointer dereference at

Re: [PATCH] Btrfs: remove redundant btrfs_trans_release_metadata"

2018-09-06 Thread Liu Bo
On Thu, Sep 6, 2018 at 11:50 AM, Nikolay Borisov wrote: > > > On 6.09.2018 09:47, Liu Bo wrote: >> On Wed, Sep 5, 2018 at 10:45 PM, Liu Bo wrote: >>> Somehow this ends up with crash in btrfs/124, I'm trying to figure out >>> what went wrong. >>> >> >> It revealed the problem addressed in

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Stefan Loewen
[root@archlinux @data]# btrfs fi us /mnt/intenso_white/ Overall: Device size: 911.51GiB Device allocated:    703.09GiB Device unallocated:  208.43GiB Device missing:  0.00B Used:    658.19GiB Free (estimated):   

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Chris Murphy
On Thu, Sep 6, 2018 at 12:36 PM, Stefan Loewen wrote: > Output of the commands is attached. fdisk Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes smart Sector Sizes: 512 bytes logical, 4096 bytes physical So clearly the case is lying

[PATCH] btrfs: fix error handling in btrfs_dev_replace_start

2018-09-06 Thread jeffm
From: Jeff Mahoney When we fail to start a transaction in btrfs_dev_replace_start, we leave dev_replace->replace_start set to STARTED but clear ->srcdev and ->tgtdev. Later, that can result in an Oops in btrfs_dev_replace_progress when having state set to STARTED or SUSPENDED implies that

Re: [PATCH] Btrfs: remove redundant btrfs_trans_release_metadata"

2018-09-06 Thread Nikolay Borisov
On 6.09.2018 09:47, Liu Bo wrote: > On Wed, Sep 5, 2018 at 10:45 PM, Liu Bo wrote: >> Somehow this ends up with crash in btrfs/124, I'm trying to figure out >> what went wrong. >> > > It revealed the problem addressed in Josef's patch[1], so with it, > this patch works just fine. What

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Stefan Loewen
Output of the commands is attached. The broken-sector-theory sounds plausible and is compatible with my new findings: I suspected the problem to be in one specific directory, let's call it "broken_dir". I created a new subvolume and copied broken_dir over. - If I copied it with cp --reflink,

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Chris Murphy
On Thu, Sep 6, 2018 at 10:03 AM, Stefan Löwen wrote: > I have one subvolume (rw) and 2 snapshots (ro) of it. > > I just tested 'btrfs send > /dev/null' and that also shows no IO > after a while but also no significant CPU usage. > During this I tried 'ls' on the source subvolume and it hangs as

Re: [PATCH v5 0/6] Btrfs: implement swap file support

2018-09-06 Thread Omar Sandoval
On Thu, Sep 06, 2018 at 01:59:54PM +0200, David Sterba wrote: > On Fri, Aug 31, 2018 at 03:36:35PM -0700, Omar Sandoval wrote: > > This series implements swap file support for Btrfs. > > > > Changes since v4 [1]: > > > > - Added a kernel doc for btrfs_get_chunk_map() > > - Got rid of "Btrfs:

Re: [PATCH v3] Btrfs: set leave_spinning in btrfs_get_extent

2018-09-06 Thread David Sterba
On Sat, Aug 25, 2018 at 01:47:09PM +0800, Liu Bo wrote: > Unless it's going to read inline extents from btree leaf to page, > btrfs_get_extent won't sleep during the period of holding path lock. > > This sets leave_spinning at first and sets path to blocking mode right > before reading inline

Re: [PATCH] Btrfs: fix alignment in declaration and prototype of btrfs_get_extent

2018-09-06 Thread David Sterba
On Sat, Aug 25, 2018 at 01:47:59PM +0800, Liu Bo wrote: > This fixes btrfs_get_extent to be consistent with our existing > declaration style. For the record, indentation styles that are accepted are both, aligning under the opening ( and tab or double tab indentation on the next line. Preferrably

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Stefan Löwen
I have one subvolume (rw) and 2 snapshots (ro) of it. I just tested 'btrfs send > /dev/null' and that also shows no IO after a while but also no significant CPU usage. During this I tried 'ls' on the source subvolume and it hangs as well. dmesg has some interesting messages I think (see

Re: [PATCH] btrfs: extent-tree.c: Remove redundant variable from btrfs_cross_ref_exist()

2018-09-06 Thread David Sterba
On Thu, Aug 30, 2018 at 10:59:16AM +0900, Misono Tomohiro wrote: > Since commit d7df2c796d7e ("Btrfs attach delayed ref updates to > delayed ref heads"), check_delaed_ref() won't return -ENOENT. > > In btrfs_cross_ref_exist(), two variable 'ret' and 'ret2' is > originally used to handle -ENOENT

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Chris Murphy
On Thu, Sep 6, 2018 at 9:04 AM, Stefan Loewen wrote: > Update: > It seems like btrfs-send is not completely hung. It somewhat regularly > wakes up every hour to transfer a few bytes. I noticed this via a > periodic 'ls -l' on the snapshot file. These are the last outputs > (uniq'ed): > >

Re: [PATCH] btrfs: defrag: use btrfs_mod_outstanding_extents in cluster_pages_for_defrag

2018-09-06 Thread David Sterba
On Wed, Sep 05, 2018 at 11:07:33AM +0800, Su Yue wrote: > Since commit 8b62f87bad9c ("Btrfs: rework outstanding_extents"), > manual operations of outstanding_extent in btrfs_inode are replaced by > btrfs_mod_outstanding_extents(). > The one in cluster_pages_for_defrag seems to be lost, so replace

Re: [PATCH v2] Btrfs: remove confusing tracepoint in btrfs_add_reserved_bytes

2018-09-06 Thread David Sterba
On Wed, Sep 05, 2018 at 09:55:27AM +0800, Liu Bo wrote: > Here we're not releasing any space, but transferring bytes from > ->bytes_may_use to ->bytes_reserved. > > Signed-off-by: Liu Bo > --- > v2: Add missing commit log. I've enhanced the changlog a bit how the tracepoint got there.

Re: btrfs send hung in pipe_wait

2018-09-06 Thread Stefan Loewen
Update: It seems like btrfs-send is not completely hung. It somewhat regularly wakes up every hour to transfer a few bytes. I noticed this via a periodic 'ls -l' on the snapshot file. These are the last outputs (uniq'ed): -rw--- 1 root root 1492797759 Sep 6 08:44 intenso_white.snapshot

Re: [PATCH 0/3] btrfs: qgroup: Deprecate unused features for btrfs_qgroup_inherit()

2018-09-06 Thread Qu Wenruo
On 2018/9/5 下午9:00, David Sterba wrote: > On Fri, Aug 31, 2018 at 10:29:27AM +0800, Qu Wenruo wrote: >> This patchset can be fetched from github: >> https://github.com/adam900710/linux/tree/qgroup_inherit_check >> Which is based on v4.19-rc1 tag. >> >> This patchset will first set

Re: [PATCH v5 0/6] Btrfs: implement swap file support

2018-09-06 Thread David Sterba
On Fri, Aug 31, 2018 at 03:36:35PM -0700, Omar Sandoval wrote: > This series implements swap file support for Btrfs. > > Changes since v4 [1]: > > - Added a kernel doc for btrfs_get_chunk_map() > - Got rid of "Btrfs: push EXCL_OP set into btrfs_rm_device()" > - Made activate error messages more

Re: [PATCH] btrfs: qgroup: Don't trace subtree if we're dropping tree reloc tree

2018-09-06 Thread Qu Wenruo
On 2018/9/6 下午7:15, David Sterba wrote: > On Thu, Sep 06, 2018 at 09:41:14AM +0800, Qu Wenruo wrote: >> >> >> On 2018/9/5 下午9:11, David Sterba wrote: >>> On Wed, Sep 05, 2018 at 01:03:39PM +0800, Qu Wenruo wrote: Tree reloc tree doesn't contribute to qgroup numbers, as we have >>> >>> I

Re: [PATCH] btrfs: qgroup: Don't trace subtree if we're dropping tree reloc tree

2018-09-06 Thread David Sterba
On Thu, Sep 06, 2018 at 09:41:14AM +0800, Qu Wenruo wrote: > > > On 2018/9/5 下午9:11, David Sterba wrote: > > On Wed, Sep 05, 2018 at 01:03:39PM +0800, Qu Wenruo wrote: > >> Tree reloc tree doesn't contribute to qgroup numbers, as we have > > > > I think you can call it just 'reloc tree', I'm

Re: Transactional btrfs

2018-09-06 Thread Austin S. Hemmelgarn
On 2018-09-06 03:23, Nathan Dehnel wrote: https://lwn.net/Articles/287289/ In 2008, HP released the source code for a filesystem called advfs so that its features could be incorporated into linux filesystems. Advfs had a feature where a group of file writes were an atomic transaction.

btrfs send hung in pipe_wait

2018-09-06 Thread Stefan Löwen
Hello linux-btrfs, I'm trying to clone a subvolume with 'btrfs send' but it always hangs for hours. I tested this on multiple systems. All showed the same result: - Manjaro (btrfs-progs v4.17.1; linux v4.18.5-1-MANJARO) - Ubuntu 18.04 in VirtualBox (btrfs-progs v4.15.1; linux

Re: [patch] file dedupe (and maybe clone) data corruption (was Re: [PATCH] generic: test for deduplication between different files)

2018-09-06 Thread Dave Chinner
On Fri, Aug 31, 2018 at 01:10:45AM -0400, Zygo Blaxell wrote: > On Thu, Aug 30, 2018 at 04:27:43PM +1000, Dave Chinner wrote: > > On Thu, Aug 23, 2018 at 08:58:49AM -0400, Zygo Blaxell wrote: > > > On Mon, Aug 20, 2018 at 08:33:49AM -0700, Darrick J. Wong wrote: > > > > On Mon, Aug 20, 2018 at

Transactional btrfs

2018-09-06 Thread Nathan Dehnel
https://lwn.net/Articles/287289/ In 2008, HP released the source code for a filesystem called advfs so that its features could be incorporated into linux filesystems. Advfs had a feature where a group of file writes were an atomic transaction.

Re: [PATCH] Btrfs: remove redundant btrfs_trans_release_metadata"

2018-09-06 Thread Liu Bo
On Wed, Sep 5, 2018 at 10:45 PM, Liu Bo wrote: > Somehow this ends up with crash in btrfs/124, I'm trying to figure out > what went wrong. > It revealed the problem addressed in Josef's patch[1], so with it, this patch works just fine. [1] btrfs: make sure we create all new bgs thanks, liubo

Re: [PATCH 22/35] btrfs: make sure we create all new bgs

2018-09-06 Thread Liu Bo
On Fri, Aug 31, 2018 at 7:03 AM, Josef Bacik wrote: > On Fri, Aug 31, 2018 at 10:31:49AM +0300, Nikolay Borisov wrote: >> >> >> On 30.08.2018 20:42, Josef Bacik wrote: >> > We can actually allocate new chunks while we're creating our bg's, so >> > instead of doing list_for_each_safe, just do