Re: Creating btrfs RAID on LUKS devs makes devices disappear

journalctl -b -o short-monotonic > journal.log And then attached the log, hopefully it's small enough to be accepted by the list server (should be). If that's not revealing it might be necessary to reboot with rd.udev.debug but start with the simple case first and see if that reveals what's going

Re: errno=-28 No space left, with kernel backtrace (blocking bug)

alpha_one_x86 posted on Thu, 11 May 2017 17:25:32 +0200 as excerpted: > Up plz, I can work with this bug. > > > On 05/11/17 01:39, alpha_one_x86 wrote: >> Hi, this bug is very blocking for me: >> >> https://bugzilla.kernel.org/show_bug.cgi?id=195257 >> >> The server is backup server, I btrfs

[RFC PATCH v3.1 2/6] btrfs: qgroup: Cleanup btrfs_qgroup_prepare_account_extents function

Quite a lot of qgroup corruption happens due to wrong timing of calling btrfs_qgroup_prepare_account_extents(). Since the safest timing is calling it just before btrfs_qgroup_account_extents(), there is no need to separate these 2 function. Merging them will make code cleaner and less bug prone.

[RFC PATCH v3.1 4/6] btrfs: qgroup: Fix qgroup reserved space underflow caused by buffered write and quota enable

[BUG] Under the following case, we can underflow qgroup reserved space. Task A|Task B --- Quota disabled | Buffered write | |- btrfs_check_data_free_space() |

[RFC PATCH v3.1 5/6] btrfs: qgroup: Introduce extent changeset for qgroup reserve functions

Introduce a new parameter, struct extent_changeset for btrfs_qgroup_reserved_data() and its callers. Such extent_changeset was used in btrfs_qgroup_reserve_data() to record which range it reserved in current reserve, so it can free it at error path. The reason we need to export it to callers is,

[RFC PATCH v3.1 3/6] btrfs: qgroup: Return actually freed bytes for qgroup release or free data

btrfs_qgroup_release/free_data() only returns 0 or minus error number(ENOMEM is the only possible error). This is normally good enough, but sometimes we need the accurate byte number it freed/released. Change it to return actually released/freed bytenr number instead of 0 for success. And

[RFC PATCH v3.1 6/6] btrfs: qgroup: Fix qgroup reserved space underflow by only freeing reserved ranges

[BUG] For the following case, btrfs can underflow qgroup reserved space at error path: (Page size 4K, function name without "btrfs_" prefix) Task A | Task B -- Buffered_write [0, 2K)

[RFC PATCH v3.1 1/6] btrfs: qgroup: Add quick exit for non-fs extents

For btrfs_qgroup_account_extent(), modify make it exit quicker for non-fs extents. This will also reduce the noise in trace_btrfs_qgroup_account_extent event. Signed-off-by: Qu Wenruo --- fs/btrfs/qgroup.c | 41 +++-- 1 file changed,

[RFC PATCH v3.1 0/6] Qgroup fixes, Non-stack version

The remaining qgroup fixes patches, based on the Chris' for-linus-4.12 branch with commit 9bcaaea7418d09691f1ffab5c49aacafe3eef9d0 as base. Can be fetched from github: https://github.com/adam900710/linux/tree/qgroup_fixes_non_stack Despite the 5th patch, patches are mostly unchanged. Only minor

Re: Btrfs/SSD

Austin S. Hemmelgarn posted on Mon, 17 Apr 2017 07:53:04 -0400 as excerpted: > * In my personal experience, Intel, Samsung, and Crucial appear to be > the best name brands (in relative order of quality). I have personally > had bad experiences with SanDisk and Kingston SSD's, but I don't have >

Re: [RFC PATCH v1 00/30] fs: inode->i_version rework and optimization

On Thu, May 11 2017, J. Bruce Fields wrote: > On Wed, Apr 05, 2017 at 02:14:09PM -0400, J. Bruce Fields wrote: >> On Wed, Apr 05, 2017 at 10:05:51AM +0200, Jan Kara wrote: >> > 1) Keep i_version as is, make clients also check for i_ctime. >> >> That would be a protocol revision, which we'd

Re: Creating btrfs RAID on LUKS devs makes devices disappear

Hello, here is the journal.log (I hope). It's quite interesting. I rebooted the machine, performed a mkfs.btrfs on dm-{2,3,4} and dm-3 was missing afterwards (around timestamp 66.*). However, I then logged into the machine from another terminal (around timestamp 118.*) which triggered

Re: errno=-28 No space left, with kernel backtrace (blocking bug)

Hi. I wish mean: I can't. I now for the btrfs maturity. But it's my unique alternative. I understand. For me this bug should be important because it block all the system, since Linux 4.1+ It's exactly what I wish, pay to have a quick fix. I don't think I wish too much, just fix this bug and

Re: [PATCH 2/8] nowait aio: Introduce RWF_NOWAIT

On Tue, May 09, 2017 at 07:22:13AM -0500, Goldwyn Rodrigues wrote: > From: Goldwyn Rodrigues > > This flag informs kernel to bail out if an AIO request will block > for reasons such as file allocations, or a writeback triggered, > or would block while allocating requests while

Re: [PATCH 3/8] nowait aio: return if direct write will trigger writeback

It might make sense to move filemap_range_has_page into a separate prep patch. Otherwise this looks fine: Reviewed-by: Christoph Hellwig -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo

Re: [RFC PATCH v3 5/6] btrfs: qgroup: Introduce extent changeset for qgroup reserve functions

At 05/11/2017 01:59 AM, Goldwyn Rodrigues wrote: On 05/09/2017 09:36 PM, Qu Wenruo wrote: Introduce a new parameter, struct extent_changeset for btrfs_qgroup_reserved_data() and its callers. Such extent_changeset was used in btrfs_qgroup_reserve_data() to record which range it reserved in

Re: [PATCH 7/8] nowait aio: xfs

Looks fine, Reviewed-by: Christoph Hellwig -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH 5/8] nowait aio: return on congested block device

Looks fine, Reviewed-by: Christoph Hellwig Although lifting the make_request limit is something a lot of users would appreciate in the near future.. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More

[PATCH] btrfs: Convert fs_info->free_chunk_space to atomic64_t

The ->free_chunk_space variable is used to track the unallocated space and access to it is protected by a spinlock, which is not used for anything else. Make the code a bit self-explanatory by switching the variable to an atomic64_t type and kill the spinlock. Signed-off-by: Nikolay Borisov

Re: [PATCH 1/8] Use RWF_* flags for AIO operations

Please add subsystem prefixes to your subject lines, e.g. fs: for all the generic fs ones, xfs: for XFS, block: for block layer changes, etc. > > - if (flags & ~(RWF_HIPRI | RWF_DSYNC | RWF_SYNC)) > - return -EOPNOTSUPP; > - > init_sync_kiocb(, filp); > - if

Re: Backing up BTRFS metadata

On Thu, May 11, 2017 at 8:56 AM, Marat Khalili wrote: > Sorry if question sounds unorthodox, Is there some simple way to read (and > backup) all BTRFS metadata from volume? btrfs-image -- Chris Murphy -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in

Re: errno=-28 No space left, with kernel backtrace (blocking bug)

Up plz, I can work with this bug. On 05/11/17 01:39, alpha_one_x86 wrote: > Hi, this bug is very blocking for me: > > https://bugzilla.kernel.org/show_bug.cgi?id=195257 > > The server is backup server, I btrfs receive (with and without -p), and > of course btrfs subvolume delete > The volume is

Backing up BTRFS metadata

Sorry if question sounds unorthodox, Is there some simple way to read (and backup) all BTRFS metadata from volume? Motivation of course is possibility to quickly recover from catastrophic filesystem failures on a logical level. Some small amount of actual data that this metadata references

Re: Backing up BTRFS metadata

On 11/05/17 18:19, Chris Murphy wrote: btrfs-image Looks great, thank you! -- With Best Regards, Marat Khalili -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at

Re: [PATCH 2/6] Btrfs: use bio_clone_bioset_partial to simplify DIO submit

On Mon, Apr 17, 2017 at 06:16:23PM -0700, Liu Bo wrote: > Currently when mapping bio to limit bio to a single stripe length, we > split bio by adding page to bio one by one, but later we don't modify > the vector of bio at all, thus we can use bio_clone_fast to use the > original bio vector

Re: runtime btrfsck

Roman Mamedov posted on Wed, 10 May 2017 13:52:55 +0500 as excerpted: > So even with a minor corruption (something wonky in just ONE block of a > multi-terabyte FS) the answer is way too often "nuke the entire thing > and restore from backups". Just another case where my "keep it small enough to

parent transid verify failed

Hello, this is some btrfs-on-luks, USB hdd as blockdevice. I can't mount my btrfs anymore, getting continuously the same syslog error: - Last output repeated twice - May 11 07:58:25 [kernel] BTRFS error (device dm-3): failed to read block groups: -5 May 11 07:58:25 [kernel] BTRFS error (device

btrfs list corruption and soft lockups while testing writeback error handling

I finally got my writeback error handling test to work on btrfs (thanks, Chris!), by making the filesystem stripe the data and mirror the metadata across two devices. The test passes now, but on one run, I got the following list corruption warning and then a soft lockup (which is probably fallout

Creating btrfs RAID on LUKS devs makes devices disappear

I should have added some more technical info. Here you go: Arch Linux with systemd 233 Kernel linux-lts 4.9.27 btrfs-progs 4.10.2 Example session: root@nas> ls /dev/dm-* /dev/dm-0 /dev/dm-1 /dev/dm-2 /dev/dm-3 /dev/dm-4 root@nas> ls -l /dev/mapper total 0 lrwxrwxrwx 1 root root 7 May

Re: [PATCH v3] btrfs: relocation: Enhance kernel error output for relocation

On Wed, May 10, 2017 at 11:39:40AM +0800, Qu Wenruo wrote: > > > At 05/10/2017 01:29 AM, David Sterba wrote: > > On Wed, Feb 15, 2017 at 09:39:05AM +0800, Qu Wenruo wrote: > >> When balance(relocation) fails, btrfs-progs will report like: > >> > >> ERROR: error during balancing '/mnt/scratch':

Qgroup reserved space like in ZFS?

Hello everyone, I just wanted to ask a short question as I couldn't find a clear answer anywhere on the net, yet: Is it currently possible to reserve space for a BTRFS subvolume? Following example: I have two subvolumes, one for root and one for home I'd like to reserve 2GB of space for my

Re: [PATCH 5/8] nowait aio: return on congested block device

On 05/11/2017 02:44 AM, Christoph Hellwig wrote: > Looks fine, > > Reviewed-by: Christoph Hellwig > > Although lifting the make_request limit is something a lot of users > would appreciate in the near future.. > Yes, I understand. That will be on my todo list next on priority.

Re: [RFC PATCH v1 00/30] fs: inode->i_version rework and optimization

On Wed, Apr 05, 2017 at 02:14:09PM -0400, J. Bruce Fields wrote: > On Wed, Apr 05, 2017 at 10:05:51AM +0200, Jan Kara wrote: > > 1) Keep i_version as is, make clients also check for i_ctime. > > That would be a protocol revision, which we'd definitely rather avoid. > > But can't we accomplish

[PATCH 04/10] fs: Introduce RWF_NOWAIT

From: Goldwyn Rodrigues RWF_NOWAIT informs kernel to bail out if an AIO request will block for reasons such as file allocations, or a writeback triggered, or would block while allocating requests while performing direct I/O. RWF_NOWAIT is translated to IOCB_NOWAIT for

Re: btrfs list corruption and soft lockups while testing writeback error handling

On Thu, 2017-05-11 at 07:13 -0400, Jeff Layton wrote: > I finally got my writeback error handling test to work on btrfs (thanks, > Chris!), by making the filesystem stripe the data and mirror the > metadata across two devices. The test passes now, but on one run, I got > the following list

Re: btrfs list corruption and soft lockups while testing writeback error handling

On 05/11/2017 03:52 PM, Jeff Layton wrote: On Thu, 2017-05-11 at 07:13 -0400, Jeff Layton wrote: I finally got my writeback error handling test to work on btrfs (thanks, Chris!), by making the filesystem stripe the data and mirror the metadata across two devices. The test passes now, but on one

[PATCH 07/10] fs: return on congested block device

From: Goldwyn Rodrigues A new bio operation flag REQ_NOWAIT is introduced to identify bio's orignating from iocb with IOCB_NOWAIT. This flag indicates to return immediately if a request cannot be made instead of retrying. Stacked devices such as md (the ones with

[PATCH 06/10] fs: Introduce IOMAP_NOWAIT

From: Goldwyn Rodrigues IOCB_NOWAIT translates to IOMAP_NOWAIT for iomaps. This is used by XFS in the XFS patch. Signed-off-by: Goldwyn Rodrigues Reviewed-by: Christoph Hellwig --- fs/iomap.c| 2 ++ include/linux/iomap.h | 1 + 2

[PATCH 09/10] xfs: nowait aio support

From: Goldwyn Rodrigues If IOCB_NOWAIT is set, bail if the i_rwsem is not lockable immediately. IF IOMAP_NOWAIT is set, return EAGAIN in xfs_file_iomap_begin if it needs allocation either due to file extension, writing to a hole, or COW or waiting for other DIOs to finish.

[PATCH 03/10] fs: Use RWF_* flags for AIO operations

From: Goldwyn Rodrigues aio_rw_flags is introduced in struct iocb (using aio_reserved1) which will carry the RWF_* flags. We cannot use aio_flags because they are not checked for validity which may break existing applications. Note, the only place RWF_HIPRI comes in effect is

[PATCH 02/10] fs: Introduce filemap_range_has_page()

From: Goldwyn Rodrigues filemap_range_has_page() return true if the file's mapping has a page within the range mentioned. This function will be used to check if a write() call will cause a writeback of previous writes. Signed-off-by: Goldwyn Rodrigues

[PATCH 10/10] btrfs: nowait aio support

From: Goldwyn Rodrigues Return EAGAIN if any of the following checks fail + i_rwsem is not lockable + NODATACOW or PREALLOC is not set + Cannot nocow at the desired location + Writing beyond end of file which is not allocated Signed-off-by: Goldwyn Rodrigues

[PATCH 08/10] ext4: nowait aio support

From: Goldwyn Rodrigues Return EAGAIN if any of the following checks fail for direct I/O: + i_rwsem is lockable + Writing beyond end of file (will trigger allocation) + Blocks are not allocated at the write location Signed-off-by: Goldwyn Rodrigues

[PATCH 05/10] fs: return if direct write will trigger writeback

From: Goldwyn Rodrigues Find out if the write will trigger a wait due to writeback. If yes, return -EAGAIN. Return -EINVAL for buffered AIO: there are multiple causes of delay such as page locks, dirty throttling logic, page loading from disk etc. which cannot be taken care

Re: Backing up BTRFS metadata

On Thu, 11 May 2017 09:19:28 -0600 Chris Murphy wrote: > On Thu, May 11, 2017 at 8:56 AM, Marat Khalili wrote: > > Sorry if question sounds unorthodox, Is there some simple way to read (and > > backup) all BTRFS metadata from volume? > > btrfs-image Hm, I

Re: Question on compression unit

Thanks Qu! I wonder if there is anyway we can easily configure the extent size (maximum extent size, extent size for files to compress, etc.)? I was trying to see if it helps reduce random read latency on compressed files by using smaller extent... On Wed, May 10, 2017 at 6:01 PM, Qu Wenruo

Re: Backing up BTRFS metadata

On Fri, May 12, 2017 at 12:22:00AM +0500, Roman Mamedov wrote: > On Thu, 11 May 2017 09:19:28 -0600 > Chris Murphy wrote: > > > On Thu, May 11, 2017 at 8:56 AM, Marat Khalili wrote: > > > Sorry if question sounds unorthodox, Is there some simple way to read

Creating btrfs RAID on LUKS devs makes devices disappear

Hello, while trying to initialize a btrfs RAID1 on my new NAS using LUKS crypt-devices for each of the btrfs RAID devices, I have seen "random" weirdness shortly after mkfs. It seems to boil down to the problem that after mkfs.btrfs, some of the /dev/dm-* nodes (as well as the corresponding

[PATCH 0/10 v8] No wait AIO

Formerly known as non-blocking AIO. This series adds nonblocking feature to asynchronous I/O writes. io_submit() can be delayed because of a number of reason: - Block allocation for files - Data writebacks for direct I/O - Sleeping because of waiting to acquire i_rwsem - Congested block

[PATCH 01/10] fs: Separate out kiocb flags setup based on RWF_* flags

From: Goldwyn Rodrigues Signed-off-by: Goldwyn Rodrigues Reviewed-by: Christoph Hellwig --- fs/read_write.c| 12 +++- include/linux/fs.h | 14 ++ 2 files changed, 17 insertions(+), 9 deletions(-) diff --git

[PATCH v2 2/2] btrfs: Add quota_override knob into sysfs

This patch adds the read-write attribute quota_override into sysfs. Any process which has cap_sys_resource can set this flag to on, and once it is set to true, processes with cap_sys_resource can exceed the quota. Signed-off-by: Sargun Dhillon --- fs/btrfs/sysfs.c | 41

[PATCH v2 1/2] btrfs: add quota override flag to enable quota override for sys_resource

This patch introduces the quota override flag to btrfs_fs_info, and a change to quota limit checking code to temporarily allow for quota to be overridden for processes with cap_sys_resource. It's useful for administrative programs, such as log rotation, that may need to temporarily use more disk

[PATCH v2 0/2] btrfs: allow mechanism to override quota

This patchset makes it so that on a per-filesystem basis one can disable quota enforcement for users with cap_sys_resource. This patchset can likely later be extended to per-qgroup, or a per-volume basis. I'm thinking of extending the sysfs interface to list the qgroups and this same interface for