Re: Ongoing Btrfs stability issues

On 16.02.2018 06:54, Alex Adriaanse wrote: > >> On Feb 15, 2018, at 2:42 PM, Nikolay Borisov wrote: >> >> On 15.02.2018 21:41, Alex Adriaanse wrote: >>> On Feb 15, 2018, at 12:00 PM, Nikolay Borisov wrote: So in all of the cases you are

[RFC PATCH josef-btrfs] block: blkcg_qos_init() can be static

Fixes: 6389939532e7 ("block: introduce blkcg-qos io controller") Signed-off-by: Fengguang Wu --- blkcg-qos.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/blkcg-qos.c b/block/blkcg-qos.c index a28adcc..6588b1c 100644 --- a/block/blkcg-qos.c

[josef-btrfs:blk-qos 13/13] block/blkcg-qos.c:340:5: sparse: symbol 'blkcg_qos_init' was not declared. Should it be static?

tree: https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git blk-qos head: 6389939532e7f4a0d8f19f9efc8a4936f8368d29 commit: 6389939532e7f4a0d8f19f9efc8a4936f8368d29 [13/13] block: introduce blkcg-qos io controller reproduce: # apt-get install sparse git checkout

Re: Ongoing Btrfs stability issues

> On Feb 15, 2018, at 2:42 PM, Nikolay Borisov wrote: > > On 15.02.2018 21:41, Alex Adriaanse wrote: >> >>> On Feb 15, 2018, at 12:00 PM, Nikolay Borisov wrote: >>> >>> So in all of the cases you are hitting some form of premature enospc. >>> There was a

[PATCH] btrfs: use kvzalloc to allocate btrfs_fs_info

From: Jeff Mahoney The srcu_struct in btrfs_fs_infoa scales in size with NR_CPUS. On kernels built with NR_CPUS=8192, this can result in kmalloc failures that prevent mounting. There is work in progress to try to resolve this for every user of srcu_struct but using kvzalloc

Re: Metadata / Data on Heterogeneous Media

On 2018年02月16日 01:15, Ellis H. Wilson III wrote: > In discussing the performance of various metadata operations over the > past few days I've had this idea in the back of my head, and wanted to > see if anybody had already thought about it before (likely, I would guess). > > It appears based on

[josef-btrfs:current-work 3/3] block/blk-wbt.c:1014:33: error: 'struct blkcg' has no member named 'css'

tree: https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git current-work head: 8e6eb5db8cf85a5af945098166954454c3169b29 commit: 8e6eb5db8cf85a5af945098166954454c3169b29 [3/3] current-work config: x86_64-randconfig-s0-02160955 (attached as .config) compiler: gcc-6 (Debian

btrfs send/receive in reverse possible?

I have snapshot A on Drive_A. I send snapshot A to an empty Drive_B. Then keep Drive_A as backup. I use Drive_B as active. I create new snapshot B on Drive_B. Can I use btrfs send/receive to send incremental differences back to Drive_A? What is the correct way of doing this? Regards, Sampson --

Re: Status of FST and mount times

On 2018年02月16日 00:30, Ellis H. Wilson III wrote: > On 02/15/2018 06:12 AM, Hans van Kranenburg wrote: >> On 02/15/2018 02:42 AM, Qu Wenruo wrote: >>> Just as said by Nikolay, the biggest problem of slow mount is the size >>> of extent tree (and HDD seek time) >>> >>> The easiest way to get a

[josef-btrfs:blk-qos 2/13] include/linux/bio.h:521:12: error: 'bio_associate_blkg' defined but not used

Hi Josef, FYI, the error/warning still remains. tree: https://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git blk-qos head: 6389939532e7f4a0d8f19f9efc8a4936f8368d29 commit: 4ece201cfafab298cd89083567403a53f0971635 [2/13] block: add bi_blkg to the bio for cgroups config:

Re: Ongoing Btrfs stability issues

On 15.02.2018 21:41, Alex Adriaanse wrote: > >> On Feb 15, 2018, at 12:00 PM, Nikolay Borisov wrote: >> >> So in all of the cases you are hitting some form of premature enospc. >> There was a fix that landed in 4.15 that should have fixed a rather >> long-standing issue with

Re: Metadata / Data on Heterogeneous Media

On 02/15/2018 02:11 PM, Hugo Mills wrote: On Thu, Feb 15, 2018 at 12:15:49PM -0500, Ellis H. Wilson III wrote: In discussing the performance of various metadata operations over the past few days I've had this idea in the back of my head, and wanted to see if anybody had already thought about it

Re: Metadata / Data on Heterogeneous Media

On 02/15/2018 02:06 PM, Adam Borowski wrote: On Thu, Feb 15, 2018 at 12:15:49PM -0500, Ellis H. Wilson III wrote: In discussing the performance of various metadata operations over the past few days I've had this idea in the back of my head, and wanted to see if anybody had already thought about

Re: Ongoing Btrfs stability issues

> On Feb 15, 2018, at 12:00 PM, Nikolay Borisov wrote: > > So in all of the cases you are hitting some form of premature enospc. > There was a fix that landed in 4.15 that should have fixed a rather > long-standing issue with the way metadata reservations are satisfied, >

Unable to Scrub or Repair a volume after bad Scrub (mountable)

Hi A few weeks ago I was a bit under the weather for a week, and so was unable to give my home server any attention. Unfortunately during this time it started a scheduled scrub and decided there was billions of errors. When I noticed this, maybe not thinking entirely straight (i.e. not trying to

Re: Metadata / Data on Heterogeneous Media

On Thu, Feb 15, 2018 at 12:15:49PM -0500, Ellis H. Wilson III wrote: > In discussing the performance of various metadata operations over > the past few days I've had this idea in the back of my head, and > wanted to see if anybody had already thought about it before > (likely, I would guess). > >

[PATCH v2 16/27] btrfs-progs: use libbtrfsutil for read-only property

From: Omar Sandoval Signed-off-by: Omar Sandoval --- messages.h | 13 props.c| 69 +++--- 2 files changed, 38 insertions(+), 44 deletions(-) diff --git a/messages.h b/messages.h index

[PATCH v2 17/27] btrfs-progs: use libbtrfsutil for sync ioctls

From: Omar Sandoval Signed-off-by: Omar Sandoval --- cmds-filesystem.c | 19 ++- cmds-qgroup.c | 10 +++--- cmds-subvolume.c | 25 + 3 files changed, 30 insertions(+), 24 deletions(-) diff --git

[PATCH v2 09/27] libbtrfsutil: add btrfs_util_[gs]et_read_only()

From: Omar Sandoval In the future, btrfs_util_[gs]et_subvolume_flags() might be useful, but since these are the only subvolume flags we've defined in all this time, this will do for now. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h

[PATCH v2 05/27] libbtrfsutil: add qgroup inheritance helpers

From: Omar Sandoval We want to hide struct btrfs_qgroup_inherit from the user because that comes from the Btrfs UAPI headers. Instead, wrap it in a struct btrfs_util_qgroup_inherit and provide helpers to manipulate it. This will be used for subvolume and snapshot creation.

[PATCH v2 06/27] libbtrfsutil: add btrfs_util_create_subvolume()

From: Omar Sandoval Doing the ioctl() directly isn't too bad, but passing in a full path is more convenient than opening the parent and passing the path component. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 34 +

[PATCH v2 13/27] libbtrfsutil: add btrfs_util_delete_subvolume()

From: Omar Sandoval We also support recursive deletion using a subvolume iterator. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 33 + libbtrfsutil/python/btrfsutilpy.h | 1 + libbtrfsutil/python/module.c

[PATCH v2 08/27] libbtrfsutil: add btrfs_util_subvolume_info()

From: Omar Sandoval This gets the the information in `btrfs subvolume show` from the root item. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 111 + libbtrfsutil/python/btrfsutilpy.h | 4 +

[PATCH v2 10/27] libbtrfsutil: add btrfs_util_[gs]et_default_subvolume()

From: Omar Sandoval set_default_subvolume() is a trivial ioctl(), but there's no ioctl() for get_default_subvolume(), so we need to search the root tree. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 41 ++

[PATCH v2 15/27] libbtrfsutil: add filesystem sync helpers

From: Omar Sandoval Namely, sync, start_sync, and wait_sync. Signed-off-by: Omar Sandoval --- Makefile | 4 +- libbtrfsutil/btrfsutil.h | 44 libbtrfsutil/filesystem.c|

Re: Metadata / Data on Heterogeneous Media

On Thu, Feb 15, 2018 at 12:15:49PM -0500, Ellis H. Wilson III wrote: > In discussing the performance of various metadata operations over the past > few days I've had this idea in the back of my head, and wanted to see if > anybody had already thought about it before (likely, I would guess). > > It

[PATCH v2 12/27] libbtrfsutil: add btrfs_util_create_snapshot()

From: Omar Sandoval Thanks to subvolume iterators, we can also implement recursive snapshot fairly easily. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 59 ++ libbtrfsutil/python/btrfsutilpy.h | 1 +

[PATCH v2 14/27] libbtrfsutil: add btrfs_util_deleted_subvolumes()

From: Omar Sandoval Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 21 +++ libbtrfsutil/python/btrfsutilpy.h | 3 + libbtrfsutil/python/module.c| 30 ++ libbtrfsutil/python/qgroup.c

[PATCH v2 11/27] libbtrfsutil: add subvolume iterator helpers

From: Omar Sandoval This is how we can implement stuff like `btrfs subvol list`. Rather than producing the entire list upfront, the iterator approach uses less memory in the common case where the whole list is not stored (O(max subvolume path length)). It supports both pre-order

[PATCH v2 20/27] btrfs-progs: use libbtrfsutil for subvol create and snapshot

From: Omar Sandoval These become trivial calls to the libbtrfsutil helpers, and we can get rid of the qgroup inherit code in progs. Signed-off-by: Omar Sandoval --- cmds-subvolume.c | 225 ++- qgroup.c

[PATCH v2 04/27] libbtrfsutil: add btrfs_util_is_subvolume() and btrfs_util_subvolume_id()

From: Omar Sandoval These are the most trivial helpers in the library and will be used to implement several of the more involved functions. Signed-off-by: Omar Sandoval --- Makefile| 2 +- libbtrfsutil/btrfsutil.h

[PATCH v2 18/27] btrfs-progs: use libbtrfsutil for set-default

From: Omar Sandoval Signed-off-by: Omar Sandoval --- cmds-subvolume.c | 43 --- 1 file changed, 8 insertions(+), 35 deletions(-) diff --git a/cmds-subvolume.c b/cmds-subvolume.c index 6006a278..700e822c 100644 ---

[PATCH v2 22/27] btrfs-progs: use libbtrfsutil for subvol show

From: Omar Sandoval Now implemented with btrfs_util_subvolume_path(), btrfs_util_subvolume_info(), and subvolume iterators. Signed-off-by: Omar Sandoval --- cmds-subvolume.c | 149 --- utils.c | 118

[PATCH v2 26/27] btrfs-progs: use libbtrfsutil for subvolume list

From: Omar Sandoval This is the most involved conversion because it replaces the libbtrfs implementation entirely. I took care to preserve the existing behavior in all cases, warts and all. This also moves the list printing code from libbtrfs to cmds-subvolume.c. This is an ABI

[PATCH v2 24/27] btrfs-progs: replace test_issubvolume() with btrfs_util_is_subvolume()

From: Omar Sandoval This gets the remaining occurrences that weren't covered by previous conversions. Signed-off-by: Omar Sandoval --- cmds-qgroup.c| 11 --- cmds-subvolume.c | 10 +++--- utils.c | 34 +-

[PATCH v2 27/27] btrfs-progs: deprecate libbtrfs helpers

From: Omar Sandoval The old libbtrfs defines some helpers which do the same thing as some libbtrfsutil helpers. Update btrfs-progs to use the libbtrfsutil APIs and mark the libbtrfs versions as deprecated, which we could ideally get rid of eventually. Signed-off-by: Omar

[PATCH v2 23/27] btrfs-progs: use libbtrfsutil for subvol sync

From: Omar Sandoval btrfs_util_f_deleted_subvolumes() replaces enumerate_dead_subvols() and btrfs_util_f_subvolume_info() replaces is_subvolume_cleaned(). Signed-off-by: Omar Sandoval --- cmds-subvolume.c | 217

[PATCH v2 25/27] btrfs-progs: add recursive snapshot/delete using libbtrfsutil

From: Omar Sandoval And update the documentation. Signed-off-by: Omar Sandoval --- Documentation/btrfs-subvolume.asciidoc | 14 -- cmds-subvolume.c | 25 + 2 files changed, 33 insertions(+), 6

[PATCH v2 02/27] Add libbtrfsutil

From: Omar Sandoval Currently, users wishing to manage Btrfs filesystems programatically have to shell out to btrfs-progs and parse the output. This isn't ideal. The goal of libbtrfsutil is to provide a library version of as many of the operations of btrfs-progs as possible and

[PATCH v2 19/27] btrfs-progs: use libbtrfsutil for get-default

From: Omar Sandoval The only thing of note here is the "top level" column. This used to mean something else, but for a long time it has been equal to the parent ID. I preserved this for backwards compatability. Signed-off-by: Omar Sandoval --- cmds-subvolume.c

[PATCH v2 21/27] btrfs-progs: use libbtrfsutil for subvol delete

From: Omar Sandoval Most of the interesting part of this command is the commit mode, so this only saves a little bit of code. Signed-off-by: Omar Sandoval --- cmds-subvolume.c | 22 -- 1 file changed, 8 insertions(+), 14 deletions(-) diff

[PATCH v2 07/27] libbtrfsutil: add btrfs_util_subvolume_path()

From: Omar Sandoval We can just walk up root backrefs with BTRFS_IOC_TREE_SEARCH and inode paths with BTRFS_IOC_INO_LOOKUP. Signed-off-by: Omar Sandoval --- libbtrfsutil/btrfsutil.h| 21 + libbtrfsutil/python/btrfsutilpy.h |

[PATCH v2 01/27] btrfs-progs: get rid of undocumented qgroup inheritance options

From: Omar Sandoval The -c option to subvol create and the -c and -x options to subvol snapshot have never been documented (and -x doesn't actually work because it's not in the getopt option string). The functionality is dubious and the kernel interface is being removed, so get

[PATCH v2 03/27] libbtrfsutil: add Python bindings

From: Omar Sandoval The C libbtrfsutil library isn't very useful for scripting, so we also want bindings for Python. Writing unit tests in Python is also much easier than doing so in C. Only Python 3 is supported; if someone really wants Python 2 support, they can write their own

[PATCH v2 00/27] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"

From: Omar Sandoval Hi, This is v2 of "btrfs-progs as a library". Most of the changes since v1 are small: - Rebased onto v4.15 - Split up btrfs_util_subvolume_path() which was accidentally squashed together with the commit adding btrfs_util_create_subvolume() - Renamed

Btrfs progs pre-release 4.15.1-rc1

Hi, a pre-release has been tagged. This is going to be a minor maintenance release. ETA for 4.15.1 is in +1 days (2018-02-16). Changes: * build * fix build on musl * support asciidoctor for doc generation * cleanups * sync some code with kernel * move code to own directory, split to

Re: Ongoing Btrfs stability issues

On 15.02.2018 18:18, Alex Adriaanse wrote: > We've been using Btrfs in production on AWS EC2 with EBS devices for over 2 > years. There is so much I love about Btrfs: CoW snapshots, compression, > subvolumes, flexibility, the tools, etc. However, lack of stability has been > a serious ongoing

Fwd: Enabling Extended Attribute Support

This e-mail message (including attachments, if any) is intended solely for the use of the individual or entity to which it is addressed and may contain information that is private and confidential. If you are not the intended recipient, you are notified that any dissemination, distribution or

Re: Status of FST and mount times

On 2018-02-15 11:58, Ellis H. Wilson III wrote: On 02/15/2018 11:51 AM, Austin S. Hemmelgarn wrote: There are scaling performance issues with directory listings on BTRFS for directories with more than a few thousand files, but they're not well documented (most people don't hit them because

Metadata / Data on Heterogeneous Media

In discussing the performance of various metadata operations over the past few days I've had this idea in the back of my head, and wanted to see if anybody had already thought about it before (likely, I would guess). It appears based on this page:

Re: Status of FST and mount times

On 02/15/2018 11:51 AM, Austin S. Hemmelgarn wrote: There are scaling performance issues with directory listings on BTRFS for directories with more than a few thousand files, but they're not well documented (most people don't hit them because most applications are designed around the

Re: [PATCH v2] btrfs: fix endianness compatibility during the SB RW

On Wed, Feb 14, 2018 at 10:53:19PM +0800, Anand Jain wrote: > > > On 02/14/2018 01:55 AM, David Sterba wrote: > > On Tue, Feb 13, 2018 at 06:27:13PM +0800, Anand Jain wrote: > >> On 02/13/2018 05:01 PM, Qu Wenruo wrote: > >>> On 2018年02月13日 11:00, Anand Jain wrote: > Fixes the endianness

Re: Status of FST and mount times

On 2018-02-15 10:42, Ellis H. Wilson III wrote: On 02/14/2018 06:24 PM, Duncan wrote: Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No compression.  No quotas enabled.  Many (potentially tens to hundreds) of subvolumes, each with tens of snapshots.  No control over size or number

Re: Status of FST and mount times

On 02/15/2018 01:14 AM, Chris Murphy wrote: On Wed, Feb 14, 2018 at 9:00 AM, Ellis H. Wilson III wrote: Frame-of-reference here: RAID0. Around 70TB raw capacity. No compression. No quotas enabled. Many (potentially tens to hundreds) of subvolumes, each with tens of

Re: [PATCH v2 2/2] btrfs: verify max_inline mount parameter

On Thu, Feb 15, 2018 at 01:20:55AM +0800, Anand Jain wrote: > > > On 02/14/2018 12:28 AM, David Sterba wrote: > > On Tue, Feb 13, 2018 at 05:49:50PM +0800, Anand Jain wrote: > >> We aren't verifying the parameter passed to the max_inline mount option, > >> so we won't report and fail the mount

Re: [PATCH 3/8] fs: btrfs: remove unused hardirq.h

On Thu, Feb 15, 2018 at 07:34:44AM +0800, Yang Shi wrote: > David acked this patch, but it looks nobody picked it up. I'll add it to the for-next patch queue. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More

Re: Status of FST and mount times

On 02/14/2018 06:24 PM, Duncan wrote: Frame-of-reference here: RAID0. Around 70TB raw capacity. No compression. No quotas enabled. Many (potentially tens to hundreds) of subvolumes, each with tens of snapshots. No control over size or number of files, but directory tree (entries per dir and

Re: Status of FST and mount times

On 02/15/2018 06:12 AM, Hans van Kranenburg wrote: On 02/15/2018 02:42 AM, Qu Wenruo wrote: Just as said by Nikolay, the biggest problem of slow mount is the size of extent tree (and HDD seek time) The easiest way to get a basic idea of how large your extent tree is using debug tree: #

Ongoing Btrfs stability issues

We've been using Btrfs in production on AWS EC2 with EBS devices for over 2 years. There is so much I love about Btrfs: CoW snapshots, compression, subvolumes, flexibility, the tools, etc. However, lack of stability has been a serious ongoing issue for us, and we're getting to the point that

Enabling Extended Attribute Support

This e-mail message (including attachments, if any) is intended solely for the use of the individual or entity to which it is addressed and may contain information that is private and confidential. If you are not the intended recipient, you are notified that any dissemination, distribution or

Re: Status of FST and mount times

On 02/15/2018 02:42 AM, Qu Wenruo wrote: > > > On 2018年02月15日 01:08, Nikolay Borisov wrote: >> >> >> On 14.02.2018 18:00, Ellis H. Wilson III wrote: >>> Hi again -- back with a few more questions: >>> >>> Frame-of-reference here: RAID0.  Around 70TB raw capacity.  No >>> compression.  No quotas

Re: [PATCH v2] btrfs: use ASSERT to report logical error in cow_file_range()

On 15.02.2018 12:07, Anand Jain wrote: > Use ASSERT to report logical error in cow_file_range(), also move > it a bit closer to when the num_bytes is derived. > > Signed-off-by: Anand Jain Reviewed-by: Nikolay Borisov > --- > v1->v2: >ASSERT

[PATCH v2] btrfs: use ASSERT to report logical error in cow_file_range()

Use ASSERT to report logical error in cow_file_range(), also move it a bit closer to when the num_bytes is derived. Signed-off-by: Anand Jain --- v1->v2: ASSERT logic changed. Thanks Nikolay. fs/btrfs/inode.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)

Re: fatal database corruption with btrfs "out of space" with ~50 GB left

On 15.02.2018 09:17, Tomasz Chmielewski wrote: > On 2018-02-15 16:02, Tomasz Chmielewski wrote: >> On 2018-02-15 13:32, Qu Wenruo wrote: >> >>> Is there any kernel message like kernel warning or backtrace? >> >> I see there was this one: >> >> Feb 13 13:53:32 lxd01 kernel: [9351710.878404]

Re: [PATCH] btrfs: use ASSERT to report logical error in cow_file_range()

On 15.02.2018 06:30, Anand Jain wrote: > Use ASSERT to report logical error in cow_file_range(), also move > it a bit closer to when the num_bytes is derived. > > Signed-off-by: Anand Jain > --- > fs/btrfs/inode.c | 5 ++--- > 1 file changed, 2 insertions(+), 3

Re: [PATCH v2] btrfs: cow_file_range() num_bytes and disk_num_bytes are same

On 15.02.2018 06:29, Anand Jain wrote: > This patch deletes local variable disk_num_bytes as its value > is same as num_bytes in the function cow_file_range(). > > Signed-off-by: Anand Jain Reviewed-by: Nikolay Borisov > --- > v1->v2: > Fix

Re: [PATCH] btrfs: remove unused function btrfs_async_submit_limit()

On 15.02.2018 05:36, Anand Jain wrote: > Commit [1] removed the need to use btrfs_async_submit_limit(), so > delete it. > > [1] > commit 736cd52e0c720103f52ab9da47b6cc3af6b083f6 > Btrfs: remove nr_async_submits and async_submit_draining > > Signed-off-by: Anand Jain