Re: BTRFS as a GlusterFS storage back-end, and what I've learned from using it as such.

2017-04-11 Thread Qu Wenruo
At 04/11/2017 11:40 PM, Austin S. Hemmelgarn wrote: About a year ago now, I decided to set up a small storage cluster to store backups (and partially replace Dropbox for my usage, but that's a separate story). I ended up using GlusterFS as the clustering software itself, and BTRFS as the bac

Re: Btrfs disk layout question

2017-04-11 Thread Chris Murphy
On Tue, Apr 11, 2017 at 3:00 PM, Adam Borowski wrote: > On Tue, Apr 11, 2017 at 12:15:32PM -0700, Amin Hassani wrote: >> I am working on a project with Btrfs and I was wondering if there is >> any way to see the disk layout of the btrfs image. Let's assume I have >> a read-only btrfs image with co

Re: BTRFS as a GlusterFS storage back-end, and what I've learned from using it as such.

2017-04-11 Thread Ravishankar N
Adding gluster-users list. I think there are a few users out there running gluster on top of btrfs, so this might benefit a broader audience. On 04/11/2017 09:10 PM, Austin S. Hemmelgarn wrote: About a year ago now, I decided to set up a small storage cluster to store backups (and partially rep

Re: [PATCH] fstests: introduce btrfs-map-logical

2017-04-11 Thread Qu Wenruo
At 04/12/2017 09:27 AM, Liu Bo wrote: A typical use case of 'btrfs-map-logical' is to translate btrfs logical address to physical address on each disk. Could we avoid usage of btrfs-map-logical here? I understand that we need to do corruption so that we can test if the repair works, but I'm

[PATCH] fstests: regression test for btrfs buffered read's repair

2017-04-11 Thread Liu Bo
This case tests whether buffered read can repair the bad copy if we have a good copy. Signed-off-by: Liu Bo --- tests/btrfs/141 | 152 tests/btrfs/141.out | 39 ++ tests/btrfs/group | 1 + 3 files changed, 192 insertions(+

[PATCH] fstests: regression test for btrfs dio read repair

2017-04-11 Thread Liu Bo
This case tests whether dio read can repair the bad copy if we have a good copy. Signed-off-by: Liu Bo --- tests/btrfs/140 | 152 tests/btrfs/140.out | 39 ++ tests/btrfs/group | 1 + 3 files changed, 192 insertions(+) cr

[PATCH] fstests: introduce btrfs-map-logical

2017-04-11 Thread Liu Bo
A typical use case of 'btrfs-map-logical' is to translate btrfs logical address to physical address on each disk. Signed-off-by: Liu Bo --- common/config | 1 + 1 file changed, 1 insertion(+) diff --git a/common/config b/common/config index 59041a3..b7c06bf 100644 --- a/common/config +++ b/comm

[PATCH] fstests: remove snapshot aware defrag test

2017-04-11 Thread Liu Bo
Since snapshot aware defrag has been disabled in kernel, and we all have learned to ignore the failure of btrfs/010, lets just remove it. Signed-off-by: Liu Bo --- tests/btrfs/010 | 93 - tests/btrfs/010.out | 2 -- tests/btrfs/group |

Re: Btrfs disk layout question

2017-04-11 Thread Adam Borowski
On Tue, Apr 11, 2017 at 12:15:32PM -0700, Amin Hassani wrote: > I am working on a project with Btrfs and I was wondering if there is > any way to see the disk layout of the btrfs image. Let's assume I have > a read-only btrfs image with compression on and only using one disk > (no raid or anything)

Re: Btrfs disk layout question

2017-04-11 Thread Chris Murphy
On Tue, Apr 11, 2017 at 1:15 PM, Amin Hassani wrote: > Hi, > > I am working on a project with Btrfs and I was wondering if there is > any way to see the disk layout of the btrfs image. Let's assume I have > a read-only btrfs image with compression on and only using one disk > (no raid or anything)

Re: [PATCH 06/12] audit: Use timespec64 to represent audit timestamps

2017-04-11 Thread Paul Moore
On Sat, Apr 8, 2017 at 1:58 PM, Deepa Dinamani wrote: >> I have no problem merging this patch into audit/next for v4.12, would >> you prefer me to do that so at least this patch is merged? > > This would be fine. > But, I think whoever takes the last 2 deletion patches should also take them. > I'm

Btrfs disk layout question

2017-04-11 Thread Amin Hassani
Hi, I am working on a project with Btrfs and I was wondering if there is any way to see the disk layout of the btrfs image. Let's assume I have a read-only btrfs image with compression on and only using one disk (no raid or anything). Is it possible to get a set of offset-lengths for each file or

[PULL] Btrfs fixes for 4.11-rc7

2017-04-11 Thread David Sterba
there are some fixes for crashes or use-after-free and one fix to the mount options handling (user-visible behaviour). Please pull to 4.11, thanks. The following changes since commit 41a75a6eb2bf4e75e0d93862171bebb3ca4efec7: Merg

[PATCH v2] btrfs-progs: send-dump: always print a space after path

2017-04-11 Thread Evan Danaher
I was shocked to discover that 'btrfs receive --dump' doesn't print a space after long filenames, so it runs together into the metadata; for example: truncate./20-00-03/this-name-is-32-characters-longsize=0 This is a trivial patch to add a single space unconditionally, so the result is th

Re: btrfs won't mount any more

2017-04-11 Thread Adam Borowski
On Tue, Apr 11, 2017 at 09:15:31AM +0200, Marc Haber wrote: > I have wrecked another btrfs file system, probably for good this time. > > It's a 80 GB filesystem from 2015, in my secondary notebook, on an > encrypted SSD. The btrfs holds the root filesystem and the rest of the > system as well. >

BTRFS as a GlusterFS storage back-end, and what I've learned from using it as such.

2017-04-11 Thread Austin S. Hemmelgarn
About a year ago now, I decided to set up a small storage cluster to store backups (and partially replace Dropbox for my usage, but that's a separate story). I ended up using GlusterFS as the clustering software itself, and BTRFS as the back-end storage. GlusterFS itself is actually a pretty

Unexpected: send/receive much slower than rsync ?

2017-04-11 Thread J. Hart
I'm trying to update from an old snapshot of a directory to a new one using send/receive. It seems a great deal slower than I was expecting, perhaps much slower than rsync and has been running for hours. Everything looks ok with how I set up the snapshots, and there are no error messages, but I

[PATCH 6/9] nowait aio: ext4

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues Return EAGAIN if any of the following checks fail for direct I/O: + i_rwsem is lockable + Writing beyond end of file (will trigger allocation) + Blocks are not allocated at the write location Signed-off-by: Goldwyn Rodrigues --- fs/ext4/file.c | 20 +++

[PATCH 8/9] nowait aio: btrfs

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues Return EAGAIN if any of the following checks fail + i_rwsem is not lockable + NODATACOW or PREALLOC is not set + Cannot nocow at the desired location + Writing beyond end of file which is not allocated Signed-off-by: Goldwyn Rodrigues --- fs/btrfs/file.c | 25 +

[PATCH 5/9] nowait aio: return on congested block device

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues A new flag BIO_NOWAIT is introduced to identify bio's orignating from iocb with IOCB_NOWAIT. This flag indicates to return immediately if a request cannot be made instead of retrying. To facilitate this, QUEUE_FLAG_NOWAIT is set to devices which support this. While curren

[PATCH 3/9] nowait aio: return if direct write will trigger writeback

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues Find out if the write will trigger a wait due to writeback. If yes, return -EAGAIN. This introduces a new function filemap_range_has_page() which returns true if the file's mapping has a page within the range mentioned. Return -EINVAL for buffered AIO: there are multiple

[PATCH 4/9] nowait-aio: Introduce IOMAP_NOWAIT

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues IOCB_NOWAIT translates to IOMAP_NOWAIT for iomaps. This is used by XFS in the XFS patch. --- fs/iomap.c| 2 ++ include/linux/iomap.h | 1 + 2 files changed, 3 insertions(+) diff --git a/fs/iomap.c b/fs/iomap.c index 141c3cd55a8b..d1c81753d411 100644 --- a/fs/

[PATCH 7/9] nowait aio: xfs

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues If IOCB_NOWAIT is set, bail if the i_rwsem is not lockable immediately. IF IOMAP_NOWAIT is set, return EAGAIN in xfs_file_iomap_begin if it needs allocation either due to file extension, writing to a hole, or COW or waiting for other DIOs to finish. Signed-off-by: Goldwy

[PATCH 9/9] nowait aio: Return -EOPNOTSUPP if filesystem does not support

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues The check is in generic_file_write_iter(), which is called by most filesystems, either through fsops.write_iter() or through the function defined by write_iter(). If not, we perform the check in the defined .write_iter() function which is called for direct IO. Signed-off-

[PATCH 0/9 v5] No wait AIO

2017-04-11 Thread Goldwyn Rodrigues
Formerly known as non-blocking AIO. This series adds nonblocking feature to asynchronous I/O writes. io_submit() can be delayed because of a number of reason: - Block allocation for files - Data writebacks for direct I/O - Sleeping because of waiting to acquire i_rwsem - Congested block device

[PATCH 2/9] nowait aio: Introduce RWF_NOWAIT

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues This flag informs kernel to bail out if an AIO request will block for reasons such as file allocations, or a writeback triggered, or would block while allocating requests while performing direct I/O. Unfortunately, aio_flags is not checked for validity, which would break

[PATCH 1/9] Use RWF_* flags for AIO operations

2017-04-11 Thread Goldwyn Rodrigues
From: Goldwyn Rodrigues RWF_* flags is used for preadv2/pwritev2 calls. Port to use it for aio operations as well. For this, aio_rw_flags is introduced in struct iocb (using aio_reserved1) which will carry these flags. This is a precursor to the nowait AIO calls. Note, the only place RWF_HIPRI

Re: [PATCH] btrfs-progs: send-dump: always print a space after path

2017-04-11 Thread Evan Danaher
Thanks for catching that! I overthought this and managed to convince myself that it was correct as it stood. Should I re-send the whole patch with that change? This is my first attempt at contributing to a project managed in the Linux kernel style, so I'm not sure about the process; documentatio

Re: [PATCH] btrfs-progs: send-dump: always print a space after path

2017-04-11 Thread Noah Massey
On Mon, Apr 10, 2017 at 8:09 PM, Evan Danaher wrote: > I was shocked to discover that 'btrfs receive --dump' doesn't print a > space after long filenames, so it runs together into the metadata; for > example: > > truncate./20-00-03/this-name-is-32-characters-longsize=0 > > This is a trivia

Re: About free space fragmentation, metadata write amplification and (no)ssd

2017-04-11 Thread Kai Krakow
Am Tue, 11 Apr 2017 07:33:41 -0400 schrieb "Austin S. Hemmelgarn" : > >> FWIW, it is possible to use a udev rule to change the rotational > >> flag from userspace. The kernel's selection algorithm for > >> determining is is somewhat sub-optimal (essentially, if it's not a > >> local disk that can

Re: [PATCH] Btrfs: fix potential use-after-free for cloned bio

2017-04-11 Thread David Sterba
On Mon, Apr 10, 2017 at 12:36:26PM -0700, Liu Bo wrote: > KASAN reports that there is a use-after-free case of bio in btrfs_map_bio. > > If we need to submit IOs to several disks at a time, the original bio > would get cloned and mapped to the destination disk, but we really should > use the origi

Re: About free space fragmentation, metadata write amplification and (no)ssd

2017-04-11 Thread Austin S. Hemmelgarn
On 2017-04-10 18:59, Hans van Kranenburg wrote: On 04/10/2017 02:23 PM, Austin S. Hemmelgarn wrote: On 2017-04-08 16:19, Hans van Kranenburg wrote: So... today a real life story / btrfs use case example from the trenches at work... tl;dr 1) btrfs is awesome, but you have to carefully choose wh

Re: btrfs filesystem keeps allocating new chunks for no apparent reason

2017-04-11 Thread Austin S. Hemmelgarn
On 2017-04-11 05:55, Adam Borowski wrote: On Tue, Apr 11, 2017 at 06:01:19AM +0200, Kai Krakow wrote: Yes, I know all this. But I don't see why you still want noatime or relatime if you use lazytime, except for super-optimizing. Lazytime gives you POSIX conformity for a problem that the other op

Re: btrfs filesystem keeps allocating new chunks for no apparent reason

2017-04-11 Thread Adam Borowski
On Tue, Apr 11, 2017 at 06:01:19AM +0200, Kai Krakow wrote: > Yes, I know all this. But I don't see why you still want noatime or > relatime if you use lazytime, except for super-optimizing. Lazytime > gives you POSIX conformity for a problem that the other options only > tried to solve. (Besides

[PATCH] Btrfs: remove some dead code

2017-04-11 Thread Dan Carpenter
btrfs_get_extent() never returns NULL pointers, so this code introduces a static checker warning. The btrfs_get_extent() is a bit complex, but trust me that it doesn't return NULLs and also if it did we would trigger the BUG_ON(!em) before the last return statement. Signed-off-by: Dan Carpenter

btrfs won't mount any more

2017-04-11 Thread Marc Haber
Hi, I have wrecked another btrfs file system, probably for good this time. It's a 80 GB filesystem from 2015, in my secondary notebook, on an encrypted SSD. The btrfs holds the root filesystem and the rest of the system as well. I have a cronjob that makes snapshots of the system directories dai