Re: btrfs hanging since 3.16-rc3 or so

2014-07-16 Thread Liu Bo
Hi, On Tue, Jul 15, 2014 at 10:47:01PM +0100, Hugo Mills wrote: cc linux-btrfs list On Tue, Jul 15, 2014 at 10:40:46PM +0900, Norbert Preining wrote: Dear all (please keep Cc) Since 3.16-rc3 or so I regularly get btrfs hanging in some transations. Usually during apt-get

Re: Unmountable btrfs filesystem - 'unable to find logical' / 'no mapping'

2014-07-16 Thread Duncan
Gareth Clay posted on Tue, 15 Jul 2014 14:35:22 +0100 as excerpted: I noticed yesterday that the mount points on my btrfs RAID1 filesystem had become read-only. On a reboot, the filesystem fails to mount. I wondered if someone here might be able offer any advice on how to recover (if

Re: [PATCH v2] Btrfs: fix abnormal long waiting in fsync

2014-07-16 Thread Miao Xie
On Wed, 16 Jul 2014 15:37:05 +0800, Liu Bo wrote: xfstests generic/127 detected this problem. With commit 7fc34a62ca4434a79c68e23e70ed26111b7a4cf8, now fsync will only flush data within the passed range. This is the cause of the above problem, -- btrfs's fsync has a stage called 'sync

[PATCH 1/2] Btrfs: fix wrong missing device counter decrease

2014-07-16 Thread Miao Xie
The missing devices are accounted by its own fs device, for example the missing devices in seed filesystem will be accounted by the fs device of the seed filesystem, not by the new filesystem which is based on the seed filesystem, so when we remove the missing device in the seed filesystem, we

[PATCH 2/2] Btrfs: fix wrong total device counter after removing a seed device

2014-07-16 Thread Miao Xie
The total device counters of all the descendant filesystem account the seed device, so after removing a seed device, we need decrease all those counters. Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- fs/btrfs/volumes.c | 12 1 file changed, 8 insertions(+), 4 deletions(-) diff

Re: Why does btrfs defrag work worse than making a copy of a file?

2014-07-16 Thread Sebastian Ochmann
On 16.07.2014 09:53, Liu Bo wrote: On Tue, Jul 15, 2014 at 11:17:26PM +0200, Sebastian Ochmann wrote: Hello, I have a VirtualBox hard drive image which is quite fragmented even after very light use; it is 1.6 GB in size and has around 5000 fragments (I'm using filefrag to determine the number

Re: btrfs hanging since 3.16-rc3 or so

2014-07-16 Thread Norbert Preining
Hi Liu, [PATCH] Btrfs: fix abnormal long waiting in fsync https://patchwork.kernel.org/patch/4564961/ Looks fine! And also is explains why aptitude was hanging as it does several fsync operations. I will have it running from now and report in case of problems. Thanks Norbert

[RFC PATCH V4 02/12] Btrfs: subpagesize-blocksize: Get rid of whole page writes.

2014-07-16 Thread Chandan Rajendra
This commit brings back functions that set/clear EXTENT_WRITEBACK bits. These are required to reliably clear PG_writeback page flag. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/extent_io.c | 134 +-- fs/btrfs/extent_io.h

[RFC PATCH V4 01/12] Btrfs: subpagesize-blocksize: Get rid of whole page reads.

2014-07-16 Thread Chandan Rajendra
Based on original patch from Aneesh Kumar K.V aneesh.ku...@linux.vnet.ibm.com bio_vec-{bv_offset, bv_len} cannot be relied upon by the end bio functions to track the file offset range operated on by the bio. Hence this patch adds two new members to 'struct btrfs_io_bio' to track the file offset

[RFC PATCH V4 03/12] Btrfs: subpagesize-blocksize: __btrfs_buffered_write: Reserve/release extents aligned to block size.

2014-07-16 Thread Chandan Rajendra
Currently, the code reserves/releases extents in multiples of PAGE_CACHE_SIZE units. Fix this. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/file.c | 32 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/fs/btrfs/file.c

[RFC PATCH V4 00/12] Btrfs: Subpagesize-blocksize: Get rid of whole page I/O.

2014-07-16 Thread Chandan Rajendra
This patchset continues with the work posted earlier at https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg34954.html. Changes from V1: 1. Remove usage of bio_vec-bv_{len,offset} in end_bio_extent_readpage() and end_bio_extent_writepage(). Changes from V2: 1. Get __extent_writepage()

[RFC PATCH V4 05/12] Btrfs: subpagesize-blocksize: Read tree blocks whose size is PAGE_CACHE_SIZE.

2014-07-16 Thread Chandan Rajendra
In the case of subpagesize-blocksize, this patch makes it possible to read only a single metadata block from the disk instead of all the metadata blocks that map into a page. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/disk-io.c | 45 -

[RFC PATCH V4 11/12] Btrfs: subpagesize-blocksize: btrfs_page_mkwrite: Reserve space in sectorsized units.

2014-07-16 Thread Chandan Rajendra
In subpagesize-blocksize scenario, if i_size occurs in a block which is not the last block in the page, then the space to be reserved should be calculated appropriately. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/inode.c | 33 ++--- 1 file

[RFC PATCH V4 04/12] Btrfs: subpagesize-blocksize: Define extent_buffer_head.

2014-07-16 Thread Chandan Rajendra
From: Chandra Seetharaman sekha...@us.ibm.com In order to handle multiple extent buffers per page, first we need to create a way to handle all the extent buffers that are attached to a page. This patch creates a new data structure 'struct extent_buffer_head', and moves fields that are common to

[RFC PATCH V4 10/12] Btrfs: subpagesize-blocksize: fallocate: Work with sectorsized units.

2014-07-16 Thread Chandan Rajendra
While at it, this commit changes btrfs_truncate_page() to truncate sectorsized blocks instead of pages. Hence the function has been renamed to btrfs_truncate_block(). Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/ctree.h | 2 +- fs/btrfs/file.c | 35

[RFC PATCH V4 12/12] Btrfs: subpagesize-blocksize: Search for all ordered extents that could span across a page.

2014-07-16 Thread Chandan Rajendra
In subpagesize-blocksize scenario it is not sufficient to search using the first byte of the page to make sure that there are no ordered extents present across the page. Fix this. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/extent_io.c | 3 ++- fs/btrfs/inode.c |

[RFC PATCH V4 06/12] Btrfs: subpagesize-blocksize: Write only dirty extent buffers belonging to a page

2014-07-16 Thread Chandan Rajendra
For the subpagesize-blocksize scenario, This patch adds the ability to write a single extent buffer to the disk. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/disk-io.c | 20 ++-- fs/btrfs/extent_io.c | 279 ++- 2 files

[RFC PATCH V4 07/12] Btrfs: subpagesize-blocksize: Allow mounting filesystems where sectorsize != PAGE_SIZE

2014-07-16 Thread Chandan Rajendra
From: Chandra Seetharaman sekha...@us.ibm.com This patch allows mounting filesystems with blocksize smaller than the PAGE_SIZE. Signed-off-by: Chandra Seetharaman sekha...@us.ibm.com Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/disk-io.c | 6 -- 1 file changed, 6

[RFC PATCH V4 08/12] Btrfs: subpagesize-blocksize: Compute and look up csums based on sectorsized blocks.

2014-07-16 Thread Chandan Rajendra
Checksums are applicable to sectorsize units. The current code uses bio-bv_len units to compute and look up checksums. This works on machines where sectorsize == PAGE_CACHE_SIZE. This patch makes the checksum computation and look up code to work with sectorsize units. Signed-off-by: Chandan

[RFC PATCH V4 09/12] Btrfs: subpagesize-blocksize: __extent_writepage: Write only dirty blocks of a page.

2014-07-16 Thread Chandan Rajendra
The code now loops across 'ordered extents' instead of 'extent maps' to figure out the dirty blocks of the page to be submitted for a write operation. Signed-off-by: Chandan Rajendra chan...@linux.vnet.ibm.com --- fs/btrfs/extent_io.c | 68 +--- 1

Re: btrfs is related to OOM death problems on my 8GB server with both 3.15.1 and 3.14?

2014-07-16 Thread Marc MERLIN
On Mon, Jul 14, 2014 at 09:24:11AM +0800, Qu Wenruo wrote: But generally, is there a tool to locate which kernel function allocated all that RAM that seems to get allocated and forgotten? This can be done by kernel memleak detection. Location: - Kernel hacking - Memory Debugging

RE: File server structure suggestion

2014-07-16 Thread Kyle Gates
Then there's raid10, which takes more drives and is faster, but is still limited to two mirrors. But while I haven't actually used raid10 myself, I do /not/ believe it's limited to pair-at-a-time additions. I believe it'll take, for instance five devices, just fine, staggering chunk

Re: [PATCH 1/1] Btrfs: fix sparse warning

2014-07-16 Thread Zach Brown
On Tue, Jul 15, 2014 at 09:17:17PM +0200, Fabian Frederick wrote: Fix the following sparse warning: fs/btrfs/send.c:518:51: warning: incorrect type in argument 2 (different address spaces) fs/btrfs/send.c:518:51:expected char const [noderef] asn:1*noident fs/btrfs/send.c:518:51:got

Re: [RFC 0/2] vfs / btrfs: add support for ustat()

2014-07-16 Thread Luis R. Rodriguez
On Tue, Jul 15, 2014 at 10:29:19PM -0700, Christoph Hellwig wrote: Isn't this the problem again the btrfs uses different assignments for st_dev than s_dev? I don't even want to think about a mess like this before that is fixed. As much as I'd like to see that happen based on discussions so

[PATCH] btrfs-progs: mkfs: don't zero extend small files

2014-07-16 Thread Zach Brown
mkfs can try to write outside of small devices. The zeroing code doesn't test the device size and runs before mkfs tests for small devices and exits. Testers experienced this as small regular files being extended as mkfs failed: $ truncate -s 1m /tmp/some-file $ strace -epwrite ./mkfs.btrfs

[RFC v2 1/2] fs/super.c: add new super block sub devices super_block_dev

2014-07-16 Thread Luis R. Rodriguez
From: Luis R. Rodriguez mcg...@suse.com Modern filesystems are using the get_anon_bdev() for internal notions of volumes, snapshots for a single super block but never exposing them directly to the VFS layer. While this works its leaves the VFS layer growing dumb over what filesystems are doing.

[RFC v2 0/2] vfs / btrfs: add support for ustat()

2014-07-16 Thread Luis R. Rodriguez
From: Luis R. Rodriguez mcg...@suse.com This makes the implementation simpler by stuffing the struct on the driver and just letting the driver iinsert it and remove it onto the sb list. This avoids the kzalloc() completely. Luis R. Rodriguez (2): fs/super.c: add new super block sub devices

[RFC v2 2/2] btrfs: use the new VFS super_block_dev

2014-07-16 Thread Luis R. Rodriguez
From: Luis R. Rodriguez mcg...@suse.com Use the new VFS layer struct super_block_dev instead of carrying the anonymous bdev's on our own. This makes the VFS layer aware of all of our anonymous dev's on the super block. Signed-off-by: Luis R. Rodriguez mcg...@suse.com --- fs/btrfs/ctree.h | 7

Is it safe to mount subvolumes of already-mounted volumes (even with different options)?

2014-07-16 Thread Sebastian Ochmann
Hello, I'm sharing a btrfs-formatted drive between multiple computers and each of the machines has a separate home directory on that drive. The root of the drive is mounted at /mnt/tray and the home directory for machine {hostname} is under /mnt/tray/Homes/{hostname}. Up until now, I have

Re: Is it safe to mount subvolumes of already-mounted volumes (even with different options)?

2014-07-16 Thread Chris Murphy
On Jul 16, 2014, at 4:18 PM, Sebastian Ochmann ochm...@informatik.uni-bonn.de wrote: Hello, I'm sharing a btrfs-formatted drive between multiple computers and each of the machines has a separate home directory on that drive. 2+ computers writing to the same block device? I don't see how

Re: [PATCH 1/2] Btrfs: fix wrong missing device counter decrease

2014-07-16 Thread Liu Bo
On Wed, Jul 16, 2014 at 06:38:01PM +0800, Miao Xie wrote: The missing devices are accounted by its own fs device, for example the missing devices in seed filesystem will be accounted by the fs device of the seed filesystem, not by the new filesystem which is based on the seed filesystem, so

[PATCH 1/4] btrfs-progs: adjust the return values for scrub

2014-07-16 Thread Gui Hecheng
o Return 0 to indicate success, when detected errors were corrected during scrubbing. P.s. This is also to facilitate scripting when return value is to be checked. o Warn the users if there are uncorrectable errors detected. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com ---

[PATCH 3/4] btrfs-progs: replace a confusing raw number with a macro

2014-07-16 Thread Gui Hecheng
The raw number 36 for the uuid string length is somewhat confusing, use a macro to define replace it. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com --- cmds-scrub.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cmds-scrub.c b/cmds-scrub.c index a604b25..03eb9ba

[PATCH 2/4] btrfs-progs: remove unnecessary judgment for fd in scrub

2014-07-16 Thread Gui Hecheng
The scrub_read_file function is always on a branch, which has (fd = 0), so there is not need to judgment the pasted in arg. Signed-off-by: Gui Hecheng guihc.f...@cn.fujitsu.com --- cmds-scrub.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/cmds-scrub.c b/cmds-scrub.c index

Re: [PATCH v2] Btrfs: fix abnormal long waiting in fsync

2014-07-16 Thread Liu Bo
On Wed, Jul 16, 2014 at 05:36:07PM +0800, Miao Xie wrote: On Wed, 16 Jul 2014 15:37:05 +0800, Liu Bo wrote: xfstests generic/127 detected this problem. With commit 7fc34a62ca4434a79c68e23e70ed26111b7a4cf8, now fsync will only flush data within the passed range. This is the cause of

Re: [PATCH 1/2] Btrfs: fix wrong missing device counter decrease

2014-07-16 Thread Miao Xie
On Thu, 17 Jul 2014 10:37:57 +0800, Liu Bo wrote: On Wed, Jul 16, 2014 at 06:38:01PM +0800, Miao Xie wrote: The missing devices are accounted by its own fs device, for example the missing devices in seed filesystem will be accounted by the fs device of the seed filesystem, not by the new

Re: Is it safe to mount subvolumes of already-mounted volumes (even with different options)?

2014-07-16 Thread Zhe Zhang
Hi Sebastian, I posted a similar question and got many helpful answers: https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg35047.html Basically, you cannot guarantee that the computing mounting /dev/sdx doesn't write to arbitrary addresses of /dev/sdxN as unallocated blocks and thus

Re: Why does btrfs defrag work worse than making a copy of a file?

2014-07-16 Thread Liu Bo
On Wed, Jul 16, 2014 at 12:50:30PM +0200, Sebastian Ochmann wrote: On 16.07.2014 09:53, Liu Bo wrote: On Tue, Jul 15, 2014 at 11:17:26PM +0200, Sebastian Ochmann wrote: Hello, I have a VirtualBox hard drive image which is quite fragmented even after very light use; it is 1.6 GB in size and

Re: [PATCH 2/2] Btrfs: fix wrong total device counter after removing a seed device

2014-07-16 Thread Miao Xie
Please ignore this patch, the fix is not right. Thanks Miao On Wed, 16 Jul 2014 18:38:37 +0800, Miao Xie wrote: The total device counters of all the descendant filesystem account the seed device, so after removing a seed device, we need decrease all those counters. Signed-off-by: Miao Xie

[PATCH 6/6] Btrfs: fix wrong extent mapping for DirectIO

2014-07-16 Thread Wang Shilong
btrfs_next_leaf() will use current leaf's last key to search and then return a bigger one. So it may still return a file extent item that is smaller than expected value and we will get an overflow here for @em-len. This is easy to reproduce for Btrfs Direct writting, it did not cause any problem,

[PATCH 3/6] Btrfs: fix off-by-one in cow_file_range_inline()

2014-07-16 Thread Wang Shilong
Btrfs could still inline file data if its size is same as page size, so don't skip max value here. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- fs/btrfs/inode.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index

[PATCH 4/6] Btrfs: fix wrong max inline data size limit

2014-07-16 Thread Wang Shilong
inline data is stored from offset of @disk_bytenr in struct btrfs_file_extent_item. So substracting total size of struct btrfs_file_extent_item is wrong, fix it. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- fs/btrfs/ctree.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

[PATCH 1/6] Btrfs: fix wrong skipping compression for an inode

2014-07-16 Thread Wang Shilong
If a file's compression ratios is bad, we will set NOCOMPRESS flag for it, and it will skip compression for that inode next time. However, if we remount fs to COMPRESS_FORCE, it still should try if we could compress pages for that inode, this patch fix wrong check for this problem.

[PATCH 2/6] Btrfs: fall into nocompression codes quickly if possible

2014-07-16 Thread Wang Shilong
If flag NOCOMPRESS is set which means bad compression ratio, we could avoid call cow_file_range_async() for this case earlier. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- fs/btrfs/inode.c | 5 + 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/fs/btrfs/inode.c

[PATCH 5/6] Btrfs: fix wrong write range for filemap_fdatawrite_range()

2014-07-16 Thread Wang Shilong
filemap_fdatawrite_range() expect the third arg to be @end not @len, fix it. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- fs/btrfs/inode.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index e2c3d63..ed8b55c 100644 ---