Re: Monitoring for disk failures

2014-01-28 Thread Anand Jain
Alin, [bug] its messy when missing device reappears after its been replaced in RAID1 I am aware of it and working on it. I also reported a more critical bug earlier as below. [bug] its messy when missing device reappears after its been replaced in RAID1 We see IO errors when disk

Re: Monitoring for disk failures

2014-01-28 Thread Anand Jain
Without actually trying it here... I believe by default that'd update only when there was an I/O error. Did you try btrfs filesystem show --all-devices? That scans differently. That will show missing with its own probes but kernel does not know that disk is missing. If that doesn't

Re: [PATCH] btrfs-progs: update INSTALL file

2014-01-28 Thread Anand Jain
Yep. I had the same opinion as well but was hesitant to make wide changes unless understand the backaground. Now would try to re-write as a whole and review will help. Thanks, Anand On 01/28/2014 02:27 AM, David Sterba wrote: On Wed, Jan 22, 2014 at 02:19:37PM +0800, Anand Jain wrote:

Re: [PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-28 Thread Anand Jain
You may have covered this but its not explicit. Could you write few lines on whats wrong with the current TREE SEARCH and how V2 is helping. Thanks, Anand On 01/27/2014 09:28 PM, Gerhard Heift wrote: This patch series adds a new ioctl TREE_SEARCH_V2 with which we could store the results

[PATCH] Btrfs: fix memory leaks on walking backrefs failure

2014-01-28 Thread Wang Shilong
When walking backrefs, we may iterate every inode's extent and add/merge them into ulist, and the caller will free memory from ulist. However, if we fail to allocate inode's extents element memory or ulist_add() fail to allocate memory, we won't add allocated memory into ulist, and the caller

[PATCH v2] Btrfs: allocate ulist with a slab allocator

2014-01-28 Thread Wang Shilong
Walking backrefs is heavily relying on ulist, so it is better to allocate ulist with a slab allocator especially with autodefrag and quota enabled. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- v1-v2: address david's comments. --- fs/btrfs/super.c | 7 +++

Re: [PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-28 Thread Gerhard Heift
2014-01-28 Anand Jain anand.j...@oracle.com: You may have covered this but its not explicit. Could you write few lines on whats wrong with the current TREE SEARCH and how V2 is helping. I wrote a program, which basically calculates a checksum over a files content checksums. First I got the

Re: [PATCH] xfstests: add test for btrfs incremental send data corruption

2014-01-28 Thread Josef Bacik
On 01/27/2014 08:40 PM, Filipe David Borba Manana wrote: Btrfs incremental send had an issue where it would detect a non-existent file hole and then overwrite the file section that hole covers with zeroes, overriding file data that it shouldn't. The respective btrfs kernel patch that fixed

Re: [PATCH] xfstests: test btrfs send issue with non-aligned clone operations

2014-01-28 Thread Josef Bacik
On 01/11/2014 04:36 PM, Filipe David Borba Manana wrote: Test for an issue in btrfs send where it sent clone operations to user space with a range (offset + length) that was not aligned with the block size. This caused the btrfs receive command to send such clone operations to the ioctl clone

Re: [PATCH v2] Btrfs: do not use extent commit root for sending

2014-01-28 Thread Josef Bacik
On 01/14/2014 11:26 AM, Wang Shilong wrote: From: Wang Shilong wangsl.f...@cn.fujitsu.com Now we have kicked off transaction from btrfs send, it is not safe that we use extent commit root to search. I happended to catch this problem when running sending and snapshot in my desktop. I'm

[PATCH] xfstests: make test btrfs/025 more robust

2014-01-28 Thread Filipe David Borba Manana
Make the test btrfs/025 not depend on the output of the btrfs tools subvolume, send, receive and filesystem commands output. The output of these commands has changed several times in the past, and it can change again in the future. Therefore just test for failure/success and not for the exact

Re: [PATCH V2 1/4] Btrfs: filter the ordered extents that has been logged

2014-01-28 Thread Josef Bacik
On 01/14/2014 07:31 AM, Miao Xie wrote: Signed-off-by: Miao Xie mi...@cn.fujitsu.com --- fs/btrfs/ordered-data.c |2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c index 69582d5..e4c3d56 100644 ---

Re: [PATCH] xfstests: make test btrfs/025 more robust

2014-01-28 Thread Josef Bacik
On 01/28/2014 09:46 AM, Filipe David Borba Manana wrote: Make the test btrfs/025 not depend on the output of the btrfs tools subvolume, send, receive and filesystem commands output. The output of these commands has changed several times in the past, and it can change again in the future.

Re: [PATCH V2 3/4] Btrfs: don't mix the ordered extents of all files together during logging the inodes

2014-01-28 Thread Josef Bacik
On 01/14/2014 07:31 AM, Miao Xie wrote: There was a problem in the old code: If we failed to log the csum, we would free all the ordered extents in the log list including those ordered extents that were logged successfully, it would make the log committer not to wait for the completion of the

Re: [PATCH v2] Btrfs: do not use extent commit root for sending

2014-01-28 Thread Wang Shilong
Hello Josef, On 01/14/2014 11:26 AM, Wang Shilong wrote: From: Wang Shilong wangsl.f...@cn.fujitsu.com Now we have kicked off transaction from btrfs send, it is not safe that we use extent commit root to search. I happended to catch this problem when running sending and snapshot in my

Re: [PATCH] btrfs: add simple debugfs interface

2014-01-28 Thread Josef Bacik
On 01/15/2014 08:25 PM, David Sterba wrote: Help during debugging to export various interesting infromation and tunables without the need of extra mount options or ioctls. Usage: * declare your variable in sysfs.h, and include where you need it * define the variable in sysfs.c and make it

Re: [PATCH] btrfs: Return EXDEV for cross file system snapshot

2014-01-28 Thread Josef Bacik
On 01/08/2014 05:46 AM, Kusanagi Kouichi wrote: EXDEV seems an appropriate error if an operation fails bacause it crosses file system boundaries. Signed-off-by: Kusanagi Kouichi sl...@ac.auone-net.jp --- fs/btrfs/ioctl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git

Re: [PATCH v3 1/2] Btrfs: rework ulist with list+rb_tree

2014-01-28 Thread Josef Bacik
On 01/27/2014 04:59 AM, Wang Shilong wrote: We are really suffering from now ulist's implementation, some developers gave their try, and i just gave some of my ideas for things: 1. use list+rb_tree instead of arrary+rb_tree 2. add cur_list to iterator rather than ulist structure. 3.

Re: [PATCH v3 1/2] Btrfs: rework ulist with list+rb_tree

2014-01-28 Thread Wang Shilong
Hello Josef, On 01/27/2014 04:59 AM, Wang Shilong wrote: We are really suffering from now ulist's implementation, some developers gave their try, and i just gave some of my ideas for things: 1. use list+rb_tree instead of arrary+rb_tree 2. add cur_list to iterator rather than ulist

Re: [PATCH V2 2/4] Btrfs: don't get the lock when adding a csum into a ordered extent

2014-01-28 Thread Josef Bacik
On 01/14/2014 07:31 AM, Miao Xie wrote: We are sure that: - one ordered extent just has one csum calculation worker, and no one access the csum list during the csum calculation except the worker. - we don't change the list and free the csum until no one reference to the ordered extent So

Re: [PATCH v3 1/2] Btrfs: rework ulist with list+rb_tree

2014-01-28 Thread Josef Bacik
On 01/28/2014 11:03 AM, Wang Shilong wrote: Hello Josef, On 01/27/2014 04:59 AM, Wang Shilong wrote: We are really suffering from now ulist's implementation, some developers gave their try, and i just gave some of my ideas for things: 1. use list+rb_tree instead of arrary+rb_tree 2.

[PATCH v4 2/2] Btrfs: do not export ulist functions

2014-01-28 Thread Wang Shilong
From: Wang Shilong wangsl.f...@cn.fujitsu.com There are not any users that use ulist except Btrfs,don't export them. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com Reviewed-by: David Sterba dste...@suse.cz --- v3-v4: none v2-v3: none v1-v2: make ulist_fini static --- fs/btrfs/ulist.c |

[PATCH v4 1/2 ] Btrfs: rework ulist with list+rb_tree

2014-01-28 Thread Wang Shilong
From: Wang Shilong wangsl.f...@cn.fujitsu.com We are really suffering from now ulist's implementation, some developers gave their try, and i just gave some of my ideas for things: 1. use list+rb_tree instead of arrary+rb_tree 2. add cur_list to iterator rather than ulist structure. 3. add

Re: [PATCH v3 1/2] Btrfs: rework ulist with list+rb_tree

2014-01-28 Thread Wang Shilong
On 01/28/2014 11:03 AM, Wang Shilong wrote: Hello Josef, On 01/27/2014 04:59 AM, Wang Shilong wrote: We are really suffering from now ulist's implementation, some developers gave their try, and i just gave some of my ideas for things: 1. use list+rb_tree instead of arrary+rb_tree

Re: Snapshots – noob questions

2014-01-28 Thread Martin
On 27/01/14 18:44, Chris Murphy wrote: On Jan 27, 2014, at 6:53 AM, KC impacto...@googlemail.com wrote: 3. If I make a snapshot of / and there are some separate partitions mounted under /mnt/ or /home/, will snapshot skip them? A snapshot of a subvolume only snapshots that subvolume, not

[PATCH] xfstests: change btrfs/029's golden output and it's filtering

2014-01-28 Thread Josef Bacik
I noticed while testing a different mkfs option that btrfs/029 was failing because it was getting the extra output from our mkfs.btrfs. After I fixed that I was still failing because my version of cp will spit out the source and destination files, not just the destination file. So redirect

Re: Snapshots – noob questions

2014-01-28 Thread Hugo Mills
On Tue, Jan 28, 2014 at 06:39:45PM +, Martin wrote: On 27/01/14 18:44, Chris Murphy wrote: On Jan 27, 2014, at 6:53 AM, KC impacto...@googlemail.com wrote: 3. If I make a snapshot of / and there are some separate partitions mounted under /mnt/ or /home/, will snapshot skip them?

Re: Snapshots – noob questions

2014-01-28 Thread Chris Murphy
On Jan 28, 2014, at 12:45 PM, KC impacto...@googlemail.com wrote: On 27/01/14 19:44, Chris Murphy wrote: On Jan 27, 2014, at 6:53 AM, KC impacto...@googlemail.com wrote: 3. If I make a snapshot of / and there are some separate partitions mounted under /mnt/ or /home/, will snapshot skip

Re: Rapid memory exhaustion during normal operation

2014-01-28 Thread Duncan
Dan Merillat posted on Sat, 25 Jan 2014 16:47:35 -0500 as excerpted: I'm trying to track this down - this started happening without changing the kernel in use, so probably a corrupted filesystem. The symptoms are that all memory is suddenly used by no apparent source. OOM killer is invoked

Re: [PATCH V2 3/4] Btrfs: don't mix the ordered extents of all files together during logging the inodes

2014-01-28 Thread Miao Xie
On Tue, 28 Jan 2014 09:55:58 -0500, Josef Bacik wrote: On 01/14/2014 07:31 AM, Miao Xie wrote: There was a problem in the old code: If we failed to log the csum, we would free all the ordered extents in the log list including those ordered extents that were logged successfully, it would

Re: [PATCH V2 2/4] Btrfs: don't get the lock when adding a csum into a ordered extent

2014-01-28 Thread Miao Xie
On tue, 28 Jan 2014 11:00:54 -0500, Josef Bacik wrote: On 01/14/2014 07:31 AM, Miao Xie wrote: We are sure that: - one ordered extent just has one csum calculation worker, and no one access the csum list during the csum calculation except the worker. - we don't change the list and

Re: Rapid memory exhaustion during normal operation

2014-01-28 Thread Chris Murphy
On Jan 25, 2014, at 2:47 PM, Dan Merillat dan.meril...@gmail.com wrote: [ 1219.366168] ntpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [ 1219.366270] CPU: 1 PID: 5479 Comm: ntpd Not tainted 3.12.8-00848-g97f15f1 #2 This and the whole call track don't have anything in

Re: Rapid memory exhaustion during normal operation

2014-01-28 Thread Duncan
Chris Murphy posted on Tue, 28 Jan 2014 20:57:45 -0700 as excerpted: On Jan 25, 2014, at 2:47 PM, Dan Merillat dan.meril...@gmail.com wrote: [ 1219.366168] ntpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [ 1219.366270] CPU: 1 PID: 5479 Comm: ntpd Not tainted