deleting files and qgroups

2014-01-27 Thread Russell Coker
(sysadm_t:s0-s0:c0.c1023)root@play:/tmp# btrfs qgroup limit 20m /tmp (sysadm_t:s0-s0:c0.c1023)root@play:/tmp# du -h test 931Mtest (sysadm_t:s0-s0:c0.c1023)root@play:/tmp# rm test rm: cannot remove 'test': Disk quota exceeded (sysadm_t:s0-s0:c0.c1023)root@play:/tmp# echo -n test -bash: test:

[PATCH] dump device list as seen by the kernel

2014-01-27 Thread Anand Jain
For debugging we need a way to understand what kernel thinks about the devices under its control mainly when device disappear and reappear. The current btrfs-progs sub commands wouldn't help because the output from the kernel is greatly fine tuned before printing on the terminal and in some

[PATCH] btrfs: introduce BTRFS_IOC_GET_DEVS

2014-01-27 Thread Anand Jain
The user land progs needs a simple way to read the raw list of disks and its parameters as btrfs kernel understands it. This patch will introduce BTRFS_IOC_GET_DEVS which dumps every thing under fs_devices. As of now btrfs-devlist uses this ioctl. In the long run this ioctl would help to

[PATCH] btrfs-progs: introduce btrfs-devlist

2014-01-27 Thread Anand Jain
This is a small (debug) program to dump the device list in the raw format from the btrfs kernel. here I use ioctl which was introduced in the below kernel patch btrfs: introduce BTRFS_IOC_GET_DEVS Signed-off-by: Anand Jain anand.j...@oracle.com --- .gitignore |1 + Makefile|

Re: [PATCH] btrfs: introduce BTRFS_IOC_GET_DEVS

2014-01-27 Thread Hugo Mills
On Mon, Jan 27, 2014 at 04:52:50PM +0800, Anand Jain wrote: The user land progs needs a simple way to read the raw list of disks and its parameters as btrfs kernel understands it. This patch will introduce BTRFS_IOC_GET_DEVS which dumps every thing under fs_devices. As of now

Re: deleting files and qgroups

2014-01-27 Thread Duncan
Russell Coker posted on Mon, 27 Jan 2014 18:59:16 +1100 as excerpted: As an aside, is it expected that a SE Linux Play Machine (with hostile root user) should survive with BTRFS? I'm preparing a new Play Machine which will run BTRFS and use qgroups, are there any bugs or missing features I

Re: [PATCH] btrfs: introduce BTRFS_IOC_GET_DEVS

2014-01-27 Thread Anand Jain
On 01/27/2014 04:47 PM, Hugo Mills wrote: On Mon, Jan 27, 2014 at 04:52:50PM +0800, Anand Jain wrote: The user land progs needs a simple way to read the raw list of disks and its parameters as btrfs kernel understands it. This patch will introduce BTRFS_IOC_GET_DEVS which dumps every thing

Btrfs send 4-5 times slower than rsync on local

2014-01-27 Thread Konstantinos Skarlatos
Hello, i am using btrfs send to copy a snapshot to another btrfs filesystem on the same machine, and it has a maximum speed of 30-35MByte/sec. Incredibly rsync is much faster, at 120-140MB/sec. Source btrfs is a 5x2TB raid 0 and target is 1x4TB. mount options:

[PATCH v3 1/2] Btrfs: rework ulist with list+rb_tree

2014-01-27 Thread Wang Shilong
We are really suffering from now ulist's implementation, some developers gave their try, and i just gave some of my ideas for things: 1. use list+rb_tree instead of arrary+rb_tree 2. add cur_list to iterator rather than ulist structure. 3. add seqnum into every node when they are added, this

[PATCH] Btrfs: allocate ulist with a slab allocator

2014-01-27 Thread Wang Shilong
Walking backrefs is heavily relying on ulist, so it is better to allocate ulist with a slab allocator especially with autodefrag and quota enabled. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- fs/btrfs/super.c | 7 +++ fs/btrfs/ulist.c | 22 --

[PATCH v3 2/2] Btrfs: do not export ulist functions

2014-01-27 Thread Wang Shilong
There are not any users that use ulist except Btrfs,don't export them. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com --- v2-v3: none. v1-v2: make ulist_fini() static since it is not called anywhere. --- fs/btrfs/ulist.c | 10 +- fs/btrfs/ulist.h | 1 - 2 files changed, 1

Monitoring for disk failures

2014-01-27 Thread Alin Dobre
Hi all! I am trying to create a very simple script that would alert in case of disk failures from a RAID Btrfs. Digging into the code, I have noticed that the btrfs fi sh command should display a warning if there is a missing disk. However, testing in a Qemu, I used drive_del via QMP to remove a

Re: Monitoring for disk failures

2014-01-27 Thread Duncan
Alin Dobre posted on Mon, 27 Jan 2014 11:43:33 + as excerpted: I am trying to create a very simple script that would alert in case of disk failures from a RAID Btrfs. Digging into the code, I have noticed that the btrfs fi sh command should display a warning if there is a missing disk.

[PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-27 Thread Gerhard Heift
This patch series adds a new ioctl TREE_SEARCH_V2 with which we could store the results in a varying buffer. Now even items larger than 3992 bytes or a large amount of items can be returned. I have a few questions: Which value should I assign to TREE_SEARCH_V2? Should we limit the buffer

[PATCH RFCv2 1/6] btrfs: search_ioctl accepts varying buffer

2014-01-27 Thread Gerhard Heift
rewrite search_ioctl to accept a buffer with varying size Signed-off-by: Gerhard Heift gerh...@heift.name --- fs/btrfs/ioctl.c | 19 --- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 3970f32..be4c780 100644 ---

[PATCH RFCv2 2/6] btrfs: search_ioctl rejects unused setted values

2014-01-27 Thread Gerhard Heift
To prevent unexpectet values in the unused fields of the search key fail early. Otherwise future extensions would break the behavior of the search if current implementations in userspace set them to values other than zero. Signed-off-by: Gerhard Heift gerh...@heift.name --- fs/btrfs/ioctl.c | 3

[PATCH RFCv2 6/6] btrfs: in tree_search extent buffer lifetime

2014-01-27 Thread Gerhard Heift
Instead of allocting and releasing a buffer for each leaf, which will be visited during a search, allocate (and expand) a buffer for the whole search. This saves a few allocations and deallocations during larger searchs, which visits more leafs. Signed-off-by: Gerhard Heift gerh...@heift.name ---

[PATCH RFCv2 5/6] btrfs: search_ioctl: direct copy to userspace

2014-01-27 Thread Gerhard Heift
By copying each found item seperatly to userspace, we only need a small buffer in the kernel. This allows to run a large search inside of a single call. Signed-off-by: Gerhard Heift gerh...@heift.name --- fs/btrfs/ioctl.c | 107 --- 1 file

[PATCH RFCv2 3/6] btrfs: copy_to_sk returns EOVERFLOW for too small buffer

2014-01-27 Thread Gerhard Heift
In copy_to_sk, if an item is too large for the given buffer, it now returns -EOVERFLOW instead of copying a search_header with len = 0. For backward compatibility for the first item it still copies such a header to the buffer, but not any other following items, which could have fitted.

[PATCH RFCv2 4/6] btrfs: new ioctl TREE_SEARCH_V2

2014-01-27 Thread Gerhard Heift
This new ioctl call allows the user to supply a buffer of varying size in which a tree search can store its results. This is much more flexible if you want to receive items which are larger than the current fixed buffer of 3992 bytes or if you want to fetch mor item at once. Currently the buffer

Snapshots – noob questions

2014-01-27 Thread KC
I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me. To make things easier, I tried to make most of them Yes/No questions: 1. When creating the filesystem, I only made btrfs

Re: Snapshots – noob questions

2014-01-27 Thread Hugo Mills
On Mon, Jan 27, 2014 at 02:53:36PM +0100, KC wrote: I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me. To make things easier, I tried to make most of them Yes/No questions:

Re: Snapshots – noob questions

2014-01-27 Thread Jim Salter
Well... you can't /create/ snapshots on different partitions / cloud storage / whatever, but you can certainly /send/ them there once you've created them. Generally the best way to do this kind of thing is with incremental replication of snapshots to another btrfs filesystem, which may be

Re: btrfs and ECC RAM

2014-01-27 Thread Calvin Walton
On Fri, 2014-01-24 at 17:45 -0700, Chris Murphy wrote: On Jan 20, 2014, at 9:08 AM, George Mitchell geo...@chinilu.com wrote: After reading the recent posts on this topic I am beginning to think there is some real confusion between check sums and parity. Yes, I often see conventional

Re: [PATCH v3 2/2] Btrfs: do not export ulist functions

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 05:59:48PM +0800, Wang Shilong wrote: There are not any users that use ulist except Btrfs,don't export them. Signed-off-by: Wang Shilong wangsl.f...@cn.fujitsu.com Agreed, Reviewed-by: David Sterba dste...@suse.cz -- To unsubscribe from this list: send the line

Re: [PATCH] Btrfs: allocate ulist with a slab allocator

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 05:59:49PM +0800, Wang Shilong wrote: --- a/fs/btrfs/ulist.c +++ b/fs/btrfs/ulist.c @@ -156,6 +157,23 @@ static int ulist_rbtree_insert(struct ulist *ulist, struct ulist_node *ins) return 0; } +int __init btrfs_ulist_init(void) +{ + btrfs_ulist_cache

Re: btrfs and ECC RAM

2014-01-27 Thread Chris Murphy
On Jan 27, 2014, at 9:08 AM, Calvin Walton calvin.wal...@kepstin.ca wrote: On Fri, 2014-01-24 at 17:45 -0700, Chris Murphy wrote: On Jan 20, 2014, at 9:08 AM, George Mitchell geo...@chinilu.com wrote: After reading the recent posts on this topic I am beginning to think there is some real

Re: [PATCH] Btrfs: add a reschedule point in btrfs_find_all_roots()

2014-01-27 Thread David Sterba
On Sun, Jan 26, 2014 at 10:32:18PM +0800, Wang Shilong wrote: I can easily trigger the following warnings when enabling quota in my virtual machine(running Opensuse), Steps are firstly creating a subvolume full of fragment extents, and then create many snapshots (500 in my test case). Good

Re: [PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 02:28:26PM +0100, Gerhard Heift wrote: This patch series adds a new ioctl TREE_SEARCH_V2 with which we could store the results in a varying buffer. Now even items larger than 3992 bytes or a large amount of items can be returned. I have a few questions: Which

Re: [PATCH RFCv2 1/6] btrfs: search_ioctl accepts varying buffer

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 02:28:27PM +0100, Gerhard Heift wrote: @@ -1931,17 +1931,22 @@ overflow: } static noinline int search_ioctl(struct inode *inode, - struct btrfs_ioctl_search_args *args) + struct btrfs_ioctl_search_key *sk,

Re: [PATCH RFCv2 2/6] btrfs: search_ioctl rejects unused setted values

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 02:28:28PM +0100, Gerhard Heift wrote: To prevent unexpectet values in the unused fields of the search key fail early. Otherwise future extensions would break the behavior of the search if current implementations in userspace set them to values other than zero.

Re: [PATCH RFCv2 4/6] btrfs: new ioctl TREE_SEARCH_V2

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 02:28:30PM +0100, Gerhard Heift wrote: --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -2032,6 +2032,52 @@ static noinline int btrfs_ioctl_tree_search(struct file *file, return ret; } +static noinline int btrfs_ioctl_tree_search_v2(struct file *file, +

Re: [PATCH] btrfs-progs: update INSTALL file

2014-01-27 Thread David Sterba
On Wed, Jan 22, 2014 at 02:19:37PM +0800, Anand Jain wrote: --- a/INSTALL +++ b/INSTALL @@ -12,7 +12,8 @@ complete: modprobe libcrc32c insmod btrfs.ko -The Btrfs utility programs require libuuid to build. This can be found +The Btrfs utility programs (btrfs-progs) require libattr zlib

Re: [PATCH RFCv2 5/6] btrfs: search_ioctl: direct copy to userspace

2014-01-27 Thread David Sterba
On Mon, Jan 27, 2014 at 02:28:31PM +0100, Gerhard Heift wrote: By copying each found item seperatly to userspace, we only need a small buffer in the kernel. This allows to run a large search inside of a single call. Signed-off-by: Gerhard Heift gerh...@heift.name --- fs/btrfs/ioctl.c | 107

Re: Snapshots – noob questions

2014-01-27 Thread Chris Murphy
On Jan 27, 2014, at 7:03 AM, Hugo Mills h...@carfax.org.uk wrote: On Mon, Jan 27, 2014 at 02:53:36PM +0100, KC wrote: I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some questions, and would appreciate if someone could clear them for me. To

Re: Snapshots – noob questions

2014-01-27 Thread Hugo Mills
On Mon, Jan 27, 2014 at 11:35:03AM -0700, Chris Murphy wrote: On Jan 27, 2014, at 7:03 AM, Hugo Mills h...@carfax.org.uk wrote: On Mon, Jan 27, 2014 at 02:53:36PM +0100, KC wrote: I have been trying to understand how snapshots work (in BRTFS and in general), but I still have some

Re: Snapshots – noob questions

2014-01-27 Thread Chris Murphy
On Jan 27, 2014, at 6:53 AM, KC impacto...@googlemail.com wrote: 3. If I make a snapshot of / and there are some separate partitions mounted under /mnt/ or /home/, will snapshot skip them? A snapshot of a subvolume only snapshots that subvolume, not other subvolumes contained in it or

Re: Snapshots – noob questions

2014-01-27 Thread Hugo Mills
On Mon, Jan 27, 2014 at 11:44:24AM -0700, Chris Murphy wrote: On Jan 27, 2014, at 6:53 AM, KC impacto...@googlemail.com wrote: 4. If I have a snapshot of /, can I completely erase this partition and later restore it in full form that snapshot, or do snapshots work only if a limited

Re: [PATCH RFCv2 2/6] btrfs: search_ioctl rejects unused setted values

2014-01-27 Thread Martin Steigerwald
Am Montag, 27. Januar 2014, 14:28:28 schrieb Gerhard Heift: To prevent unexpectet values in the unused fields of the search key fail unexpected early. Otherwise future extensions would break the behavior of the search if current implementations in userspace set them to values other than zero.

Re: [PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-27 Thread Goffredo Baroncelli
Hi, On 2014-01-27 14:28, Gerhard Heift wrote: This patch series adds a new ioctl TREE_SEARCH_V2 with which we could store the results in a varying buffer. Now even items larger than 3992 bytes or a large amount of items can be returned. One of the main strangeness of the current

Re: [PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-27 Thread Hugo Mills
On Mon, Jan 27, 2014 at 08:10:52PM +0100, Goffredo Baroncelli wrote: Hi, On 2014-01-27 14:28, Gerhard Heift wrote: This patch series adds a new ioctl TREE_SEARCH_V2 with which we could store the results in a varying buffer. Now even items larger than 3992 bytes or a large amount of

Re: Snapshots – noob questions

2014-01-27 Thread Chris Murphy
On Jan 27, 2014, at 11:50 AM, Hugo Mills h...@carfax.org.uk wrote: On Mon, Jan 27, 2014 at 11:44:24AM -0700, Chris Murphy wrote: On Jan 27, 2014, at 6:53 AM, KC impacto...@googlemail.com wrote: 4. If I have a snapshot of /, can I completely erase this partition and later restore it in

Re: [PATCH] btrfs-progs: update INSTALL file

2014-01-27 Thread Daniel Cegiełka
2014-01-22 Anand Jain anand.j...@oracle.com: with the changes that has happened since last time it was updated Signed-off-by: Anand Jain anand.j...@oracle.com --- INSTALL |6 ++ 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/INSTALL b/INSTALL index 8ead607..a86878a

Re: [PATCH RFCv2] new ioctl TREE_SEARCH_V2

2014-01-27 Thread Goffredo Baroncelli
Hi Hugo, On 2014-01-27 20:31, Hugo Mills wrote: One of the main strangeness of the current TREE_SEARCH ioctl, which I found is the fact that the search was not in a rectangular region, but in a linear region. This is due the fact that after a ioctl call, we have to set the min_* fields

Re: [PATCH] btrfs-progs: update INSTALL file

2014-01-27 Thread Darrick J. Wong
On Mon, Jan 27, 2014 at 10:21:32PM +0100, Daniel Cegiełka wrote: 2014-01-22 Anand Jain anand.j...@oracle.com: with the changes that has happened since last time it was updated Signed-off-by: Anand Jain anand.j...@oracle.com --- INSTALL |6 ++ 1 files changed, 2 insertions(+),

Re: [PATCH RFCv2 2/6] btrfs: search_ioctl rejects unused setted values

2014-01-27 Thread Gerhard Heift
2014-01-27 David Sterba dste...@suse.cz: On Mon, Jan 27, 2014 at 02:28:28PM +0100, Gerhard Heift wrote: To prevent unexpectet values in the unused fields of the search key fail early. Otherwise future extensions would break the behavior of the search if current implementations in userspace

Re: [PATCH RFCv2 4/6] btrfs: new ioctl TREE_SEARCH_V2

2014-01-27 Thread Gerhard Heift
2014-01-27 David Sterba dste...@suse.cz: On Mon, Jan 27, 2014 at 02:28:30PM +0100, Gerhard Heift wrote: --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -2032,6 +2032,52 @@ static noinline int btrfs_ioctl_tree_search(struct file *file, return ret; } +static noinline int

Re: [PATCH RFCv2 5/6] btrfs: search_ioctl: direct copy to userspace

2014-01-27 Thread Gerhard Heift
2014-01-27 David Sterba dste...@suse.cz: On Mon, Jan 27, 2014 at 02:28:31PM +0100, Gerhard Heift wrote: By copying each found item seperatly to userspace, we only need a small buffer in the kernel. This allows to run a large search inside of a single call. Signed-off-by: Gerhard Heift

Re: Btrfs send 4-5 times slower than rsync on local

2014-01-27 Thread Josef Bacik
On 01/27/2014 04:14 AM, Konstantinos Skarlatos wrote: Hello, i am using btrfs send to copy a snapshot to another btrfs filesystem on the same machine, and it has a maximum speed of 30-35MByte/sec. Incredibly rsync is much faster, at 120-140MB/sec. Source btrfs is a 5x2TB raid 0 and target is

[PATCH] Btrfs: fix send file hole detection leading to data corruption

2014-01-27 Thread Filipe David Borba Manana
There was a case where file hole detection was incorrect and it would cause an incremental send to override a section of a file with zeroes. This happened in the case where between the last leaf we processed which contained a file extent item for our current inode and the leaf we're currently are

[PATCH] xfstests: add test for btrfs incremental send data corruption

2014-01-27 Thread Filipe David Borba Manana
Btrfs incremental send had an issue where it would detect a non-existent file hole and then overwrite the file section that hole covers with zeroes, overriding file data that it shouldn't. The respective btrfs kernel patch that fixed this issue is titled: Btrfs: fix send file hole detection

Re: [PATCH] Btrfs: only process as many file extents as there are refs

2014-01-27 Thread Wang Shilong
On 01/25/2014 03:24 AM, Josef Bacik wrote: The backref walking code will search down to the key it is looking for and then proceed to walk _all_ of the extents on the file until it hits the end. This is suboptimal with large files, we only need to look for as many extents as we have references