Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Lu Fengqi]

At 06/03/2016 10:02 PM, Josef Bacik wrote:

On 06/01/2016 01:48 AM, Lu Fengqi wrote:

Only in the case of different root_id or different object_id,
check_shared
identified extent as the shared. However, If a extent was referred by
different offset of same file, it should also be identified as shared.
In addition, check_shared's loop scale is at least  n^3, so if a extent
has too many references,  even causes soft hang up.

First, add all delayed_ref to the ref_tree and calculate the unqiue_refs,
if the unique_refs is greater than one, return BACKREF_FOUND_SHARED.
Then individually add the  on-disk reference(inline/keyed) to the
ref_tree
and calculate the unique_refs of the ref_tree to check if the unique_refs
is greater than one.Because once there are two references to return
SHARED, so the time complexity is close to the constant.

Reported-by: Tsutomu Itoh 
Signed-off-by: Lu Fengqi 


This is a lot of work for just wanting to know if something is shared.
Instead lets adjust this slightly.  Instead of passing down a
root_objectid/inum and noticing this and returned shared, add a new way
to iterate refs.  Currently we go gather all the refs and then do the
iterate dance, which is what takes so long.  So instead add another
helper that calls the provided function every time it has a match, and
then we can pass in whatever context we want, and we return when
something matches.  This way we don't have all this extra accounting,
and we're no longer passing root_objectid/inum around and testing for
some magic scenario.  Thanks,

Josef





With this patch, we can quickly find extent that has more than one 
reference(delayed, inline and keyed) and return SHARED immediately. 
However, for indirect refs, we have to continue to resolve indirect refs 
to their parent bytenr, and check if this parent bytenr is shared. So, 
the original function is necessary. The original refs list reduces the 
efficiency of search, so maybe we can use rb_tree to replace it for 
optimize the original function in the furture. And, we just want to 
solve the problem of check_shared now.


--
Thanks,
Lu


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Josef Bacik]

On 06/01/2016 01:48 AM, Lu Fengqi wrote:

Only in the case of different root_id or different object_id, check_shared
identified extent as the shared. However, If a extent was referred by
different offset of same file, it should also be identified as shared.
In addition, check_shared's loop scale is at least  n^3, so if a extent
has too many references,  even causes soft hang up.

First, add all delayed_ref to the ref_tree and calculate the unqiue_refs,
if the unique_refs is greater than one, return BACKREF_FOUND_SHARED.
Then individually add the  on-disk reference(inline/keyed) to the ref_tree
and calculate the unique_refs of the ref_tree to check if the unique_refs
is greater than one.Because once there are two references to return
SHARED, so the time complexity is close to the constant.

Reported-by: Tsutomu Itoh 
Signed-off-by: Lu Fengqi 


This is a lot of work for just wanting to know if something is shared. 
Instead lets adjust this slightly.  Instead of passing down a 
root_objectid/inum and noticing this and returned shared, add a new way 
to iterate refs.  Currently we go gather all the refs and then do the 
iterate dance, which is what takes so long.  So instead add another 
helper that calls the provided function every time it has a match, and 
then we can pass in whatever context we want, and we return when 
something matches.  This way we don't have all this extra accounting, 
and we're no longer passing root_objectid/inum around and testing for 
some magic scenario.  Thanks,


Josef

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Mark Fasheh]

On Thu, Jun 02, 2016 at 02:17:40PM -0700, Mark Fasheh wrote:
> On Thu, Jun 02, 2016 at 04:56:06PM -0400, Jeff Mahoney wrote:
> > On 6/2/16 3:08 PM, Mark Fasheh wrote:
> > > On Thu, Jun 02, 2016 at 07:07:32PM +0200, David Sterba wrote:
> > >> On Wed, Jun 01, 2016 at 02:15:22PM -0700, Mark Fasheh wrote:
> >  +/* dynamically allocate and initialize a ref_root */
> >  +static struct ref_root *ref_root_alloc(void)
> >  +{
> >  +  struct ref_root *ref_tree;
> >  +
> >  +  ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
> > >>>
> > >>> I'm pretty sure we want GFP_NOFS here.
> > >>
> > >> Then please explain to me why/where the reasoning below is wrong:
> > > 
> > > The general reasoning of when to use GFP_NOFS below is fine, I don't
> > > disagree with that at all. If there is no way a recursion back into btrfs
> > > can't happen at that allocation site then we can use GFP_KERNEL.
> > > 
> > > That said, have you closely audited this path? Does the allocation happen
> > > completely outside any locks that might be shared with the writeout path?
> > > What happens if we have to do writeout of the inode being fiemapped in 
> > > order
> > > to allocate space? If the answer to all my questions is "there is no way
> > > this can deadlock" then by all means, we should use GFP_KERNEL. Otherwise
> > > GFP_NOFS is a sensible guard against possible future deadlocks.
> > 
> > This is exactly the situation we discussed at LSF/MM this year.  The MM
> > folks are pushing back because the fs folks tend to use GFP_NOFS as a
> > talisman.  The audit needs to happen, otherwise that last sentence is
> > another talisman.
> 
> There's nothing here I disagree with. I'm not seeing a strong technical
> justification, which is what I want (being called from an ioctl means
> nothing in this case).

A small amount of searching shows me that extent_fiemap() does
lock_extent_bits() and writepage_delalloc() also calls lock_extent_bits()
(via find_lock_delalloc_range()).

I'm no expert on the extent locking but that seems pretty deadlocky to me.
--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Mark Fasheh]

On Thu, Jun 02, 2016 at 04:56:06PM -0400, Jeff Mahoney wrote:
> On 6/2/16 3:08 PM, Mark Fasheh wrote:
> > On Thu, Jun 02, 2016 at 07:07:32PM +0200, David Sterba wrote:
> >> On Wed, Jun 01, 2016 at 02:15:22PM -0700, Mark Fasheh wrote:
>  +/* dynamically allocate and initialize a ref_root */
>  +static struct ref_root *ref_root_alloc(void)
>  +{
>  +struct ref_root *ref_tree;
>  +
>  +ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
> >>>
> >>> I'm pretty sure we want GFP_NOFS here.
> >>
> >> Then please explain to me why/where the reasoning below is wrong:
> > 
> > The general reasoning of when to use GFP_NOFS below is fine, I don't
> > disagree with that at all. If there is no way a recursion back into btrfs
> > can't happen at that allocation site then we can use GFP_KERNEL.
> > 
> > That said, have you closely audited this path? Does the allocation happen
> > completely outside any locks that might be shared with the writeout path?
> > What happens if we have to do writeout of the inode being fiemapped in order
> > to allocate space? If the answer to all my questions is "there is no way
> > this can deadlock" then by all means, we should use GFP_KERNEL. Otherwise
> > GFP_NOFS is a sensible guard against possible future deadlocks.
> 
> This is exactly the situation we discussed at LSF/MM this year.  The MM
> folks are pushing back because the fs folks tend to use GFP_NOFS as a
> talisman.  The audit needs to happen, otherwise that last sentence is
> another talisman.

There's nothing here I disagree with. I'm not seeing a strong technical
justification, which is what I want (being called from an ioctl means
nothing in this case).
--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Jeff Mahoney]

On 6/2/16 3:08 PM, Mark Fasheh wrote:
> On Thu, Jun 02, 2016 at 07:07:32PM +0200, David Sterba wrote:
>> On Wed, Jun 01, 2016 at 02:15:22PM -0700, Mark Fasheh wrote:
 +/* dynamically allocate and initialize a ref_root */
 +static struct ref_root *ref_root_alloc(void)
 +{
 +  struct ref_root *ref_tree;
 +
 +  ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
>>>
>>> I'm pretty sure we want GFP_NOFS here.
>>
>> Then please explain to me why/where the reasoning below is wrong:
> 
> The general reasoning of when to use GFP_NOFS below is fine, I don't
> disagree with that at all. If there is no way a recursion back into btrfs
> can't happen at that allocation site then we can use GFP_KERNEL.
> 
> That said, have you closely audited this path? Does the allocation happen
> completely outside any locks that might be shared with the writeout path?
> What happens if we have to do writeout of the inode being fiemapped in order
> to allocate space? If the answer to all my questions is "there is no way
> this can deadlock" then by all means, we should use GFP_KERNEL. Otherwise
> GFP_NOFS is a sensible guard against possible future deadlocks.

This is exactly the situation we discussed at LSF/MM this year.  The MM
folks are pushing back because the fs folks tend to use GFP_NOFS as a
talisman.  The audit needs to happen, otherwise that last sentence is
another talisman.

-Jeff

-- 
Jeff Mahoney
SUSE Labs



signature.asc
Description: OpenPGP digital signature

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Mark Fasheh]

On Thu, Jun 02, 2016 at 01:46:27PM +0800, Lu Fengqi wrote:
> 
> At 06/02/2016 05:15 AM, Mark Fasheh wrote:
> >Thanks for trying to fix this problem, comments below.
> >
> >On Wed, Jun 01, 2016 at 01:48:05PM +0800, Lu Fengqi wrote:
> >>Only in the case of different root_id or different object_id, check_shared
> >>identified extent as the shared. However, If a extent was referred by
> >>different offset of same file, it should also be identified as shared.
> >>In addition, check_shared's loop scale is at least  n^3, so if a extent
> >>has too many references,  even causes soft hang up.
> >>
> >>First, add all delayed_ref to the ref_tree and calculate the unqiue_refs,
> >>if the unique_refs is greater than one, return BACKREF_FOUND_SHARED.
> >>Then individually add the  on-disk reference(inline/keyed) to the ref_tree
> >>and calculate the unique_refs of the ref_tree to check if the unique_refs
> >>is greater than one.Because once there are two references to return
> >>SHARED, so the time complexity is close to the constant.
> >Constant time in the best case, but still n^3 in the worst case right? I'm
> >not complaining btw, I just want to be sure we're not over promising  :)
> Only in case of a large number of delayed_ref, the worst case time
> complexity will be n^2*logn. Otherwise, it will be constant even if
> there are many on-disk references.

Ahh ok so it's driven more by the # of delayed refs. That makes sense,
thanks.


> >>@@ -34,6 +35,253 @@ struct extent_inode_elem {
> >>struct extent_inode_elem *next;
> >>  };
> >>
> >>+/*
> >>+ * ref_root is used as the root of the ref tree that hold a collection
> >>+ * of unique references.
> >>+ */
> >>+struct ref_root {
> >>+   struct rb_root rb_root;
> >>+
> >>+   /*
> >>+* the unique_refs represents the number of ref_nodes with a positive
> >>+* count stored in the tree. Even if a ref_node(the count is greater
> >>+* than one) is added, the unique_refs will only increase one.
> >>+*/
> >>+   unsigned int unique_refs;
> >>+};
> >>+
> >>+/* ref_node is used to store a unique reference to the ref tree. */
> >>+struct ref_node {
> >>+   struct rb_node rb_node;
> >>+
> >>+   /* for NORMAL_REF, otherwise all these fields should be set to 0 */
> >>+   u64 root_id;
> >>+   u64 object_id;
> >>+   u64 offset;
> >>+
> >>+   /* for SHARED_REF, otherwise parent field should be set to 0 */
> >>+   u64 parent;
> >>+
> >>+   /* ref to the ref_mod of btrfs_delayed_ref_node(delayed-ref.h) */
> >>+   int ref_mod;
> >>+};
> >Why are we mirroring so much of the backref structures here? It seems like
> >we're just throwing layers on top of layers. Can't we modify the backref
> >structures and code to handle whatever small amount of unique accounting you
> >must do?
> The original structure(struct __prelim_ref) store reference in list,
> and I have to perform many search operations that not suitable for
> list. However, if I modify the original structure, it would require
> a lot of rework. So I just want to fix fiemap with this patch. If
> necessary, we can use this structure to replace the original
> structure later.

Well there's room for an rb_node on that structure so we can solve the 'it
only uses a list' problem trivially. I definitely understand your reluctance
to modify the backref code, but to me that just sounds like we need someone
who is familiar with that code to review your work and provide advice when
needed.

Otherwise, I believe my point holds. If there's some technical reason why
this is a bad idea, that's a different story. So far though this just seems
like a situation where we need some extra review from the primary
developers. I cc'd Josef in the hopes he could shed some light for us.


> >>+/* dynamically allocate and initialize a ref_root */
> >>+static struct ref_root *ref_root_alloc(void)
> >>+{
> >>+   struct ref_root *ref_tree;
> >>+
> >>+   ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
> >I'm pretty sure we want GFP_NOFS here.
> Yes, perhaps you're right.
> >Because there's no need to narrow the allocation constraints. GFP_NOFS
> >is necessary when the caller is on a critical path that must not recurse
> >back to the filesystem through the allocation (ie. if the allocator
> >decides to free some memory and tries tro write dirty data). FIEMAP is
> >called from an ioctl.
> But David seems to have a different point of view with you, so I
> would like to ask for his advice again.

Sounds good, hopefully David and I can figure it out  :)

Thanks again Lu,
--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Mark Fasheh]

On Thu, Jun 02, 2016 at 07:07:32PM +0200, David Sterba wrote:
> On Wed, Jun 01, 2016 at 02:15:22PM -0700, Mark Fasheh wrote:
> > > +/* dynamically allocate and initialize a ref_root */
> > > +static struct ref_root *ref_root_alloc(void)
> > > +{
> > > + struct ref_root *ref_tree;
> > > +
> > > + ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
> > 
> > I'm pretty sure we want GFP_NOFS here.
> 
> Then please explain to me why/where the reasoning below is wrong:

The general reasoning of when to use GFP_NOFS below is fine, I don't
disagree with that at all. If there is no way a recursion back into btrfs
can't happen at that allocation site then we can use GFP_KERNEL.

That said, have you closely audited this path? Does the allocation happen
completely outside any locks that might be shared with the writeout path?
What happens if we have to do writeout of the inode being fiemapped in order
to allocate space? If the answer to all my questions is "there is no way
this can deadlock" then by all means, we should use GFP_KERNEL. Otherwise
GFP_NOFS is a sensible guard against possible future deadlocks.
--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[David Sterba]

On Wed, Jun 01, 2016 at 02:15:22PM -0700, Mark Fasheh wrote:
> > +/* dynamically allocate and initialize a ref_root */
> > +static struct ref_root *ref_root_alloc(void)
> > +{
> > +   struct ref_root *ref_tree;
> > +
> > +   ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
> 
> I'm pretty sure we want GFP_NOFS here.

Then please explain to me why/where the reasoning below is wrong:

> > Because there's no need to narrow the allocation constraints. GFP_NOFS
> > is necessary when the caller is on a critical path that must not recurse
> > back to the filesystem through the allocation (ie. if the allocator
> > decides to free some memory and tries tro write dirty data). FIEMAP is
> > called from an ioctl.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Lu Fengqi]

At 06/02/2016 05:15 AM, Mark Fasheh wrote:

Thanks for trying to fix this problem, comments below.

On Wed, Jun 01, 2016 at 01:48:05PM +0800, Lu Fengqi wrote:

Only in the case of different root_id or different object_id, check_shared
identified extent as the shared. However, If a extent was referred by
different offset of same file, it should also be identified as shared.
In addition, check_shared's loop scale is at least  n^3, so if a extent
has too many references,  even causes soft hang up.

First, add all delayed_ref to the ref_tree and calculate the unqiue_refs,
if the unique_refs is greater than one, return BACKREF_FOUND_SHARED.
Then individually add the  on-disk reference(inline/keyed) to the ref_tree
and calculate the unique_refs of the ref_tree to check if the unique_refs
is greater than one.Because once there are two references to return
SHARED, so the time complexity is close to the constant.

Constant time in the best case, but still n^3 in the worst case right? I'm
not complaining btw, I just want to be sure we're not over promising  :)
Only in case of a large number of delayed_ref, the worst case time 
complexity will be n^2*logn. Otherwise, it will be constant even if 
there are many on-disk references.

@@ -34,6 +35,253 @@ struct extent_inode_elem {
struct extent_inode_elem *next;
  };

+/*
+ * ref_root is used as the root of the ref tree that hold a collection
+ * of unique references.
+ */
+struct ref_root {
+   struct rb_root rb_root;
+
+   /*
+* the unique_refs represents the number of ref_nodes with a positive
+* count stored in the tree. Even if a ref_node(the count is greater
+* than one) is added, the unique_refs will only increase one.
+*/
+   unsigned int unique_refs;
+};
+
+/* ref_node is used to store a unique reference to the ref tree. */
+struct ref_node {
+   struct rb_node rb_node;
+
+   /* for NORMAL_REF, otherwise all these fields should be set to 0 */
+   u64 root_id;
+   u64 object_id;
+   u64 offset;
+
+   /* for SHARED_REF, otherwise parent field should be set to 0 */
+   u64 parent;
+
+   /* ref to the ref_mod of btrfs_delayed_ref_node(delayed-ref.h) */
+   int ref_mod;
+};

Why are we mirroring so much of the backref structures here? It seems like
we're just throwing layers on top of layers. Can't we modify the backref
structures and code to handle whatever small amount of unique accounting you
must do?
The original structure(struct __prelim_ref) store reference in list, and 
I have to perform many search operations that not suitable for list. 
However, if I modify the original structure, it would require a lot of 
rework. So I just want to fix fiemap with this patch. If necessary, we 
can use this structure to replace the original structure later.



+/* dynamically allocate and initialize a ref_root */
+static struct ref_root *ref_root_alloc(void)
+{
+   struct ref_root *ref_tree;
+
+   ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);

I'm pretty sure we want GFP_NOFS here.

Yes, perhaps you're right.

Because there's no need to narrow the allocation constraints. GFP_NOFS
is necessary when the caller is on a critical path that must not recurse
back to the filesystem through the allocation (ie. if the allocator
decides to free some memory and tries tro write dirty data). FIEMAP is
called from an ioctl.
But David seems to have a different point of view with you, so I would 
like to ask for his advice again.

+
+/*
+ * search ref_node with (root_id, object_id, offset, parent) in the tree
+ *
+ * if found, the pointer of the ref_node will be returned;
+ * if not found, NULL will be returned and pos will point to the rb_node for
+ * insert, pos_parent will point to pos'parent for insert;
+*/
+static struct ref_node *__ref_tree_search(struct ref_root *ref_tree,
+ struct rb_node ***pos,
+ struct rb_node **pos_parent,
+ u64 root_id, u64 object_id,
+ u64 offset, u64 parent)
+{
+   struct ref_node *cur = NULL;
+
+   *pos = _tree->rb_root.rb_node;
+
+   while (**pos) {
+   *pos_parent = **pos;
+   cur = rb_entry(*pos_parent, struct ref_node, rb_node);
+
+   if (cur->root_id < root_id) {
+   *pos = &(**pos)->rb_right;
+   continue;
+   } else if (cur->root_id > root_id) {
+   *pos = &(**pos)->rb_left;
+   continue;
+   }
+
+   if (cur->object_id < object_id) {
+   *pos = &(**pos)->rb_right;
+   continue;
+   } else if (cur->object_id > object_id) {
+   *pos = &(**pos)->rb_left;
+   continue;
+   }
+
+   if (cur->offset < offset) {
+  

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Mark Fasheh]

On Wed, Jun 01, 2016 at 02:15:22PM -0700, Mark Fasheh wrote:
> > +static int ref_tree_add(struct ref_root *ref_tree, u64 root_id, u64 
> > object_id,
> > +   u64 offset, u64 parent, int count)
> > +{
> > +   struct ref_node *node = NULL;
> > +   struct rb_node **pos = NULL;
> > +   struct rb_node *pos_parent = NULL;
> > +   int origin_count;
> > +   int ret;
> > +
> > +   if (!count)
> > +   return 0;
> > +
> > +   node = __ref_tree_search(ref_tree, , _parent, root_id,
> > +object_id, offset, parent);
> > +   if (node == NULL) {
> > +   node = kmalloc(sizeof(*node), GFP_KERNEL);
> > +   if (!node)
> > +   return -ENOMEM;
> > +
> > +   node->root_id = root_id;
> > +   node->object_id = object_id;
> > +   node->offset = offset;
> > +   node->parent = parent;
> > +   node->ref_mod = count;
> > +
> > +   ret = ref_tree_insert(ref_tree, pos, pos_parent, node);
> > +   ASSERT(!ret);
> > +   if (ret) {
> > +   kfree(node);
> > +   return ret;
> > +   }
> 
> If you put the open coded comparisons into their own function, then it
> should be trivial to call that here and we can have a 'standard' looking
> rbtree insert instead of this custom version. See the rbtree.h header for an
> example.

oops, I meant Documentation/rbtree.txt instead of include/linux/rbtree.h.
--Mark

--
Mark Fasheh
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Mark Fasheh]

Thanks for trying to fix this problem, comments below.

On Wed, Jun 01, 2016 at 01:48:05PM +0800, Lu Fengqi wrote:
> Only in the case of different root_id or different object_id, check_shared
> identified extent as the shared. However, If a extent was referred by
> different offset of same file, it should also be identified as shared.
> In addition, check_shared's loop scale is at least  n^3, so if a extent
> has too many references,  even causes soft hang up.
> 
> First, add all delayed_ref to the ref_tree and calculate the unqiue_refs,
> if the unique_refs is greater than one, return BACKREF_FOUND_SHARED.
> Then individually add the  on-disk reference(inline/keyed) to the ref_tree
> and calculate the unique_refs of the ref_tree to check if the unique_refs
> is greater than one.Because once there are two references to return
> SHARED, so the time complexity is close to the constant.

Constant time in the best case, but still n^3 in the worst case right? I'm
not complaining btw, I just want to be sure we're not over promising  :)


> @@ -34,6 +35,253 @@ struct extent_inode_elem {
>   struct extent_inode_elem *next;
>  };
>  
> +/*
> + * ref_root is used as the root of the ref tree that hold a collection
> + * of unique references.
> + */
> +struct ref_root {
> + struct rb_root rb_root;
> +
> + /*
> +  * the unique_refs represents the number of ref_nodes with a positive
> +  * count stored in the tree. Even if a ref_node(the count is greater
> +  * than one) is added, the unique_refs will only increase one.
> +  */
> + unsigned int unique_refs;
> +};
> +
> +/* ref_node is used to store a unique reference to the ref tree. */
> +struct ref_node {
> + struct rb_node rb_node;
> +
> + /* for NORMAL_REF, otherwise all these fields should be set to 0 */
> + u64 root_id;
> + u64 object_id;
> + u64 offset;
> +
> + /* for SHARED_REF, otherwise parent field should be set to 0 */
> + u64 parent;
> +
> + /* ref to the ref_mod of btrfs_delayed_ref_node(delayed-ref.h) */
> + int ref_mod;
> +};

Why are we mirroring so much of the backref structures here? It seems like
we're just throwing layers on top of layers. Can't we modify the backref
structures and code to handle whatever small amount of unique accounting you
must do?



> +/* dynamically allocate and initialize a ref_root */
> +static struct ref_root *ref_root_alloc(void)
> +{
> + struct ref_root *ref_tree;
> +
> + ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);

I'm pretty sure we want GFP_NOFS here.


> +
> +/*
> + * search ref_node with (root_id, object_id, offset, parent) in the tree
> + *
> + * if found, the pointer of the ref_node will be returned;
> + * if not found, NULL will be returned and pos will point to the rb_node for
> + * insert, pos_parent will point to pos'parent for insert;
> +*/
> +static struct ref_node *__ref_tree_search(struct ref_root *ref_tree,
> +   struct rb_node ***pos,
> +   struct rb_node **pos_parent,
> +   u64 root_id, u64 object_id,
> +   u64 offset, u64 parent)
> +{
> + struct ref_node *cur = NULL;
> +
> + *pos = _tree->rb_root.rb_node;
> +
> + while (**pos) {
> + *pos_parent = **pos;
> + cur = rb_entry(*pos_parent, struct ref_node, rb_node);
> +
> + if (cur->root_id < root_id) {
> + *pos = &(**pos)->rb_right;
> + continue;
> + } else if (cur->root_id > root_id) {
> + *pos = &(**pos)->rb_left;
> + continue;
> + }
> +
> + if (cur->object_id < object_id) {
> + *pos = &(**pos)->rb_right;
> + continue;
> + } else if (cur->object_id > object_id) {
> + *pos = &(**pos)->rb_left;
> + continue;
> + }
> +
> + if (cur->offset < offset) {
> + *pos = &(**pos)->rb_right;
> + continue;
> + } else if (cur->offset > offset) {
> + *pos = &(**pos)->rb_left;
> + continue;
> + }
> +
> + if (cur->parent < parent) {
> + *pos = &(**pos)->rb_right;
> + continue;
> + } else if (cur->parent > parent) {
> + *pos = &(**pos)->rb_left;
> + continue;
> + }

These 4 comparisons need to be in their own cmp() function - we want to
avoid open coded comparisons with an rbtree search. This is a pretty
standard way to handle it.


> +
> + return cur;
> + }
> +
> + return NULL;
> +}
> +
> +/*
> + * insert a ref_node to the ref tree
> + * @pos used for specifiy the position to insert
> + * @pos_parent for specifiy pos's parent
> + *
> + * 

[PATCH v2] btrfs: fix check_shared for fiemap ioctl

[Lu Fengqi]

Only in the case of different root_id or different object_id, check_shared
identified extent as the shared. However, If a extent was referred by
different offset of same file, it should also be identified as shared.
In addition, check_shared's loop scale is at least  n^3, so if a extent
has too many references,  even causes soft hang up.

First, add all delayed_ref to the ref_tree and calculate the unqiue_refs,
if the unique_refs is greater than one, return BACKREF_FOUND_SHARED.
Then individually add the  on-disk reference(inline/keyed) to the ref_tree
and calculate the unique_refs of the ref_tree to check if the unique_refs
is greater than one.Because once there are two references to return
SHARED, so the time complexity is close to the constant.

Reported-by: Tsutomu Itoh 
Signed-off-by: Lu Fengqi 
---
 fs/btrfs/backref.c   | 349 +--
 fs/btrfs/extent_io.c |  18 ++-
 2 files changed, 357 insertions(+), 10 deletions(-)

diff --git a/fs/btrfs/backref.c b/fs/btrfs/backref.c
index 80e8472..325a144 100644
--- a/fs/btrfs/backref.c
+++ b/fs/btrfs/backref.c
@@ -17,6 +17,7 @@
  */
 
 #include 
+#include 
 #include "ctree.h"
 #include "disk-io.h"
 #include "backref.h"
@@ -34,6 +35,253 @@ struct extent_inode_elem {
struct extent_inode_elem *next;
 };
 
+/*
+ * ref_root is used as the root of the ref tree that hold a collection
+ * of unique references.
+ */
+struct ref_root {
+   struct rb_root rb_root;
+
+   /*
+* the unique_refs represents the number of ref_nodes with a positive
+* count stored in the tree. Even if a ref_node(the count is greater
+* than one) is added, the unique_refs will only increase one.
+*/
+   unsigned int unique_refs;
+};
+
+/* ref_node is used to store a unique reference to the ref tree. */
+struct ref_node {
+   struct rb_node rb_node;
+
+   /* for NORMAL_REF, otherwise all these fields should be set to 0 */
+   u64 root_id;
+   u64 object_id;
+   u64 offset;
+
+   /* for SHARED_REF, otherwise parent field should be set to 0 */
+   u64 parent;
+
+   /* ref to the ref_mod of btrfs_delayed_ref_node(delayed-ref.h) */
+   int ref_mod;
+};
+
+/* dynamically allocate and initialize a ref_root */
+static struct ref_root *ref_root_alloc(void)
+{
+   struct ref_root *ref_tree;
+
+   ref_tree = kmalloc(sizeof(*ref_tree), GFP_KERNEL);
+   if (!ref_tree)
+   return NULL;
+
+   ref_tree->rb_root = RB_ROOT;
+   ref_tree->unique_refs = 0;
+
+   return ref_tree;
+}
+
+/* free all nodes in the ref tree, and reinit ref_root */
+static void ref_root_fini(struct ref_root *ref_tree)
+{
+   struct ref_node *node;
+   struct rb_node *next;
+
+   while ((next = rb_first(_tree->rb_root)) != NULL) {
+   node = rb_entry(next, struct ref_node, rb_node);
+   rb_erase(next, _tree->rb_root);
+   kfree(node);
+   }
+
+   ref_tree->rb_root = RB_ROOT;
+   ref_tree->unique_refs = 0;
+}
+
+/* free dynamically allocated ref_root */
+static void ref_root_free(struct ref_root *ref_tree)
+{
+   if (!ref_tree)
+   return;
+
+   ref_root_fini(ref_tree);
+   kfree(ref_tree);
+}
+
+/*
+ * search ref_node with (root_id, object_id, offset, parent) in the tree
+ *
+ * if found, the pointer of the ref_node will be returned;
+ * if not found, NULL will be returned and pos will point to the rb_node for
+ * insert, pos_parent will point to pos'parent for insert;
+*/
+static struct ref_node *__ref_tree_search(struct ref_root *ref_tree,
+ struct rb_node ***pos,
+ struct rb_node **pos_parent,
+ u64 root_id, u64 object_id,
+ u64 offset, u64 parent)
+{
+   struct ref_node *cur = NULL;
+
+   *pos = _tree->rb_root.rb_node;
+
+   while (**pos) {
+   *pos_parent = **pos;
+   cur = rb_entry(*pos_parent, struct ref_node, rb_node);
+
+   if (cur->root_id < root_id) {
+   *pos = &(**pos)->rb_right;
+   continue;
+   } else if (cur->root_id > root_id) {
+   *pos = &(**pos)->rb_left;
+   continue;
+   }
+
+   if (cur->object_id < object_id) {
+   *pos = &(**pos)->rb_right;
+   continue;
+   } else if (cur->object_id > object_id) {
+   *pos = &(**pos)->rb_left;
+   continue;
+   }
+
+   if (cur->offset < offset) {
+   *pos = &(**pos)->rb_right;
+   continue;
+   } else if (cur->offset > offset) {
+   *pos = &(**pos)->rb_left;
+