Re: degraded permanent mount option

[Christophe Yayon]

I think you are right, i do not see any systemd message when degraded option is 
missing and have to remount manually with degraded.

It seems it is better to use mdadm for raid and btrfs over it as i understand. 
Even in recent kernel ?
I hav me to do some bench and compare...

Thanks

--
Christophe Yayon

> On 27 Jan 2018, at 07:43, Andrei Borzenkov  wrote:
> 
> 27.01.2018 09:40, Christophe Yayon пишет:
>> Hi, 
>> 
>> I am using archlinux with kernel 4.14, there is btrfs module in initrd.
>> In fstab root is mounted via UUID. As far as I know the UUID is the same
>> for all devices in raid array.
>> The system boot with no problem with degraded and only 1/2 root device.
> 
> Then your initramfs does not use systemd.
> 
>> --
>>  Christophe Yayon
>>  cyayon-l...@nbux.org
>> 
>> 
>> 
>>> On Sat, Jan 27, 2018, at 06:50, Andrei Borzenkov wrote:
>>> 26.01.2018 17:47, Christophe Yayon пишет:
 Hi Austin,
 
 Thanks for your answer. It was my opinion too as the "degraded"
 seems to be flagged as "Mostly OK" on btrfs wiki status page. I am
 running Archlinux with recent kernel on all my servers (because of
 use of btrfs as my main filesystem, i need a recent kernel).> >
 Your idea to add a separate entry in grub.cfg with
 rootflags=degraded is attractive, i will do this...> >
 Just a last question, i thank that it was necessary to add
 "degraded" option in grub.cfg AND fstab to allow boot in degraded
 mode. I am not sure that only grub.cfg is sufficient...> > Yesterday, i 
 have done some test and boot a a system with only 1 of
 2 drive in my root raid1 array. No problem with systemd,>
>>> Are you using systemd in your initramfs (whatever
>>> implementation you are> using)? I just tested with dracut using systemd 
>>> dracut module and it
>>> does not work - it hangs forever waiting for device. Of course,
>>> there is> no way to abort it and go into command line ...
>>> 
>>> Oh, wait - what device names are you using? I'm using mount by
>>> UUID and> this is where the problem starts - /dev/disk/by-uuid/xxx will
>>> not appear> unless all devices have been seen once ...
>>> 
>>> ... and it still does not work even if I change it to root=/dev/sda1
>>> explicitly because sda1 will *not* be announced as "present" to
>>> systemd> until all devices have been seen once ...
>>> 
>>> So no, it does not work with systemd *in initramfs*. Absolutely.
>> 
>> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: degraded permanent mount option

[Andrei Borzenkov]

27.01.2018 09:40, Christophe Yayon пишет:
> Hi, 
> 
> I am using archlinux with kernel 4.14, there is btrfs module in initrd.
> In fstab root is mounted via UUID. As far as I know the UUID is the same
> for all devices in raid array.
> The system boot with no problem with degraded and only 1/2 root device.

Then your initramfs does not use systemd.

> --
>   Christophe Yayon
>   cyayon-l...@nbux.org
> 
> 
> 
> On Sat, Jan 27, 2018, at 06:50, Andrei Borzenkov wrote:
>> 26.01.2018 17:47, Christophe Yayon пишет:
>>> Hi Austin,
>>>
>>> Thanks for your answer. It was my opinion too as the "degraded"
>>> seems to be flagged as "Mostly OK" on btrfs wiki status page. I am
>>> running Archlinux with recent kernel on all my servers (because of
>>> use of btrfs as my main filesystem, i need a recent kernel).> >
>>> Your idea to add a separate entry in grub.cfg with
>>> rootflags=degraded is attractive, i will do this...> >
>>> Just a last question, i thank that it was necessary to add
>>> "degraded" option in grub.cfg AND fstab to allow boot in degraded
>>> mode. I am not sure that only grub.cfg is sufficient...> > Yesterday, i 
>>> have done some test and boot a a system with only 1 of
>>> 2 drive in my root raid1 array. No problem with systemd,>
>> Are you using systemd in your initramfs (whatever
>> implementation you are> using)? I just tested with dracut using systemd 
>> dracut module and it
>> does not work - it hangs forever waiting for device. Of course,
>> there is> no way to abort it and go into command line ...
>>
>> Oh, wait - what device names are you using? I'm using mount by
>> UUID and> this is where the problem starts - /dev/disk/by-uuid/xxx will
>> not appear> unless all devices have been seen once ...
>>
>> ... and it still does not work even if I change it to root=/dev/sda1
>> explicitly because sda1 will *not* be announced as "present" to
>> systemd> until all devices have been seen once ...
>>
>> So no, it does not work with systemd *in initramfs*. Absolutely.
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] btrfs: Fix UAF

[Nikolay Borisov]

Commit 4fde46f0cc71 ("Btrfs: free the stale device") introduced
btrfs_free_stale_device which iterates the device lists for all
registered btrfs filesystems and deletes those devices which aren't
mounted. In a btrfs_devices structure has only 1 device attached to it
and it is unused then btrfs_free_stale_devices will proceed to also
free the btrfs_fs_devices struct itself. Currently this leads to a UAF
since list_for_each_entry will try to perform a check on the already-
freed memory to see if it has to terminated the loop.

The fix is to use 'break' when we know we are freeing the current
fs_devs.

Fixes: 4fde46f0cc71 ("Btrfs: free the stale device")
Signed-off-by: Nikolay Borisov 
---
 fs/btrfs/volumes.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index f7147740b68e..c3ab55336ee0 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -645,6 +645,7 @@ static void btrfs_free_stale_devices(const char *path,
btrfs_sysfs_remove_fsid(fs_devs);
list_del(_devs->list);
free_fs_devices(fs_devs);
+   break;
} else {
fs_devs->num_devices--;
list_del(>dev_list);
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: degraded permanent mount option

[Andrei Borzenkov]

26.01.2018 17:47, Christophe Yayon пишет:
> Hi Austin,
> 
> Thanks for your answer. It was my opinion too as the "degraded" seems to be 
> flagged as "Mostly OK" on btrfs wiki status page. I am running Archlinux with 
> recent kernel on all my servers (because of use of btrfs as my main 
> filesystem, i need a recent kernel).
> 
> Your idea to add a separate entry in grub.cfg with rootflags=degraded is 
> attractive, i will do this...
> 
> Just a last question, i thank that it was necessary to add "degraded" option 
> in grub.cfg AND fstab to allow boot in degraded mode. I am not sure that only 
> grub.cfg is sufficient... 
> Yesterday, i have done some test and boot a a system with only 1 of 2 drive 
> in my root raid1 array. No problem with systemd,

Are you using systemd in your initramfs (whatever implementation you are
using)? I just tested with dracut using systemd dracut module and it
does not work - it hangs forever waiting for device. Of course, there is
no way to abort it and go into command line ...

Oh, wait - what device names are you using? I'm using mount by UUID and
this is where the problem starts - /dev/disk/by-uuid/xxx will not appear
unless all devices have been seen once ...

... and it still does not work even if I change it to root=/dev/sda1
explicitly because sda1 will *not* be announced as "present" to systemd
until all devices have been seen once ...

So no, it does not work with systemd *in initramfs*. Absolutely.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 11/26] libbtrfsutil: add btrfs_util_create_snapshot()

[Omar Sandoval]

On Sat, Jan 27, 2018 at 01:00:58PM +0800, Qu Wenruo wrote:
> 
> 
> On 2018年01月27日 03:46, Omar Sandoval wrote:
> > On Fri, Jan 26, 2018 at 08:31:06PM +0100, Goffredo Baroncelli wrote:
> >> On 01/26/2018 07:40 PM, Omar Sandoval wrote:
> >>> From: Omar Sandoval 
> >>
> >>
> >> Hi,
> >>
> >> this is a great work; only few comments:
> >> 1) I found not intuitive the naming of the function: i.e. you have 
> >>
> >> btrfs_util_create_snapshot()
> >> btrfs_util_f_create_snapshot()
> >>
> >> To me it seems more clear to have
> >>
> >> btrfs_util_create_snapshot()
> >> btrfs_util_create_snapshot_f()
> >>
> >> I think that it is better move the 'f' at the end: at the begin you have 
> >> the library "btrfs_util", in the middle you have the library function 
> >> 'create_snapshot', at the end there is the function variant ('f', because 
> >> it uses a file descriptor).
> >>
> >> This is my opinion, even tough there are both examples like you 
> >> (stat/fstat/lstat) and like my one (capt_get_fd/cap_get_file)...
> > 
> > Yup, I was going off of the fstat/fsync/etc. convention. I don't
> > particularly like, e.g., btrfs_create_snapshot_f(), but
> > btrfs_create_snapshot_fd() isn't so bad.
> 
> _fd() suffix sounds more reasonable to me too.
> 
> > 
> >> 2) I find the prefix 'btrfs_util_' a bit verbose. Why not a simple 
> >> 'btrfs_', even at the cost of a possible renaming of the conflicting 
> >> function in the current btrfs code.
> > 
> > That's a reasonable idea, I mostly wanted to avoid naming conflicts but
> > if this is the "one true Btrfs library" it shouldn't be a concern.
> 
> Unfortunately, at least there is also some planned work to bring a
> shared code base between kernel and btrfs-progs, which is also named
> libbtrfs, inspired by libxfs.

That's right, I forgot about that. There's definitely value in having a
distinction between btrfs_util_ (userspace interfaces) and btrfs_
(filesystem disk format).

> And depending on the respect of view, some developer may prefer the
> short btrfs_ prefix for libbtrfs, while other developers/users will
> definitely prefer btrfs_ prefix for libbtrfsutil.
> 
> What about shorted prefix like butil_ or btrutil_?

Those aren't very informative, I think sticking with btrfs_util_ is
fine, it's not that bad to type out.

> Thanks,
> Qu
> 
> > 
> > I'll wait a bit for people to bikeshed on the naming before I go and
> > rename everything, but I'm leaning towards the shorter name and
> > appending _fd instead of prepending f_.
> > 
> >> 3) regarding the btrfs_util_create_snapshot() function, I think that it 
> >> would be useful to add some more information:
> >> a) if used recursive is NOT atomic
> >> b) if used recursive, root capabilities are needed
> >>
> >> The same for the other functions: mark with a 'root required' tag all the 
> >> functions which require the root capabilities.
> > 
> > That's a great point, I'll document that.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 11/26] libbtrfsutil: add btrfs_util_create_snapshot()

[Qu Wenruo]

On 2018年01月27日 03:46, Omar Sandoval wrote:
> On Fri, Jan 26, 2018 at 08:31:06PM +0100, Goffredo Baroncelli wrote:
>> On 01/26/2018 07:40 PM, Omar Sandoval wrote:
>>> From: Omar Sandoval 
>>
>>
>> Hi,
>>
>> this is a great work; only few comments:
>> 1) I found not intuitive the naming of the function: i.e. you have 
>>
>> btrfs_util_create_snapshot()
>> btrfs_util_f_create_snapshot()
>>
>> To me it seems more clear to have
>>
>> btrfs_util_create_snapshot()
>> btrfs_util_create_snapshot_f()
>>
>> I think that it is better move the 'f' at the end: at the begin you have the 
>> library "btrfs_util", in the middle you have the library function 
>> 'create_snapshot', at the end there is the function variant ('f', because it 
>> uses a file descriptor).
>>
>> This is my opinion, even tough there are both examples like you 
>> (stat/fstat/lstat) and like my one (capt_get_fd/cap_get_file)...
> 
> Yup, I was going off of the fstat/fsync/etc. convention. I don't
> particularly like, e.g., btrfs_create_snapshot_f(), but
> btrfs_create_snapshot_fd() isn't so bad.

_fd() suffix sounds more reasonable to me too.

> 
>> 2) I find the prefix 'btrfs_util_' a bit verbose. Why not a simple 'btrfs_', 
>> even at the cost of a possible renaming of the conflicting function in the 
>> current btrfs code.
> 
> That's a reasonable idea, I mostly wanted to avoid naming conflicts but
> if this is the "one true Btrfs library" it shouldn't be a concern.

Unfortunately, at least there is also some planned work to bring a
shared code base between kernel and btrfs-progs, which is also named
libbtrfs, inspired by libxfs.

And depending on the respect of view, some developer may prefer the
short btrfs_ prefix for libbtrfs, while other developers/users will
definitely prefer btrfs_ prefix for libbtrfsutil.

What about shorted prefix like butil_ or btrutil_?

Thanks,
Qu

> 
> I'll wait a bit for people to bikeshed on the naming before I go and
> rename everything, but I'm leaning towards the shorter name and
> appending _fd instead of prepending f_.
> 
>> 3) regarding the btrfs_util_create_snapshot() function, I think that it 
>> would be useful to add some more information:
>> a) if used recursive is NOT atomic
>> b) if used recursive, root capabilities are needed
>>
>> The same for the other functions: mark with a 'root required' tag all the 
>> functions which require the root capabilities.
> 
> That's a great point, I'll document that.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] btrfs: print error if primary super block write fails

[Qu Wenruo]

On 2018年01月27日 06:47, Howard McLauchlan wrote:
> Presently, failing a primary super block write but succeeding in at
> least one super block write in general will appear to users as if
> nothing important went wrong. However, upon unmounting and re-mounting,
> the file system will be in a rolled back state. This was discovered
> with a BCC program that uses bpf_override_return() to fail super block
> writes.
> 
> This patch outputs an error clarifying that the primary super block
> write has failed, so users can expect potentially erroneous behaviour.
> It also forces wait_dev_supers() to return an error to its caller if
> the primary super block write fails.
> 
> Signed-off-by: Howard McLauchlan 
> ---
>  fs/btrfs/disk-io.c | 17 ++---
>  1 file changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
> index 5da18ebc9222..8f96e1e4c613 100644
> --- a/fs/btrfs/disk-io.c
> +++ b/fs/btrfs/disk-io.c
> @@ -3293,11 +3293,13 @@ static int write_dev_supers(struct btrfs_device 
> *device,
>   * Return number of errors when buffer head is not found or not marked up to
>   * date.
>   */
> -static int wait_dev_supers(struct btrfs_device *device, int max_mirrors)
> +static int wait_dev_supers(struct btrfs_fs_info *fs_info,

fs_info can be fetch from device->fs_info.

> +struct btrfs_device *device, int max_mirrors)
>  {
>   struct buffer_head *bh;
>   int i;
>   int errors = 0;
> + bool primary_failed = false;
>   u64 bytenr;
>  
>   if (max_mirrors == 0)
> @@ -3314,11 +3316,14 @@ static int wait_dev_supers(struct btrfs_device 
> *device, int max_mirrors)
> BTRFS_SUPER_INFO_SIZE);
>   if (!bh) {
>   errors++;
> + primary_failed = (i == 0) || primary_failed;
>   continue;
>   }
>   wait_on_buffer(bh);
> - if (!buffer_uptodate(bh))
> + if (!buffer_uptodate(bh)) {
>   errors++;
> + primary_failed = (i == 0) || primary_failed;
> + }
>  
>   /* drop our reference */
>   brelse(bh);
> @@ -3327,6 +3332,12 @@ static int wait_dev_supers(struct btrfs_device 
> *device, int max_mirrors)
>   brelse(bh);
>   }
>  
> + /* log error, force error return */
> + if (primary_failed) {
> + btrfs_err(fs_info, "error encountered writing primary super 
> block");

It would be better if device id can also be outputted.

Otherwise it looks good.

Thanks,
Qu

> + return -1;
> + }
> +
>   return errors < i ? 0 : -1;
>  }
>  
> @@ -3557,7 +3568,7 @@ int write_all_supers(struct btrfs_fs_info *fs_info, int 
> max_mirrors)
>   if (!dev->in_fs_metadata || !dev->writeable)
>   continue;
>  
> - ret = wait_dev_supers(dev, max_mirrors);
> + ret = wait_dev_supers(fs_info, dev, max_mirrors);
>   if (ret)
>   total_errors++;
>   }
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 01/26] btrfs-progs: get rid of undocumented qgroup inheritance options

[Qu Wenruo]

On 2018年01月27日 02:40, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> The -c option to subvol create and the -c and -x options to subvol
> snapshot have never been documented (and -x doesn't actually work
> because it's not in the getopt option string). The functionality is
> dubious and the kernel interface is being removed, so get rid of these.
> 

Similar patch is already submitted.

https://patchwork.kernel.org/patch/10125007/

Thanks,
Qu

> Signed-off-by: Omar Sandoval 
> ---
>  cmds-subvolume.c | 25 ++---
>  qgroup.c | 42 --
>  qgroup.h |  2 --
>  3 files changed, 2 insertions(+), 67 deletions(-)
> 
> diff --git a/cmds-subvolume.c b/cmds-subvolume.c
> index dc626a64..f8e34bc8 100644
> --- a/cmds-subvolume.c
> +++ b/cmds-subvolume.c
> @@ -129,18 +129,11 @@ static int cmd_subvol_create(int argc, char **argv)
>   DIR *dirstream = NULL;
>  
>   while (1) {
> - int c = getopt(argc, argv, "c:i:");
> + int c = getopt(argc, argv, "i:");
>   if (c < 0)
>   break;
>  
>   switch (c) {
> - case 'c':
> - res = qgroup_inherit_add_copy(, optarg, 0);
> - if (res) {
> - retval = res;
> - goto out;
> - }
> - break;
>   case 'i':
>   res = qgroup_inherit_add_group(, optarg);
>   if (res) {
> @@ -665,18 +658,11 @@ static int cmd_subvol_snapshot(int argc, char **argv)
>  
>   memset(, 0, sizeof(args));
>   while (1) {
> - int c = getopt(argc, argv, "c:i:r");
> + int c = getopt(argc, argv, "i:r");
>   if (c < 0)
>   break;
>  
>   switch (c) {
> - case 'c':
> - res = qgroup_inherit_add_copy(, optarg, 0);
> - if (res) {
> - retval = res;
> - goto out;
> - }
> - break;
>   case 'i':
>   res = qgroup_inherit_add_group(, optarg);
>   if (res) {
> @@ -687,13 +673,6 @@ static int cmd_subvol_snapshot(int argc, char **argv)
>   case 'r':
>   readonly = 1;
>   break;
> - case 'x':
> - res = qgroup_inherit_add_copy(, optarg, 1);
> - if (res) {
> - retval = res;
> - goto out;
> - }
> - break;
>   default:
>   usage(cmd_subvol_snapshot_usage);
>   }
> diff --git a/qgroup.c b/qgroup.c
> index 156825fd..e9158a26 100644
> --- a/qgroup.c
> +++ b/qgroup.c
> @@ -1310,45 +1310,3 @@ int qgroup_inherit_add_group(struct 
> btrfs_qgroup_inherit **inherit, char *arg)
>  
>   return 0;
>  }
> -
> -int qgroup_inherit_add_copy(struct btrfs_qgroup_inherit **inherit, char *arg,
> - int type)
> -{
> - int ret;
> - u64 qgroup_src;
> - u64 qgroup_dst;
> - char *p;
> - int pos = 0;
> -
> - p = strchr(arg, ':');
> - if (!p) {
> -bad:
> - error("invalid copy specification, missing separator :");
> - return -EINVAL;
> - }
> - *p = 0;
> - qgroup_src = parse_qgroupid(arg);
> - qgroup_dst = parse_qgroupid(p + 1);
> - *p = ':';
> -
> - if (!qgroup_src || !qgroup_dst)
> - goto bad;
> -
> - if (*inherit)
> - pos = (*inherit)->num_qgroups +
> -   (*inherit)->num_ref_copies * 2 * type;
> -
> - ret = qgroup_inherit_realloc(inherit, 2, pos);
> - if (ret)
> - return ret;
> -
> - (*inherit)->qgroups[pos++] = qgroup_src;
> - (*inherit)->qgroups[pos++] = qgroup_dst;
> -
> - if (!type)
> - ++(*inherit)->num_ref_copies;
> - else
> - ++(*inherit)->num_excl_copies;
> -
> - return 0;
> -}
> diff --git a/qgroup.h b/qgroup.h
> index 875fbdf3..bb6610d7 100644
> --- a/qgroup.h
> +++ b/qgroup.h
> @@ -92,7 +92,5 @@ int btrfs_qgroup_setup_comparer(struct 
> btrfs_qgroup_comparer_set **comp_set,
>   int is_descending);
>  int qgroup_inherit_size(struct btrfs_qgroup_inherit *p);
>  int qgroup_inherit_add_group(struct btrfs_qgroup_inherit **inherit, char 
> *arg);
> -int qgroup_inherit_add_copy(struct btrfs_qgroup_inherit **inherit, char *arg,
> - int type);
>  
>  #endif
> 



signature.asc
Description: OpenPGP digital signature

Re: [RFC PATCH] btrfs: Gracefully handle errors in btrfs_qgroup_trace_extent_post

[Qu Wenruo]

On 2018年01月27日 10:26, Qu Wenruo wrote:
> 
> 
> On 2018年01月26日 22:13, Nikolay Borisov wrote:
>> Running generic/019 with qgroups on the scratch device enabled is
>> almost guaranteed to trigger the BUG_ON in btrfs_free_tree_block. It's
>> supposed to trigger only on -ENOMEM, in reality, however, it's possible
>> to get -EIO from btrfs_qgroup_trace_extent_post. This function just
>> finds the roots of the extent being tracked and sets the qrecord->old_roots
>> list. If this operation fails nothing critical happens except the
>> quota accounting can be considered wrong. In such case just set the
>> INCONSISTENT flag for the quota and be done with it.
>>
>> Signed-off-by: Nikolay Borisov 
>> ---
>>
>> Qu, 
>>
>> This fixes the crash for me, however I'm not entirely sure it's really the 
>> best fix since it's leaking the usage of INCONSISTENT out of qgroup.c. 
>> Ideally
>> I'd like to avoid this. Since you have more experience with qgroups and also 
>> you
>> introduced the extent tracking code in add_delayed_tree_ref how does 
>> look to you? 

And for leaking INCONSISTENT flag out of qgroup scope, it could be
integrated into btrfs_qgroup_trace_extent_post() so we automatically
mark qgroup inconsistent without the help from callers.

Thanks,
Qu

>>
>>
>>  fs/btrfs/delayed-ref.c | 9 +++--
>>  1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
>> index a1a40cf382e3..1e9aa18cc0d8 100644
>> --- a/fs/btrfs/delayed-ref.c
>> +++ b/fs/btrfs/delayed-ref.c
>> @@ -820,8 +820,13 @@ int btrfs_add_delayed_tree_ref(struct btrfs_fs_info 
>> *fs_info,
>>   num_bytes, parent, ref_root, level, action);
>>  spin_unlock(_refs->lock);
>>  
>> -if (qrecord_inserted)
>> -return btrfs_qgroup_trace_extent_post(fs_info, record);
>> +if (qrecord_inserted) {
>> +int ret = btrfs_qgroup_trace_extent_post(fs_info, record);
> 
> Since generic/019 is using make_fail_request, it can cause some tree
> reads fail.
> 
> And since btrfs_qgroup_trace_extent_post() cause btrfs_find_all_roots()
> to fill @old_roots for new extents, it will definitely read tree blocks,
> and when its request failed, it returns -EIO as expected.
> 
> So yes, the qgroup code break the old assumption that
> btrfs_add_delayed_tree|data_ref() could only return -ENOMEM.
> 
> 
> But compared to delayed-ref, qgroup is not a critical payload (if
> delayed-ref is screwed up, extent tree and possible data extent will be
> corrupted, while qgroup failure only affects qgroup), so it's OK to keep
> the old assumption.
> 
>> +if (ret != -ENOMEM)
> Here we don't really need to handle ENOMEM specially.
> Just check ( ret < 0 ) should be enough.
> 
>> +fs_info->qgroup_flags |= 
>> BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
> 
> And extra btrfs_warn() would help.
> 
>> +else
>> +return -ENOMEM;
> 
> Since btrfs is not a critical payload compared to delayed ref, returning
> 0 should not be a problem as we have already marked qgroup inconsistent.
> (With comment explaining why it's OK to return 0)
> 
> Thanks,
> Qu
> 
>> +}
>>  return 0;
>>  
>>  free_head_ref:
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [RFC PATCH] btrfs: Gracefully handle errors in btrfs_qgroup_trace_extent_post

[Qu Wenruo]

On 2018年01月26日 22:13, Nikolay Borisov wrote:
> Running generic/019 with qgroups on the scratch device enabled is
> almost guaranteed to trigger the BUG_ON in btrfs_free_tree_block. It's
> supposed to trigger only on -ENOMEM, in reality, however, it's possible
> to get -EIO from btrfs_qgroup_trace_extent_post. This function just
> finds the roots of the extent being tracked and sets the qrecord->old_roots
> list. If this operation fails nothing critical happens except the
> quota accounting can be considered wrong. In such case just set the
> INCONSISTENT flag for the quota and be done with it.
> 
> Signed-off-by: Nikolay Borisov 
> ---
> 
> Qu, 
> 
> This fixes the crash for me, however I'm not entirely sure it's really the 
> best fix since it's leaking the usage of INCONSISTENT out of qgroup.c. Ideally
> I'd like to avoid this. Since you have more experience with qgroups and also 
> you
> introduced the extent tracking code in add_delayed_tree_ref how does 
> look to you? 
> 
> 
>  fs/btrfs/delayed-ref.c | 9 +++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
> index a1a40cf382e3..1e9aa18cc0d8 100644
> --- a/fs/btrfs/delayed-ref.c
> +++ b/fs/btrfs/delayed-ref.c
> @@ -820,8 +820,13 @@ int btrfs_add_delayed_tree_ref(struct btrfs_fs_info 
> *fs_info,
>num_bytes, parent, ref_root, level, action);
>   spin_unlock(_refs->lock);
>  
> - if (qrecord_inserted)
> - return btrfs_qgroup_trace_extent_post(fs_info, record);
> + if (qrecord_inserted) {
> + int ret = btrfs_qgroup_trace_extent_post(fs_info, record);

Since generic/019 is using make_fail_request, it can cause some tree
reads fail.

And since btrfs_qgroup_trace_extent_post() cause btrfs_find_all_roots()
to fill @old_roots for new extents, it will definitely read tree blocks,
and when its request failed, it returns -EIO as expected.

So yes, the qgroup code break the old assumption that
btrfs_add_delayed_tree|data_ref() could only return -ENOMEM.


But compared to delayed-ref, qgroup is not a critical payload (if
delayed-ref is screwed up, extent tree and possible data extent will be
corrupted, while qgroup failure only affects qgroup), so it's OK to keep
the old assumption.

> + if (ret != -ENOMEM)
Here we don't really need to handle ENOMEM specially.
Just check ( ret < 0 ) should be enough.

> + fs_info->qgroup_flags |= 
> BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;

And extra btrfs_warn() would help.

> + else
> + return -ENOMEM;

Since btrfs is not a critical payload compared to delayed ref, returning
0 should not be a problem as we have already marked qgroup inconsistent.
(With comment explaining why it's OK to return 0)

Thanks,
Qu

> + }
>   return 0;
>  
>  free_head_ref:
> 



signature.asc
Description: OpenPGP digital signature

[PATCH] btrfs: print error if primary super block write fails

[Howard McLauchlan]

Presently, failing a primary super block write but succeeding in at
least one super block write in general will appear to users as if
nothing important went wrong. However, upon unmounting and re-mounting,
the file system will be in a rolled back state. This was discovered
with a BCC program that uses bpf_override_return() to fail super block
writes.

This patch outputs an error clarifying that the primary super block
write has failed, so users can expect potentially erroneous behaviour.
It also forces wait_dev_supers() to return an error to its caller if
the primary super block write fails.

Signed-off-by: Howard McLauchlan 
---
 fs/btrfs/disk-io.c | 17 ++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 5da18ebc9222..8f96e1e4c613 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3293,11 +3293,13 @@ static int write_dev_supers(struct btrfs_device *device,
  * Return number of errors when buffer head is not found or not marked up to
  * date.
  */
-static int wait_dev_supers(struct btrfs_device *device, int max_mirrors)
+static int wait_dev_supers(struct btrfs_fs_info *fs_info,
+  struct btrfs_device *device, int max_mirrors)
 {
struct buffer_head *bh;
int i;
int errors = 0;
+   bool primary_failed = false;
u64 bytenr;
 
if (max_mirrors == 0)
@@ -3314,11 +3316,14 @@ static int wait_dev_supers(struct btrfs_device *device, 
int max_mirrors)
  BTRFS_SUPER_INFO_SIZE);
if (!bh) {
errors++;
+   primary_failed = (i == 0) || primary_failed;
continue;
}
wait_on_buffer(bh);
-   if (!buffer_uptodate(bh))
+   if (!buffer_uptodate(bh)) {
errors++;
+   primary_failed = (i == 0) || primary_failed;
+   }
 
/* drop our reference */
brelse(bh);
@@ -3327,6 +3332,12 @@ static int wait_dev_supers(struct btrfs_device *device, 
int max_mirrors)
brelse(bh);
}
 
+   /* log error, force error return */
+   if (primary_failed) {
+   btrfs_err(fs_info, "error encountered writing primary super 
block");
+   return -1;
+   }
+
return errors < i ? 0 : -1;
 }
 
@@ -3557,7 +3568,7 @@ int write_all_supers(struct btrfs_fs_info *fs_info, int 
max_mirrors)
if (!dev->in_fs_metadata || !dev->writeable)
continue;
 
-   ret = wait_dev_supers(dev, max_mirrors);
+   ret = wait_dev_supers(fs_info, dev, max_mirrors);
if (ret)
total_errors++;
}
-- 
2.14.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: degraded permanent mount option

[Christophe Yayon]

Hi Chris,

Thanks for this complete answer.

I have to do some benchmark with mdadm raid and btrfs native raid...

Thanks

--
Christophe Yayon

> On 26 Jan 2018, at 22:54, Chris Murphy  wrote:
> 
>> On Fri, Jan 26, 2018 at 7:02 AM, Christophe Yayon  
>> wrote:
>> 
>> Just a little question about "degraded" mount option. Is it a good idea to 
>> add this option (permanent) in fstab and grub rootflags for raid1/10 array ? 
>> Just to allow the system to boot again if a single hdd fail.
> 
> No because it's going to open a window where a delayed member drive
> will mean the volume is mounted degraded, which will happen silently.
> And current behavior in such a case, any new writes go to single
> chunks. Again it's silent. When the delayed drive appears, it's not
> going to be added, the volume is still treated as degraded. And even
> when you remount to bring them all together in a normal mount, Btrfs
> will not automatically sync the drives, so you will still have some
> single chunk writes on one drive not the other. So you have a window
> of time where there can be data loss if a real failure occurs, and you
> need degraded mounting. Further, right now Btrfs will only do one
> degraded rw mount, and you *must* fix that degradedness before it is
> umounted or else you will only ever be able to mount it again ro.
> There are unmerged patches to work around this, so you'd need to
> commit to building your own kernel. I can't see any way of reliably
> using Btrfs in production for the described use case otherwise. You
> can't depend on getting the delayed or replacement drive restored, and
> the volume made healthy again, because ostensibly the whole point of
> the setup is having good uptime and you won't have that assurance
> unless you carry these patches.
> 
> Also note that there are two kinds of degraded writes. a.) drive was
> missing at mount time, and volume is mounted degraded, for raid1
> volumes you get single chunks written; to sync once the missing drive
> appears you do a btrfs balance -dconvert=raid1,soft
> -mconvert=raid1,soft which should be fairly fast; b.) if the drive
> goes missing after a normal mount, Btrfs continues to write out raid1
> chunks; to sync once the missing drive appears you have to do a full
> scrub or balance of the entire volume there's no shortcut.
> 
> Anyway, for the described use case I think you're better off with
> mdadm or LVM raid1 or raid10, and then format with Btrfs and DUP
> metadata (default mkfs) in which case you get full error detection and
> metadata error detection and correction, as well as the uptime you
> want.
> 
> -- 
> Chris Murphy

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: degraded permanent mount option

[Chris Murphy]

On Fri, Jan 26, 2018 at 7:02 AM, Christophe Yayon  wrote:

> Just a little question about "degraded" mount option. Is it a good idea to 
> add this option (permanent) in fstab and grub rootflags for raid1/10 array ? 
> Just to allow the system to boot again if a single hdd fail.

No because it's going to open a window where a delayed member drive
will mean the volume is mounted degraded, which will happen silently.
And current behavior in such a case, any new writes go to single
chunks. Again it's silent. When the delayed drive appears, it's not
going to be added, the volume is still treated as degraded. And even
when you remount to bring them all together in a normal mount, Btrfs
will not automatically sync the drives, so you will still have some
single chunk writes on one drive not the other. So you have a window
of time where there can be data loss if a real failure occurs, and you
need degraded mounting. Further, right now Btrfs will only do one
degraded rw mount, and you *must* fix that degradedness before it is
umounted or else you will only ever be able to mount it again ro.
There are unmerged patches to work around this, so you'd need to
commit to building your own kernel. I can't see any way of reliably
using Btrfs in production for the described use case otherwise. You
can't depend on getting the delayed or replacement drive restored, and
the volume made healthy again, because ostensibly the whole point of
the setup is having good uptime and you won't have that assurance
unless you carry these patches.

Also note that there are two kinds of degraded writes. a.) drive was
missing at mount time, and volume is mounted degraded, for raid1
volumes you get single chunks written; to sync once the missing drive
appears you do a btrfs balance -dconvert=raid1,soft
-mconvert=raid1,soft which should be fairly fast; b.) if the drive
goes missing after a normal mount, Btrfs continues to write out raid1
chunks; to sync once the missing drive appears you have to do a full
scrub or balance of the entire volume there's no shortcut.

Anyway, for the described use case I think you're better off with
mdadm or LVM raid1 or raid10, and then format with Btrfs and DUP
metadata (default mkfs) in which case you get full error detection and
metadata error detection and correction, as well as the uptime you
want.

-- 
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 11/26] libbtrfsutil: add btrfs_util_create_snapshot()

[Omar Sandoval]

On Fri, Jan 26, 2018 at 08:31:06PM +0100, Goffredo Baroncelli wrote:
> On 01/26/2018 07:40 PM, Omar Sandoval wrote:
> > From: Omar Sandoval 
> 
> 
> Hi,
> 
> this is a great work; only few comments:
> 1) I found not intuitive the naming of the function: i.e. you have 
> 
> btrfs_util_create_snapshot()
> btrfs_util_f_create_snapshot()
> 
> To me it seems more clear to have
> 
> btrfs_util_create_snapshot()
> btrfs_util_create_snapshot_f()
> 
> I think that it is better move the 'f' at the end: at the begin you have the 
> library "btrfs_util", in the middle you have the library function 
> 'create_snapshot', at the end there is the function variant ('f', because it 
> uses a file descriptor).
> 
> This is my opinion, even tough there are both examples like you 
> (stat/fstat/lstat) and like my one (capt_get_fd/cap_get_file)...

Yup, I was going off of the fstat/fsync/etc. convention. I don't
particularly like, e.g., btrfs_create_snapshot_f(), but
btrfs_create_snapshot_fd() isn't so bad.

> 2) I find the prefix 'btrfs_util_' a bit verbose. Why not a simple 'btrfs_', 
> even at the cost of a possible renaming of the conflicting function in the 
> current btrfs code.

That's a reasonable idea, I mostly wanted to avoid naming conflicts but
if this is the "one true Btrfs library" it shouldn't be a concern.

I'll wait a bit for people to bikeshed on the naming before I go and
rename everything, but I'm leaning towards the shorter name and
appending _fd instead of prepending f_.

> 3) regarding the btrfs_util_create_snapshot() function, I think that it would 
> be useful to add some more information:
> a) if used recursive is NOT atomic
> b) if used recursive, root capabilities are needed
> 
> The same for the other functions: mark with a 'root required' tag all the 
> functions which require the root capabilities.

That's a great point, I'll document that.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 11/26] libbtrfsutil: add btrfs_util_create_snapshot()

[Goffredo Baroncelli]

On 01/26/2018 07:40 PM, Omar Sandoval wrote:
> From: Omar Sandoval 


Hi,

this is a great work; only few comments:
1) I found not intuitive the naming of the function: i.e. you have 

btrfs_util_create_snapshot()
btrfs_util_f_create_snapshot()

To me it seems more clear to have

btrfs_util_create_snapshot()
btrfs_util_create_snapshot_f()

I think that it is better move the 'f' at the end: at the begin you have the 
library "btrfs_util", in the middle you have the library function 
'create_snapshot', at the end there is the function variant ('f', because it 
uses a file descriptor).

This is my opinion, even tough there are both examples like you 
(stat/fstat/lstat) and like my one (capt_get_fd/cap_get_file)...


2) I find the prefix 'btrfs_util_' a bit verbose. Why not a simple 'btrfs_', 
even at the cost of a possible renaming of the conflicting function in the 
current btrfs code.

3) regarding the btrfs_util_create_snapshot() function, I think that it would 
be useful to add some more information:
a) if used recursive is NOT atomic
b) if used recursive, root capabilities are needed

The same for the other functions: mark with a 'root required' tag all the 
functions which require the root capabilities.

BR
G.Baroncelli



> 
> Thanks to subvolume iterators, we can also implement recursive snapshot
> fairly easily.
> 
> Signed-off-by: Omar Sandoval 
> ---
>  libbtrfsutil/btrfsutil.h|  55 +
>  libbtrfsutil/python/btrfsutilpy.h   |   1 +
>  libbtrfsutil/python/module.c|  11 ++
>  libbtrfsutil/python/subvolume.c |  49 
>  libbtrfsutil/python/tests/test_qgroup.py|  10 ++
>  libbtrfsutil/python/tests/test_subvolume.py |  55 +
>  libbtrfsutil/subvolume.c| 166 
> 
>  7 files changed, 347 insertions(+)
> 
> diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
> index 3f78a34d..829f72b2 100644
> --- a/libbtrfsutil/btrfsutil.h
> +++ b/libbtrfsutil/btrfsutil.h
> @@ -332,6 +332,61 @@ enum btrfs_util_error btrfs_util_f_create_subvolume(int 
> parent_fd,
>   uint64_t *async_transid,
>   struct 
> btrfs_util_qgroup_inherit *qgroup_inherit);
>  
> +/**
> + * BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE - Also snapshot subvolumes beneath 
> the
> + * source subvolume onto the same location on the new snapshot. Note that 
> this
> + * modifies the newly-created snapshot, so it cannot be combined with
> + * %BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY.
> + */
> +#define BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE (1 << 0)
> +/**
> + * BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY - Create a read-only snapshot.
> + */
> +#define BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY (1 << 1)
> +#define BTRFS_UTIL_CREATE_SNAPSHOT_MASK ((1 << 2) - 1)
> +
> +/**
> + * btrfs_util_create_snapshot() - Create a new snapshot from a source 
> subvolume
> + * path.
> + * @source: Path of the existing subvolume to snapshot.
> + * @path: Where to create the snapshot.
> + * @flags: Bitmask of BTRFS_UTIL_CREATE_SNAPSHOT_* flags.
> + * @async_transid: See btrfs_util_create_subvolume(). If
> + * %BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE was in @flags, then this will 
> contain
> + * the largest transaction ID of all created subvolumes.
> + * @qgroup_inherit: See btrfs_util_create_subvolume().
> + *
> + * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
> + */
> +enum btrfs_util_error btrfs_util_create_snapshot(const char *source,
> +  const char *path, int flags,
> +  uint64_t *async_transid,
> +  struct 
> btrfs_util_qgroup_inherit *qgroup_inherit);
> +
> +/**
> + * btrfs_util_f_create_snapshot() - See btrfs_util_create_snapshot().
> + */
> +enum btrfs_util_error btrfs_util_f_create_snapshot(int fd, const char *path,
> +int flags,
> +uint64_t *async_transid,
> +struct 
> btrfs_util_qgroup_inherit *qgroup_inherit);
> +
> +/**
> + * btrfs_util_f2_create_snapshot() - Create a new snapshot from a source
> + * subvolume file descriptor and a target parent file descriptor and name.
> + * @fd: File descriptor of the existing subvolume to snapshot.
> + * @parent_fd: File descriptor of the parent directory where the snapshot 
> should
> + * be created.
> + * @name: Name of the snapshot to create.
> + * @flags: See btrfs_util_create_snapshot().
> + * @async_transid: See btrfs_util_create_snapshot().
> + * @qgroup_inherit: See btrfs_util_create_snapshot().
> + */
> +enum btrfs_util_error btrfs_util_f2_create_snapshot(int fd, int parent_fd,
> + const char *name, int flags,
> +   

Re: [PATCH 00/26] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"

[Hugo Mills]

On Fri, Jan 26, 2018 at 10:40:48AM -0800, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> Hello,
> 
> One of the features requests I get most often is a library to do the
> sorts of operations that we do with btrfs-progs. We can shell out to
> btrfs-progs, but the output format isn't always easily parsasble, and
> shelling out isn't always ideal. There's libbtrfs, but it's very
> incomplete, doesn't have a well thought out API, and is licensed under
> the GPL, making it hard to use for many projects.
> 
> libbtrfsutil is a new library written from scratch to address these
> issues. The implementation is completely independent of the existing
> btrfs-progs code, including kerncompat.h, and has a clean API and naming
> scheme. It is licensed under the LGPL. It also includes Python bindings
> by default. I will maintain the library code.

   *speechless*

   That's awesome, Omar (although with the python bindings, you've
probably just ruined Hans' day ;) ).

   Hugo.

> Patch 1 is a preparation cleanup which can go in independently. Patch 2
> adds the build system stuff for the library, and patch 3 does the same
> for the Python bindings. Patches 4-14 implement the library helpers,
> currently subvolume helpers and the sync ioctls. Patches 15-26 replace
> the btrfs-progs and libbtrfs code to use libbtrfsutil instead. I took
> care to preserve backwards-compatibility. `btrfs subvol list` in
> particular had some buggy behaviors for -o and -a that I emulated in the
> new code, see the comments in the code.
> 
> These patches are also available on my GitHub:
> https://github.com/osandov/btrfs-progs/tree/libbtrfsutil. That branch
> will rebase as I update this series.
> 
> Please share feedback regarding the API, implementation, or anything
> else.
> 
> Thanks!
> 
> Omar Sandoval (26):
>   btrfs-progs: get rid of undocumented qgroup inheritance options
>   Add libbtrfsutil
>   libbtrfsutil: add Python bindings
>   libbtrfsutil: add btrfs_util_is_subvolume() and
> btrfs_util_subvolume_id()
>   libbtrfsutil: add qgroup inheritance helpers
>   libbtrfsutil: add btrfs_util_create_subvolume()
>   libbtrfsutil: add btrfs_util_subvolume_info()
>   libbtrfsutil: add btrfs_util_[gs]et_read_only()
>   libbtrfsutil: add btrfs_util_[gs]et_default_subvolume()
>   libbtrfsutil: add subvolume iterator helpers
>   libbtrfsutil: add btrfs_util_create_snapshot()
>   libbtrfsutil: add btrfs_util_delete_subvolume()
>   libbtrfsutil: add btrfs_util_deleted_subvolumes()
>   libbtrfsutil: add filesystem sync helpers
>   btrfs-progs: use libbtrfsutil for read-only property
>   btrfs-progs: use libbtrfsutil for sync ioctls
>   btrfs-progs: use libbtrfsutil for set-default
>   btrfs-progs: use libbtrfsutil for get-default
>   btrfs-progs: use libbtrfsutil for subvol create and snapshot
>   btrfs-progs: use libbtrfsutil for subvol delete
>   btrfs-progs: use libbtrfsutil for subvol show
>   btrfs-progs: use libbtrfsutil for subvol sync
>   btrfs-progs: replace test_issubvolume() with btrfs_util_is_subvolume()
>   btrfs-progs: add recursive snapshot/delete using libbtrfsutil
>   btrfs-progs: deprecate libbtrfs helpers with libbtrfsutil equivalents
>   btrfs-progs: use libbtrfsutil for subvolume list
> 
>  .gitignore   |2 +
>  Documentation/btrfs-subvolume.asciidoc   |   14 +-
>  INSTALL  |4 +
>  Makefile |   84 +-
>  Makefile.inc.in  |2 +
>  btrfs-list.c | 1201 +++---
>  btrfs-list.h |   22 +-
>  cmds-filesystem.c|   20 +-
>  cmds-inspect.c   |9 +-
>  cmds-qgroup.c|   20 +-
>  cmds-receive.c   |   12 +-
>  cmds-subvolume.c |  807 +--
>  configure.ac |   15 +
>  libbtrfsutil/COPYING |  674 +
>  libbtrfsutil/COPYING.LESSER  |  165 +++
>  libbtrfsutil/README.md   |   35 +
>  libbtrfsutil/btrfsutil.h |  621 
>  libbtrfsutil/errors.c|   55 +
>  libbtrfsutil/filesystem.c|  103 ++
>  libbtrfsutil/internal.h  |   36 +
>  libbtrfsutil/python/.gitignore   |7 +
>  libbtrfsutil/python/btrfsutilpy.h|   84 ++
>  libbtrfsutil/python/error.c  |  202 
>  libbtrfsutil/python/filesystem.c |   94 ++
>  libbtrfsutil/python/module.c |  321 ++
>  libbtrfsutil/python/qgroup.c |  141 +++
>  libbtrfsutil/python/setup.py |  103 ++
>  libbtrfsutil/python/subvolume.c  |  665 +
>  libbtrfsutil/python/tests/__init__.py|   66 ++
>  

[PATCH 16/26] btrfs-progs: use libbtrfsutil for sync ioctls

[Omar Sandoval]

From: Omar Sandoval 

Signed-off-by: Omar Sandoval 
---
 cmds-filesystem.c | 20 ++--
 cmds-qgroup.c |  7 ---
 cmds-subvolume.c  | 23 ++-
 3 files changed, 24 insertions(+), 26 deletions(-)

diff --git a/cmds-filesystem.c b/cmds-filesystem.c
index 17d399d5..2353fdb0 100644
--- a/cmds-filesystem.c
+++ b/cmds-filesystem.c
@@ -28,6 +28,8 @@
 #include 
 #include 
 
+#include 
+
 #include "kerncompat.h"
 #include "ctree.h"
 #include "utils.h"
@@ -815,26 +817,16 @@ static const char * const cmd_filesystem_sync_usage[] = {
 
 static int cmd_filesystem_sync(int argc, char **argv)
 {
-   int fd, res, e;
-   char*path;
-   DIR *dirstream = NULL;
+   enum btrfs_util_error err;
 
clean_args_no_options(argc, argv, cmd_filesystem_sync_usage);
 
if (check_argc_exact(argc - optind, 1))
usage(cmd_filesystem_sync_usage);
 
-   path = argv[optind];
-
-   fd = btrfs_open_dir(path, , 1);
-   if (fd < 0)
-   return 1;
-
-   res = ioctl(fd, BTRFS_IOC_SYNC);
-   e = errno;
-   close_file_or_dir(fd, dirstream);
-   if( res < 0 ){
-   error("sync ioctl failed on '%s': %s", path, strerror(e));
+   err = btrfs_util_sync(argv[optind]);
+   if (err) {
+   error_btrfs_util(err);
return 1;
}
 
diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index 38382ea9..fac692d4 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -302,6 +302,7 @@ static int cmd_qgroup_show(int argc, char **argv)
int filter_flag = 0;
unsigned unit_mode;
int sync = 0;
+   enum btrfs_util_error err;
 
struct btrfs_qgroup_comparer_set *comparer_set;
struct btrfs_qgroup_filter_set *filter_set;
@@ -375,10 +376,10 @@ static int cmd_qgroup_show(int argc, char **argv)
}
 
if (sync) {
-   ret = ioctl(fd, BTRFS_IOC_SYNC);
-   if (ret < 0)
+   err = btrfs_util_f_sync(fd);
+   if (err)
warning("sync ioctl failed on '%s': %s", path,
- strerror(errno));
+   strerror(errno));
}
 
if (filter_flag) {
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index f8e34bc8..89a4ebb0 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -219,12 +219,18 @@ out:
 
 static int wait_for_commit(int fd)
 {
-   int ret;
+   enum btrfs_util_error err;
+   uint64_t transid;
 
-   ret = ioctl(fd, BTRFS_IOC_START_SYNC, NULL);
-   if (ret < 0)
-   return ret;
-   return ioctl(fd, BTRFS_IOC_WAIT_SYNC, NULL);
+   err = btrfs_util_f_start_sync(fd, );
+   if (err)
+   return -1;
+
+   err = btrfs_util_f_wait_sync(fd, transid);
+   if (err)
+   return -1;
+
+   return 0;
 }
 
 static const char * const cmd_subvol_delete_usage[] = {
@@ -940,10 +946,9 @@ static int cmd_subvol_find_new(int argc, char **argv)
if (fd < 0)
return 1;
 
-   ret = ioctl(fd, BTRFS_IOC_SYNC);
-   if (ret < 0) {
-   error("sync ioctl failed on '%s': %s",
-   subvol, strerror(errno));
+   err = btrfs_util_f_sync(fd);
+   if (err) {
+   error_btrfs_util(err);
close_file_or_dir(fd, dirstream);
return 1;
}
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 09/26] libbtrfsutil: add btrfs_util_[gs]et_default_subvolume()

[Omar Sandoval]

From: Omar Sandoval 

set_default_subvolume() is a trivial ioctl(), but there's no ioctl() for
get_default_subvolume(), so we need to search the root tree.

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h|  37 +
 libbtrfsutil/python/btrfsutilpy.h   |   2 +
 libbtrfsutil/python/module.c|  14 
 libbtrfsutil/python/subvolume.c |  50 
 libbtrfsutil/python/tests/test_subvolume.py |  14 
 libbtrfsutil/subvolume.c| 116 
 6 files changed, 233 insertions(+)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index d187d3a7..1643ea7b 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -260,6 +260,43 @@ enum btrfs_util_error 
btrfs_util_set_subvolume_read_only(const char *path,
 enum btrfs_util_error btrfs_util_f_set_subvolume_read_only(int fd,
   bool read_only);
 
+/**
+ * btrfs_util_get_default_subvolume() - Get the default subvolume for a
+ * filesystem.
+ * @path: Path on a Btrfs filesystem.
+ * @id_ret: Returned subvolume ID.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_get_default_subvolume(const char *path,
+  uint64_t *id_ret);
+
+/**
+ * btrfs_util_f_get_default_subvolume() - See
+ * btrfs_util_get_default_subvolume().
+ */
+enum btrfs_util_error btrfs_util_f_get_default_subvolume(int fd,
+uint64_t *id_ret);
+
+/**
+ * btrfs_util_set_default_subvolume() - Set the default subvolume for a
+ * filesystem.
+ * @path: Path in a Btrfs filesystem. This may be any path in the filesystem; 
it
+ * does not have to refer to a subvolume unless @id is zero.
+ * @id: ID of subvolume to set as the default. If zero is given, the subvolume
+ * ID of @path is used.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_set_default_subvolume(const char *path,
+  uint64_t id);
+
+/**
+ * btrfs_util_f_set_default_subvolume() - See
+ * btrfs_util_set_default_subvolume().
+ */
+enum btrfs_util_error btrfs_util_f_set_default_subvolume(int fd, uint64_t id);
+
 struct btrfs_util_qgroup_inherit;
 
 /**
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index 21253e51..41314d4a 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -66,6 +66,8 @@ PyObject *subvolume_path(PyObject *self, PyObject *args, 
PyObject *kwds);
 PyObject *subvolume_info(PyObject *self, PyObject *args, PyObject *kwds);
 PyObject *get_subvolume_read_only(PyObject *self, PyObject *args, PyObject 
*kwds);
 PyObject *set_subvolume_read_only(PyObject *self, PyObject *args, PyObject 
*kwds);
+PyObject *get_default_subvolume(PyObject *self, PyObject *args, PyObject 
*kwds);
+PyObject *set_default_subvolume(PyObject *self, PyObject *args, PyObject 
*kwds);
 PyObject *create_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
 
 void add_module_constants(PyObject *m);
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index 3395fb14..0ac4d63a 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -173,6 +173,20 @@ static PyMethodDef btrfsutil_methods[] = {
 "Arguments:\n"
 "path -- string, bytes, path-like object, or open file descriptor\n"
 "read_only -- bool flag value"},
+   {"get_default_subvolume", (PyCFunction)get_default_subvolume,
+METH_VARARGS | METH_KEYWORDS,
+"get_default_subvolume(path) -> int\n\n"
+"Get the ID of the default subvolume of a filesystem.\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor"},
+   {"set_default_subvolume", (PyCFunction)set_default_subvolume,
+METH_VARARGS | METH_KEYWORDS,
+"set_default_subvolume(path, id=0)\n\n"
+"Set the default subvolume of a filesystem.\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor\n"
+"id -- if not zero, set the default subvolume to the subvolume with\n"
+"this ID instead of the given path"},
{"create_subvolume", (PyCFunction)create_subvolume,
 METH_VARARGS | METH_KEYWORDS,
 "create_subvolume(path, async=False)\n\n"
diff --git a/libbtrfsutil/python/subvolume.c b/libbtrfsutil/python/subvolume.c
index 9ae7729f..7b30e03c 100644
--- a/libbtrfsutil/python/subvolume.c
+++ b/libbtrfsutil/python/subvolume.c
@@ -268,6 +268,56 @@ PyObject *set_subvolume_read_only(PyObject *self, PyObject 
*args, PyObject *kwds
Py_RETURN_NONE;
 }
 
+PyObject *get_default_subvolume(PyObject *self, PyObject *args, PyObject *kwds)
+{
+  

[PATCH 15/26] btrfs-progs: use libbtrfsutil for read-only property

[Omar Sandoval]

From: Omar Sandoval 

Signed-off-by: Omar Sandoval 
---
 props.c | 69 -
 1 file changed, 25 insertions(+), 44 deletions(-)

diff --git a/props.c b/props.c
index cddbd927..e4edba06 100644
--- a/props.c
+++ b/props.c
@@ -21,6 +21,8 @@
 #include 
 #include 
 
+#include 
+
 #include "ctree.h"
 #include "commands.h"
 #include "utils.h"
@@ -41,56 +43,35 @@ static int prop_read_only(enum prop_object_type type,
  const char *name,
  const char *value)
 {
-   int ret = 0;
-   int fd = -1;
-   u64 flags = 0;
-
-   fd = open(object, O_RDONLY);
-   if (fd < 0) {
-   ret = -errno;
-   error("failed to open %s: %s", object, strerror(-ret));
-   goto out;
-   }
+   enum btrfs_util_error err;
+   bool read_only;
 
-   ret = ioctl(fd, BTRFS_IOC_SUBVOL_GETFLAGS, );
-   if (ret < 0) {
-   ret = -errno;
-   error("failed to get flags for %s: %s", object,
-   strerror(-ret));
-   goto out;
-   }
-
-   if (!value) {
-   if (flags & BTRFS_SUBVOL_RDONLY)
-   fprintf(stdout, "ro=true\n");
-   else
-   fprintf(stdout, "ro=false\n");
-   ret = 0;
-   goto out;
-   }
+   if (value) {
+   if (!strcmp(value, "true")) {
+   read_only = true;
+   } else if (!strcmp(value, "false")) {
+   read_only = false;
+   } else {
+   error("invalid value for property: %s", value);
+   return -EINVAL;
+   }
 
-   if (!strcmp(value, "true")) {
-   flags |= BTRFS_SUBVOL_RDONLY;
-   } else if (!strcmp(value, "false")) {
-   flags = flags & ~BTRFS_SUBVOL_RDONLY;
+   err = btrfs_util_set_subvolume_read_only(object, read_only);
+   if (err) {
+   error_btrfs_util(err);
+   return -errno;
+   }
} else {
-   ret = -EINVAL;
-   error("invalid value for property: %s", value);
-   goto out;
-   }
+   err = btrfs_util_get_subvolume_read_only(object, _only);
+   if (err) {
+   error_btrfs_util(err);
+   return -errno;
+   }
 
-   ret = ioctl(fd, BTRFS_IOC_SUBVOL_SETFLAGS, );
-   if (ret < 0) {
-   ret = -errno;
-   error("failed to set flags for %s: %s", object,
-   strerror(-ret));
-   goto out;
+   printf("ro=%s\n", read_only ? "true" : "false");
}
 
-out:
-   if (fd != -1)
-   close(fd);
-   return ret;
+   return 0;
 }
 
 static int prop_label(enum prop_object_type type,
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 08/26] libbtrfsutil: add btrfs_util_[gs]et_read_only()

[Omar Sandoval]

From: Omar Sandoval 

In the future, btrfs_util_[gs]et_subvolume_flags() might be useful, but
since these are the only subvolume flags we've defined in all this time,
this will do for now.

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h| 33 ++
 libbtrfsutil/python/btrfsutilpy.h   |  2 +
 libbtrfsutil/python/module.c| 13 ++
 libbtrfsutil/python/subvolume.c | 53 ++
 libbtrfsutil/python/tests/test_subvolume.py | 17 +++
 libbtrfsutil/subvolume.c| 69 +
 6 files changed, 187 insertions(+)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 41d5c1e7..d187d3a7 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -227,6 +227,39 @@ enum btrfs_util_error btrfs_util_subvolume_info(const char 
*path, uint64_t id,
 enum btrfs_util_error btrfs_util_f_subvolume_info(int fd, uint64_t id,
  struct 
btrfs_util_subvolume_info *subvol);
 
+/**
+ * btrfs_util_get_subvolume_read_only() - Get whether a subvolume is read-only.
+ * @path: Subvolume path.
+ * @ret: Returned read-only flag.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_get_subvolume_read_only(const char *path,
+bool *ret);
+
+/**
+ * btrfs_util_f_get_subvolume_read_only() - See
+ * btrfs_util_get_subvolume_read_only().
+ */
+enum btrfs_util_error btrfs_util_f_get_subvolume_read_only(int fd, bool *ret);
+
+/**
+ * btrfs_util_set_subvolume_read_only() - Set whether a subvolume is read-only.
+ * @path: Subvolume path.
+ * @read_only: New value of read-only flag.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_set_subvolume_read_only(const char *path,
+bool read_only);
+
+/**
+ * btrfs_util_f_set_subvolume_read_only() - See
+ * btrfs_util_set_subvolume_read_only().
+ */
+enum btrfs_util_error btrfs_util_f_set_subvolume_read_only(int fd,
+  bool read_only);
+
 struct btrfs_util_qgroup_inherit;
 
 /**
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index e601cb8b..21253e51 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -64,6 +64,8 @@ PyObject *is_subvolume(PyObject *self, PyObject *args, 
PyObject *kwds);
 PyObject *subvolume_id(PyObject *self, PyObject *args, PyObject *kwds);
 PyObject *subvolume_path(PyObject *self, PyObject *args, PyObject *kwds);
 PyObject *subvolume_info(PyObject *self, PyObject *args, PyObject *kwds);
+PyObject *get_subvolume_read_only(PyObject *self, PyObject *args, PyObject 
*kwds);
+PyObject *set_subvolume_read_only(PyObject *self, PyObject *args, PyObject 
*kwds);
 PyObject *create_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
 
 void add_module_constants(PyObject *m);
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index b1469fc9..3395fb14 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -160,6 +160,19 @@ static PyMethodDef btrfsutil_methods[] = {
 "path -- string, bytes, path-like object, or open file descriptor\n"
 "id -- if not zero, instead of returning information about the\n"
 "given path, return information about the subvolume with this ID"},
+   {"get_subvolume_read_only", (PyCFunction)get_subvolume_read_only,
+METH_VARARGS | METH_KEYWORDS,
+"get_subvolume_read_only(path) -> bool\n\n"
+"Get whether a subvolume is read-only.\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor"},
+   {"set_subvolume_read_only", (PyCFunction)set_subvolume_read_only,
+METH_VARARGS | METH_KEYWORDS,
+"set_subvolume_read_only(path, read_only=True)\n\n"
+"Set whether a subvolume is read-only.\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor\n"
+"read_only -- bool flag value"},
{"create_subvolume", (PyCFunction)create_subvolume,
 METH_VARARGS | METH_KEYWORDS,
 "create_subvolume(path, async=False)\n\n"
diff --git a/libbtrfsutil/python/subvolume.c b/libbtrfsutil/python/subvolume.c
index 9c323995..9ae7729f 100644
--- a/libbtrfsutil/python/subvolume.c
+++ b/libbtrfsutil/python/subvolume.c
@@ -215,6 +215,59 @@ PyStructSequence_Desc SubvolumeInfo_desc = {
 
 PyTypeObject SubvolumeInfo_type;
 
+PyObject *get_subvolume_read_only(PyObject *self, PyObject *args, PyObject 
*kwds)
+{
+   static char *keywords[] = {"path", NULL};
+   struct path_arg path = {.allow_fd = true};
+   enum btrfs_util_error err;
+   bool read_only;
+
+

[PATCH 13/26] libbtrfsutil: add btrfs_util_deleted_subvolumes()

[Omar Sandoval]

From: Omar Sandoval 

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h| 19 +++
 libbtrfsutil/python/btrfsutilpy.h   |  3 +
 libbtrfsutil/python/module.c| 30 ++
 libbtrfsutil/python/qgroup.c| 17 +-
 libbtrfsutil/python/subvolume.c | 30 ++
 libbtrfsutil/python/tests/test_subvolume.py |  8 +++
 libbtrfsutil/subvolume.c| 87 +
 7 files changed, 179 insertions(+), 15 deletions(-)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 080268ce..74b05e29 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -509,6 +509,25 @@ enum btrfs_util_error 
btrfs_util_subvolume_iterator_next_info(struct btrfs_util_
  char **path_ret,
  struct 
btrfs_util_subvolume_info *subvol);
 
+/**
+ * btrfs_util_deleted_subvolumes() - Get a list of subvolume which have been
+ * deleted but not yet cleaned up.
+ * @path: Path on a Btrfs filesystem.
+ * @ids: Returned array of subvolume IDs.
+ * @n: Returned number of IDs in the @ids array.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_deleted_subvolumes(const char *path,
+   uint64_t **ids,
+   size_t *n);
+
+/**
+ * btrfs_util_f_deleted_subvolumes() - See btrfs_util_deleted_subvolumes().
+ */
+enum btrfs_util_error btrfs_util_f_deleted_subvolumes(int fd, uint64_t **ids,
+ size_t *n);
+
 /**
  * btrfs_util_create_qgroup_inherit() - Create a qgroup inheritance specifier
  * for btrfs_util_create_subvolume() or btrfs_util_create_snapshot().
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index b3ec047f..be5122e2 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -54,6 +54,8 @@ struct path_arg {
 int path_converter(PyObject *o, void *p);
 void path_cleanup(struct path_arg *path);
 
+PyObject *list_from_uint64_array(const uint64_t *arr, size_t n);
+
 void SetFromBtrfsUtilError(enum btrfs_util_error err);
 void SetFromBtrfsUtilErrorWithPath(enum btrfs_util_error err,
   struct path_arg *path);
@@ -72,6 +74,7 @@ PyObject *set_default_subvolume(PyObject *self, PyObject 
*args, PyObject *kwds);
 PyObject *create_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
 PyObject *create_snapshot(PyObject *self, PyObject *args, PyObject *kwds);
 PyObject *delete_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
+PyObject *deleted_subvolumes(PyObject *self, PyObject *args, PyObject *kwds);
 
 void add_module_constants(PyObject *m);
 
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index e995a1be..eaa062ac 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -125,6 +125,29 @@ err:
return 0;
 }
 
+PyObject *list_from_uint64_array(const uint64_t *arr, size_t n)
+{
+PyObject *ret;
+size_t i;
+
+ret = PyList_New(n);
+if (!ret)
+   return NULL;
+
+for (i = 0; i < n; i++) {
+   PyObject *tmp;
+
+   tmp = PyLong_FromUnsignedLongLong(arr[i]);
+   if (!tmp) {
+   Py_DECREF(ret);
+   return NULL;
+   }
+   PyList_SET_ITEM(ret, i, tmp);
+}
+
+return ret;
+}
+
 void path_cleanup(struct path_arg *path)
 {
Py_CLEAR(path->object);
@@ -214,6 +237,13 @@ static PyMethodDef btrfsutil_methods[] = {
 "path -- string, bytes, or path-like object\n"
 "recursive -- if the given subvolume has child subvolumes, delete\n"
 "them instead of failing"},
+   {"deleted_subvolumes", (PyCFunction)deleted_subvolumes,
+METH_VARARGS | METH_KEYWORDS,
+"deleted_subvolumes(path)\n\n"
+"Get the list of subvolume IDs which have been deleted but not yet\n"
+"cleaned up\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor"},
{},
 };
 
diff --git a/libbtrfsutil/python/qgroup.c b/libbtrfsutil/python/qgroup.c
index 69716d92..44ac5ebc 100644
--- a/libbtrfsutil/python/qgroup.c
+++ b/libbtrfsutil/python/qgroup.c
@@ -55,25 +55,12 @@ static PyObject *QgroupInherit_getattro(QgroupInherit 
*self, PyObject *nameobj)
 }
 
 if (strcmp(name, "groups") == 0) {
-   PyObject *ret, *tmp;
const uint64_t *arr;
-   size_t n, i;
+   size_t n;
 
btrfs_util_qgroup_inherit_get_groups(self->inherit, , );
-   ret = PyList_New(n);
-   if (!ret)
-   return NULL;
-
-   for (i = 0; i < n; i++) {
-   tmp = 

[PATCH 12/26] libbtrfsutil: add btrfs_util_delete_subvolume()

[Omar Sandoval]

From: Omar Sandoval 

We also support recursive deletion using a subvolume iterator.

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h|  30 
 libbtrfsutil/python/btrfsutilpy.h   |   1 +
 libbtrfsutil/python/module.c|   8 +++
 libbtrfsutil/python/subvolume.c |  27 
 libbtrfsutil/python/tests/test_subvolume.py |  48 +
 libbtrfsutil/subvolume.c| 103 
 6 files changed, 217 insertions(+)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 829f72b2..080268ce 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -387,6 +387,36 @@ enum btrfs_util_error btrfs_util_f2_create_snapshot(int 
fd, int parent_fd,
uint64_t *async_transid,
struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
 
+/**
+ * BTRFS_UTIL_DELETE_SUBVOLUME_RECURSIVE - Delete subvolumes beneath the given
+ * subvolume before attempting to delete the given subvolume. By default,
+ * deleting a subvolume with child subvolumes is an error.
+ */
+#define BTRFS_UTIL_DELETE_SUBVOLUME_RECURSIVE (1 << 0)
+#define BTRFS_UTIL_DELETE_SUBVOLUME_MASK ((1 << 1) - 1)
+
+/**
+ * btrfs_util_delete_subvolume() - Delete a subvolume or snapshot.
+ * @path: Path of the subvolume to delete.
+ * @flags: Bitmask of BTRFS_UTIL_DELETE_SUBVOLUME_* flags.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_delete_subvolume(const char *path, int flags);
+
+/**
+ * btrfs_util_f_delete_subvolume() - Delete a subvolume or snapshot given its
+ * parent and name.
+ * @parent_fd: File descriptor of the subvolume's parent directory.
+ * @name: Name of the subvolume.
+ * @flags: See btrfs_util_delete_subvolume().
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_f_delete_subvolume(int parent_fd,
+   const char *name,
+   int flags);
+
 struct btrfs_util_subvolume_iterator;
 
 /**
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index d552e416..b3ec047f 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -71,6 +71,7 @@ PyObject *get_default_subvolume(PyObject *self, PyObject 
*args, PyObject *kwds);
 PyObject *set_default_subvolume(PyObject *self, PyObject *args, PyObject 
*kwds);
 PyObject *create_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
 PyObject *create_snapshot(PyObject *self, PyObject *args, PyObject *kwds);
+PyObject *delete_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
 
 void add_module_constants(PyObject *m);
 
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index d8f797cb..e995a1be 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -206,6 +206,14 @@ static PyMethodDef btrfsutil_methods[] = {
 "read_only -- create a read-only snapshot\n"
 "async -- create the subvolume without waiting for it to commit to\n"
 "disk and return the transaction ID"},
+   {"delete_subvolume", (PyCFunction)delete_subvolume,
+METH_VARARGS | METH_KEYWORDS,
+"delete_subvolume(path, recursive=False)\n\n"
+"Delete a subvolume or snapshot.\n\n"
+"Arguments:\n"
+"path -- string, bytes, or path-like object\n"
+"recursive -- if the given subvolume has child subvolumes, delete\n"
+"them instead of failing"},
{},
 };
 
diff --git a/libbtrfsutil/python/subvolume.c b/libbtrfsutil/python/subvolume.c
index 763a06d9..f740ebdc 100644
--- a/libbtrfsutil/python/subvolume.c
+++ b/libbtrfsutil/python/subvolume.c
@@ -396,6 +396,33 @@ PyObject *create_snapshot(PyObject *self, PyObject *args, 
PyObject *kwds)
Py_RETURN_NONE;
 }
 
+PyObject *delete_subvolume(PyObject *self, PyObject *args, PyObject *kwds)
+{
+   static char *keywords[] = {"path", "recursive", NULL};
+   struct path_arg path = {.allow_fd = false};
+   enum btrfs_util_error err;
+   int recursive = 0;
+   int flags = 0;
+
+   if (!PyArg_ParseTupleAndKeywords(args, kwds, "O&|p:delete_subvolume",
+keywords, _converter, ,
+))
+   return NULL;
+
+   if (recursive)
+   flags |= BTRFS_UTIL_DELETE_SUBVOLUME_RECURSIVE;
+
+   err = btrfs_util_delete_subvolume(path.path, flags);
+   if (err) {
+   SetFromBtrfsUtilErrorWithPath(err, );
+   path_cleanup();
+   return NULL;
+   }
+
+   path_cleanup();
+   Py_RETURN_NONE;
+}
+
 typedef struct {
PyObject_HEAD
struct 

[PATCH 11/26] libbtrfsutil: add btrfs_util_create_snapshot()

[Omar Sandoval]

From: Omar Sandoval 

Thanks to subvolume iterators, we can also implement recursive snapshot
fairly easily.

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h|  55 +
 libbtrfsutil/python/btrfsutilpy.h   |   1 +
 libbtrfsutil/python/module.c|  11 ++
 libbtrfsutil/python/subvolume.c |  49 
 libbtrfsutil/python/tests/test_qgroup.py|  10 ++
 libbtrfsutil/python/tests/test_subvolume.py |  55 +
 libbtrfsutil/subvolume.c| 166 
 7 files changed, 347 insertions(+)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 3f78a34d..829f72b2 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -332,6 +332,61 @@ enum btrfs_util_error btrfs_util_f_create_subvolume(int 
parent_fd,
uint64_t *async_transid,
struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
 
+/**
+ * BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE - Also snapshot subvolumes beneath the
+ * source subvolume onto the same location on the new snapshot. Note that this
+ * modifies the newly-created snapshot, so it cannot be combined with
+ * %BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY.
+ */
+#define BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE (1 << 0)
+/**
+ * BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY - Create a read-only snapshot.
+ */
+#define BTRFS_UTIL_CREATE_SNAPSHOT_READ_ONLY (1 << 1)
+#define BTRFS_UTIL_CREATE_SNAPSHOT_MASK ((1 << 2) - 1)
+
+/**
+ * btrfs_util_create_snapshot() - Create a new snapshot from a source subvolume
+ * path.
+ * @source: Path of the existing subvolume to snapshot.
+ * @path: Where to create the snapshot.
+ * @flags: Bitmask of BTRFS_UTIL_CREATE_SNAPSHOT_* flags.
+ * @async_transid: See btrfs_util_create_subvolume(). If
+ * %BTRFS_UTIL_CREATE_SNAPSHOT_RECURSIVE was in @flags, then this will contain
+ * the largest transaction ID of all created subvolumes.
+ * @qgroup_inherit: See btrfs_util_create_subvolume().
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_create_snapshot(const char *source,
+const char *path, int flags,
+uint64_t *async_transid,
+struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
+
+/**
+ * btrfs_util_f_create_snapshot() - See btrfs_util_create_snapshot().
+ */
+enum btrfs_util_error btrfs_util_f_create_snapshot(int fd, const char *path,
+  int flags,
+  uint64_t *async_transid,
+  struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
+
+/**
+ * btrfs_util_f2_create_snapshot() - Create a new snapshot from a source
+ * subvolume file descriptor and a target parent file descriptor and name.
+ * @fd: File descriptor of the existing subvolume to snapshot.
+ * @parent_fd: File descriptor of the parent directory where the snapshot 
should
+ * be created.
+ * @name: Name of the snapshot to create.
+ * @flags: See btrfs_util_create_snapshot().
+ * @async_transid: See btrfs_util_create_snapshot().
+ * @qgroup_inherit: See btrfs_util_create_snapshot().
+ */
+enum btrfs_util_error btrfs_util_f2_create_snapshot(int fd, int parent_fd,
+   const char *name, int flags,
+   uint64_t *async_transid,
+   struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
+
 struct btrfs_util_subvolume_iterator;
 
 /**
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index a9c15219..d552e416 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -70,6 +70,7 @@ PyObject *set_subvolume_read_only(PyObject *self, PyObject 
*args, PyObject *kwds
 PyObject *get_default_subvolume(PyObject *self, PyObject *args, PyObject 
*kwds);
 PyObject *set_default_subvolume(PyObject *self, PyObject *args, PyObject 
*kwds);
 PyObject *create_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
+PyObject *create_snapshot(PyObject *self, PyObject *args, PyObject *kwds);
 
 void add_module_constants(PyObject *m);
 
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index daf0747f..d8f797cb 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -195,6 +195,17 @@ static PyMethodDef btrfsutil_methods[] = {
 "path -- string, bytes, or path-like object\n"
 "async -- create the subvolume without waiting for it to commit to\n"
 "disk and return the transaction ID"},
+   {"create_snapshot", (PyCFunction)create_snapshot,
+METH_VARARGS | METH_KEYWORDS,

[PATCH 17/26] btrfs-progs: use libbtrfsutil for set-default

[Omar Sandoval]

From: Omar Sandoval 

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 47 ++-
 messages.h   | 14 ++
 2 files changed, 24 insertions(+), 37 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index 89a4ebb0..aaf88af8 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -28,6 +28,8 @@
 #include 
 #include 
 
+#include 
+
 #include "kerncompat.h"
 #include "ioctl.h"
 #include "qgroup.h"
@@ -853,11 +855,9 @@ static const char * const cmd_subvol_set_default_usage[] = 
{
 
 static int cmd_subvol_set_default(int argc, char **argv)
 {
-   int ret=0, fd, e;
-   u64 objectid;
-   char*path;
-   char*subvolid;
-   DIR *dirstream = NULL;
+   u64 objectid;
+   char *path;
+   enum btrfs_util_error err;
 
clean_args_no_options(argc, argv, cmd_subvol_set_default_usage);
 
@@ -867,44 +867,17 @@ static int cmd_subvol_set_default(int argc, char **argv)
 
if (argc - optind == 1) {
/* path to the subvolume is specified */
+   objectid = 0;
path = argv[optind];
-
-   ret = test_issubvolume(path);
-   if (ret < 0) {
-   error("stat error: %s", strerror(-ret));
-   return 1;
-   } else if (!ret) {
-   error("'%s' is not a subvolume", path);
-   return 1;
-   }
-
-   fd = btrfs_open_dir(path, , 1);
-   if (fd < 0)
-   return 1;
-
-   ret = lookup_path_rootid(fd, );
-   if (ret) {
-   error("unable to get subvol id: %s", strerror(-ret));
-   close_file_or_dir(fd, dirstream);
-   return 1;
-   }
} else {
/* subvol id and path to the filesystem are specified */
-   subvolid = argv[optind];
+   objectid = arg_strtou64(argv[optind]);
path = argv[optind + 1];
-   objectid = arg_strtou64(subvolid);
-
-   fd = btrfs_open_dir(path, , 1);
-   if (fd < 0)
-   return 1;
}
 
-   ret = ioctl(fd, BTRFS_IOC_DEFAULT_SUBVOL, );
-   e = errno;
-   close_file_or_dir(fd, dirstream);
-   if (ret < 0) {
-   error("unable to set a new default subvolume: %s",
-   strerror(e));
+   err = btrfs_util_set_default_subvolume(path, objectid);
+   if (err) {
+   error_btrfs_util(err);
return 1;
}
return 0;
diff --git a/messages.h b/messages.h
index 4999c7b9..cfabb86f 100644
--- a/messages.h
+++ b/messages.h
@@ -54,6 +54,20 @@
DO_ABORT_ON_ERROR;  \
} while (0)
 
+#define error_btrfs_util(err)  \
+   do {\
+   const char *errno_str = strerror(errno);\
+   const char *lib_str = btrfs_util_strerror(err)  \
+   PRINT_TRACE_ON_ERROR;   \
+   PRINT_VERBOSE_ERROR;\
+   if (lib_str && strcmp(errno_str, lib_str) != 0) \
+   __btrfs_error("%s: %s", lib_str, errno_str);\
+   else\
+   __btrfs_error("%s", errno_str); \
+   DO_ABORT_ON_ERROR;  \
+   } while (0)
+
+
 #define warning(fmt, ...)  \
do {\
PRINT_TRACE_ON_ERROR;   \
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 24/26] btrfs-progs: add recursive snapshot/delete using libbtrfsutil

[Omar Sandoval]

From: Omar Sandoval 

And update the documentation.

Signed-off-by: Omar Sandoval 
---
 Documentation/btrfs-subvolume.asciidoc | 14 --
 cmds-subvolume.c   | 25 +
 2 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/Documentation/btrfs-subvolume.asciidoc 
b/Documentation/btrfs-subvolume.asciidoc
index a8c4af4b..7a2ec8d2 100644
--- a/Documentation/btrfs-subvolume.asciidoc
+++ b/Documentation/btrfs-subvolume.asciidoc
@@ -81,6 +81,12 @@ wait for transaction commit at the end of the operation
 +
 -C|--commit-each
 wait for transaction commit after deleting each subvolume
++
+-R|--recursive
+delete subvolumes beneath each subvolume recursively
++
+-v|--verbose
+output more verbosely
 
 *find-new*  ::
 List the recently modified files in a subvolume, after  ID.
@@ -157,7 +163,7 @@ The id can be obtained from *btrfs subvolume list*, *btrfs 
subvolume show* or
 *show* ::
 Show information of a given subvolume in the .
 
-*snapshot* [-r]  |[/]::
+*snapshot* [-r|-R]  |[/]::
 Create a snapshot of the subvolume  with the
 name  in the  directory.
 +
@@ -167,7 +173,11 @@ If  is not a subvolume, btrfs returns an error.
 `Options`
 +
 -r
-Make the new snapshot read only.
+make the new snapshot read-only
++
+-R
+recursively snapshot subvolumes beneath the source; this option cannot be
+combined with -r
 
 *sync*  [subvolid...]::
 Wait until given subvolume(s) are completely removed from the filesystem after
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index 5c75799c..3521e7b7 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -187,6 +187,7 @@ static const char * const cmd_subvol_delete_usage[] = {
"",
"-c|--commit-after  wait for transaction commit at the end of the 
operation",
"-C|--commit-each   wait for transaction commit after deleting each 
subvolume",
+   "-R|--recursive delete subvolumes beneath each subvolume 
recursively",
"-v|--verbose   verbose output of operations",
NULL
 };
@@ -203,6 +204,7 @@ static int cmd_subvol_delete(int argc, char **argv)
DIR *dirstream = NULL;
int verbose = 0;
int commit_mode = 0;
+   int flags = 0;
u8 fsid[BTRFS_FSID_SIZE];
char uuidbuf[BTRFS_UUID_UNPARSED_SIZE];
struct seen_fsid *seen_fsid_hash[SEEN_FSID_HASH_SIZE] = { NULL, };
@@ -214,11 +216,12 @@ static int cmd_subvol_delete(int argc, char **argv)
static const struct option long_options[] = {
{"commit-after", no_argument, NULL, 'c'},
{"commit-each", no_argument, NULL, 'C'},
+   {"recursive", no_argument, NULL, 'R'},
{"verbose", no_argument, NULL, 'v'},
{NULL, 0, NULL, 0}
};
 
-   c = getopt_long(argc, argv, "cCv", long_options, NULL);
+   c = getopt_long(argc, argv, "cCRv", long_options, NULL);
if (c < 0)
break;
 
@@ -229,6 +232,9 @@ static int cmd_subvol_delete(int argc, char **argv)
case 'C':
commit_mode = COMMIT_EACH;
break;
+   case 'R':
+   flags |= BTRFS_UTIL_DELETE_SUBVOLUME_RECURSIVE;
+   break;
case 'v':
verbose++;
break;
@@ -281,7 +287,7 @@ again:
commit_mode == COMMIT_EACH || (commit_mode == COMMIT_AFTER && 
cnt + 1 == argc)
? "commit" : "no-commit", dname, vname);
 
-   err = btrfs_util_f_delete_subvolume(fd, vname, 0);
+   err = btrfs_util_f_delete_subvolume(fd, vname, flags);
if (err) {
error_btrfs_util(err);
ret = 1;
@@ -571,13 +577,15 @@ out:
 }
 
 static const char * const cmd_subvol_snapshot_usage[] = {
-   "btrfs subvolume snapshot [-r] [-i ]  
|[/]",
+   "btrfs subvolume snapshot [-r|-R] [-i ]  
|[/]",
"Create a snapshot of the subvolume",
"Create a writable/readonly snapshot of the subvolume  with",
"the name  in the  directory.  If only  is given,",
"the subvolume will be named the basename of .",
"",
"-r create a readonly snapshot",
+   "-R recursively snapshot subvolumes beneath the source; 
this",
+   "   option cannot be combined with -r",
"-i   add the newly created snapshot to a qgroup. This",
"   option can be given multiple times.",
NULL
@@ -591,7 +599,7 @@ static int cmd_subvol_snapshot(int argc, char **argv)
int retval = 1;
 
while (1) {
-   int c = getopt(argc, argv, "i:r");
+   int c = getopt(argc, argv, "i:rR");
if (c < 0)
break;
 
@@ -603,11 +611,20 @@ static int 

[PATCH 05/26] libbtrfsutil: add qgroup inheritance helpers

[Omar Sandoval]

From: Omar Sandoval 

We want to hide struct btrfs_qgroup_inherit from the user because that
comes from the Btrfs UAPI headers. Instead, wrap it in a struct
btrfs_util_qgroup_inherit and provide helpers to manipulate it. This
will be used for subvolume and snapshot creation.

Signed-off-by: Omar Sandoval 
---
 Makefile |   3 +-
 libbtrfsutil/btrfsutil.h |  45 +
 libbtrfsutil/python/btrfsutilpy.h|   6 ++
 libbtrfsutil/python/module.c |   8 ++
 libbtrfsutil/python/qgroup.c | 154 +++
 libbtrfsutil/python/setup.py |   1 +
 libbtrfsutil/python/tests/test_qgroup.py |  36 
 libbtrfsutil/qgroup.c|  86 +
 8 files changed, 338 insertions(+), 1 deletion(-)
 create mode 100644 libbtrfsutil/python/qgroup.c
 create mode 100644 libbtrfsutil/python/tests/test_qgroup.py
 create mode 100644 libbtrfsutil/qgroup.c

diff --git a/Makefile b/Makefile
index 48a558a9..2f798f11 100644
--- a/Makefile
+++ b/Makefile
@@ -123,7 +123,8 @@ libbtrfs_headers = send-stream.h send-utils.h send.h 
kernel-lib/rbtree.h btrfs-l
   kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
   extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
 libbtrfsutil_version := 0.1
-libbtrfsutil_objects = libbtrfsutil/errors.o libbtrfsutil/subvolume.o
+libbtrfsutil_objects = libbtrfsutil/errors.o libbtrfsutil/qgroup.o \
+  libbtrfsutil/subvolume.o
 convert_objects = convert/main.o convert/common.o convert/source-fs.o \
  convert/source-ext2.o convert/source-reiserfs.o
 mkfs_objects = mkfs/main.o mkfs/common.o
diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index dff6599d..d401f4d7 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -20,6 +20,7 @@
 #ifndef BTRFS_UTIL_H
 #define BTRFS_UTIL_H
 
+#include 
 #include 
 
 #ifdef __cplusplus
@@ -98,6 +99,50 @@ enum btrfs_util_error btrfs_util_subvolume_id(const char 
*path,
  */
 enum btrfs_util_error btrfs_util_f_subvolume_id(int fd, uint64_t *id_ret);
 
+struct btrfs_util_qgroup_inherit;
+
+/**
+ * btrfs_util_create_qgroup_inherit() - Create a qgroup inheritance specifier
+ * for btrfs_util_create_subvolume() or btrfs_util_create_snapshot().
+ * @flags: Must be zero.
+ * @ret: Returned qgroup inheritance specifier.
+ *
+ * The returned structure must be freed with
+ * btrfs_util_destroy_qgroup_inherit().
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_create_qgroup_inherit(int flags,
+  struct 
btrfs_util_qgroup_inherit **ret);
+
+/**
+ * btrfs_util_destroy_qgroup_inherit() - Destroy a qgroup inheritance specifier
+ * previously created with btrfs_util_create_qgroup_inherit().
+ * @inherit: Specifier to destroy.
+ */
+void btrfs_util_destroy_qgroup_inherit(struct btrfs_util_qgroup_inherit 
*inherit);
+
+/**
+ * btrfs_util_qgroup_inherit_add_group() - Add inheritance from a qgroup to a
+ * qgroup inheritance specifier.
+ * @inherit: Specifier to modify. May be reallocated.
+ * @qgroupid: ID of qgroup to inherit from.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_qgroup_inherit_add_group(struct 
btrfs_util_qgroup_inherit **inherit,
+ uint64_t qgroupid);
+
+/**
+ * btrfs_util_qgroup_inherit_get_groups() - Get the qgroups a qgroup 
inheritance
+ * specifier contains.
+ * @inherit: Qgroup inheritance specifier.
+ * @groups: Returned array of qgroup IDs.
+ * @n: Returned number of entries in the @groups array.
+ */
+void btrfs_util_qgroup_inherit_get_groups(const struct 
btrfs_util_qgroup_inherit *inherit,
+ const uint64_t **groups, size_t *n);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index 9a04fda7..a36f2671 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -29,7 +29,13 @@
 
 #include 
 
+typedef struct {
+   PyObject_HEAD
+   struct btrfs_util_qgroup_inherit *inherit;
+} QgroupInherit;
+
 extern PyTypeObject BtrfsUtilError_type;
+extern PyTypeObject QgroupInherit_type;
 
 /*
  * Helpers for path arguments based on posixmodule.c in CPython.
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index d492cbc7..de7d17a1 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -164,6 +164,10 @@ PyInit_btrfsutil(void)
if (PyType_Ready(_type) < 0)
return NULL;
 
+   QgroupInherit_type.tp_new = PyType_GenericNew;
+   if (PyType_Ready(_type) < 0)
+   return NULL;
+
m = PyModule_Create();
if (!m)

[PATCH 14/26] libbtrfsutil: add filesystem sync helpers

[Omar Sandoval]

From: Omar Sandoval 

Namely, sync, start_sync, and wait_sync.

Signed-off-by: Omar Sandoval 
---
 Makefile |   4 +-
 libbtrfsutil/btrfsutil.h |  44 
 libbtrfsutil/filesystem.c| 103 +++
 libbtrfsutil/internal.h  |   7 ++
 libbtrfsutil/python/btrfsutilpy.h|   3 +
 libbtrfsutil/python/filesystem.c |  94 
 libbtrfsutil/python/module.c |  21 ++
 libbtrfsutil/python/setup.py |   1 +
 libbtrfsutil/python/tests/test_filesystem.py |  73 +++
 libbtrfsutil/subvolume.c |   7 --
 10 files changed, 348 insertions(+), 9 deletions(-)
 create mode 100644 libbtrfsutil/filesystem.c
 create mode 100644 libbtrfsutil/python/filesystem.c
 create mode 100644 libbtrfsutil/python/tests/test_filesystem.py

diff --git a/Makefile b/Makefile
index 2f798f11..8bbe1c5e 100644
--- a/Makefile
+++ b/Makefile
@@ -123,8 +123,8 @@ libbtrfs_headers = send-stream.h send-utils.h send.h 
kernel-lib/rbtree.h btrfs-l
   kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
   extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
 libbtrfsutil_version := 0.1
-libbtrfsutil_objects = libbtrfsutil/errors.o libbtrfsutil/qgroup.o \
-  libbtrfsutil/subvolume.o
+libbtrfsutil_objects = libbtrfsutil/errors.o libbtrfsutil/filesystem.o \
+  libbtrfsutil/qgroup.o libbtrfsutil/subvolume.o
 convert_objects = convert/main.o convert/common.o convert/source-fs.o \
  convert/source-ext2.o convert/source-reiserfs.o
 mkfs_objects = mkfs/main.o mkfs/common.o
diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 74b05e29..d968f4de 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -70,6 +70,50 @@ enum btrfs_util_error {
  */
 const char *btrfs_util_strerror(enum btrfs_util_error err);
 
+/**
+ * btrfs_util_sync() - Force a sync on a specific Btrfs filesystem.
+ * @path: Path on a Btrfs filesystem.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_sync(const char *path);
+
+/**
+ * btrfs_util_f_sync() - See btrfs_util_sync().
+ */
+enum btrfs_util_error btrfs_util_f_sync(int fd);
+
+/**
+ * btrfs_util_start_sync() - Start a sync on a specific Btrfs filesystem but
+ * don't wait for it.
+ * @path: Path on a Btrfs filesystem.
+ * @transid: Returned transaction ID which can be waited on with
+ * btrfs_util_wait_sync(). This can be %NULL.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_start_sync(const char *path,
+   uint64_t *transid);
+
+/**
+ * btrfs_util_f_start_sync() - See btrfs_util_start_sync().
+ */
+enum btrfs_util_error btrfs_util_f_start_sync(int fd, uint64_t *transid);
+
+/**
+ * btrfs_util_wait_sync() - Wait for a transaction with a given ID to sync.
+ * @path: Path on a Btrfs filesystem.
+ * @transid: Transaction ID to wait for, or zero for the current transaction.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_wait_sync(const char *path, uint64_t transid);
+
+/**
+ * btrfs_util_f_wait_sync() - See btrfs_util_wait_sync().
+ */
+enum btrfs_util_error btrfs_util_f_wait_sync(int fd, uint64_t transid);
+
 /**
  * btrfs_util_is_subvolume() - Return whether a given path is a Btrfs 
subvolume.
  * @path: Path to check.
diff --git a/libbtrfsutil/filesystem.c b/libbtrfsutil/filesystem.c
new file mode 100644
index ..7199f920
--- /dev/null
+++ b/libbtrfsutil/filesystem.c
@@ -0,0 +1,103 @@
+/*
+ * Copyright (C) 2018 Facebook
+ *
+ * This file is part of libbtrfsutil.
+ *
+ * libbtrfsutil is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * libbtrfsutil is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with libbtrfsutil.  If not, see .
+ */
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include "btrfsutil.h"
+#include "internal.h"
+
+enum btrfs_util_error btrfs_util_sync(const char *path)
+{
+   enum btrfs_util_error err;
+   int fd;
+
+   fd = open(path, O_RDONLY);
+   if (fd == -1)
+   return BTRFS_UTIL_ERROR_OPEN_FAILED;
+
+   err = btrfs_util_f_sync(fd);
+   

[PATCH 06/26] libbtrfsutil: add btrfs_util_create_subvolume()

[Omar Sandoval]

From: Omar Sandoval 

Doing the ioctl() directly isn't too bad, but passing in a full path is
more convenient than opening the parent and passing the path component.

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h|  52 ++
 libbtrfsutil/internal.h |  29 
 libbtrfsutil/python/btrfsutilpy.h   |   2 +
 libbtrfsutil/python/module.c|  16 ++
 libbtrfsutil/python/subvolume.c |  59 +++
 libbtrfsutil/python/tests/test_qgroup.py|  11 ++
 libbtrfsutil/python/tests/test_subvolume.py |  80 -
 libbtrfsutil/subvolume.c| 241 
 8 files changed, 489 insertions(+), 1 deletion(-)
 create mode 100644 libbtrfsutil/internal.h

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index d401f4d7..20029f18 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -99,8 +99,60 @@ enum btrfs_util_error btrfs_util_subvolume_id(const char 
*path,
  */
 enum btrfs_util_error btrfs_util_f_subvolume_id(int fd, uint64_t *id_ret);
 
+/**
+ * btrfs_util_subvolume_path() - Get the path of the subvolume with a given ID
+ * relative to the filesystem root.
+ * @path: Path on a Btrfs filesystem.
+ * @id: ID of subvolume to set as the default. If zero is given, the subvolume
+ * ID of @path is used.
+ * @path_ret: Returned path.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_subvolume_path(const char *path, uint64_t id,
+   char **path_ret);
+
+/**
+ * btrfs_util_f_subvolume_path() - See btrfs_util_subvolume_path().
+ */
+enum btrfs_util_error btrfs_util_f_subvolume_path(int fd, uint64_t id,
+ char **path_ret);
+
 struct btrfs_util_qgroup_inherit;
 
+/**
+ * btrfs_util_create_subvolume() - Create a new subvolume.
+ * @path: Where to create the subvolume.
+ * @flags: Must be zero.
+ * @async_transid: If not NULL, create the subvolume asynchronously (i.e.,
+ * without waiting for it to commit it to disk) and return the transaction ID
+ * that it was created in. This transaction ID can be waited on with
+ * btrfs_util_wait_sync().
+ * @qgroup_inherit: Qgroups to inherit from, or NULL.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_create_subvolume(const char *path, int flags,
+ uint64_t *async_transid,
+ struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
+
+/**
+ * btrfs_util_f_create_subvolume() - Create a new subvolume given its parent 
and
+ * name.
+ * @parent_fd: File descriptor of the parent directory where the subvolume
+ * should be created.
+ * @name: Name of the subvolume to create.
+ * @flags: See btrfs_util_create_subvolume().
+ * @async_transid: See btrfs_util_create_subvolume().
+ * @qgroup_inherit: See btrfs_util_create_subvolume().
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_f_create_subvolume(int parent_fd,
+   const char *name, int flags,
+   uint64_t *async_transid,
+   struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
+
 /**
  * btrfs_util_create_qgroup_inherit() - Create a qgroup inheritance specifier
  * for btrfs_util_create_subvolume() or btrfs_util_create_snapshot().
diff --git a/libbtrfsutil/internal.h b/libbtrfsutil/internal.h
new file mode 100644
index ..bc91ad88
--- /dev/null
+++ b/libbtrfsutil/internal.h
@@ -0,0 +1,29 @@
+/*
+ * Copyright (C) 2018 Facebook
+ *
+ * This file is part of libbtrfsutil.
+ *
+ * libbtrfsutil is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * libbtrfsutil is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with libbtrfsutil.  If not, see .
+ */
+
+#ifndef BTRFS_UTIL_INTERNAL_H
+#define BTRFS_UTIL_INTERNAL_H
+
+#include 
+
+#define le16_to_cpu __le16_to_cpu
+#define le32_to_cpu __le32_to_cpu
+#define le64_to_cpu __le64_to_cpu
+
+#endif /* BTRFS_UTIL_INTERNAL_H */
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index a36f2671..32e83fe5 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ 

[PATCH 20/26] btrfs-progs: use libbtrfsutil for subvol delete

[Omar Sandoval]

From: Omar Sandoval 

Most of the interesting part of this command is the commit mode, so this
only saves a little bit of code.

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 23 ---
 1 file changed, 8 insertions(+), 15 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index 6b474fe4..c5e03011 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -225,7 +225,6 @@ static int cmd_subvol_delete(int argc, char **argv)
int res, ret = 0;
int cnt;
int fd = -1;
-   struct btrfs_ioctl_vol_args args;
char*dname, *vname, *cpath;
char*dupdname = NULL;
char*dupvname = NULL;
@@ -237,6 +236,7 @@ static int cmd_subvol_delete(int argc, char **argv)
char uuidbuf[BTRFS_UUID_UNPARSED_SIZE];
struct seen_fsid *seen_fsid_hash[SEEN_FSID_HASH_SIZE] = { NULL, };
enum { COMMIT_AFTER = 1, COMMIT_EACH = 2 };
+   enum btrfs_util_error err;
 
while (1) {
int c;
@@ -280,14 +280,9 @@ static int cmd_subvol_delete(int argc, char **argv)
 again:
path = argv[cnt];
 
-   res = test_issubvolume(path);
-   if (res < 0) {
-   error("cannot access subvolume %s: %s", path, strerror(-res));
-   ret = 1;
-   goto out;
-   }
-   if (!res) {
-   error("not a subvolume: %s", path);
+   err = btrfs_util_is_subvolume(path);
+   if (err) {
+   error_btrfs_util(err);
ret = 1;
goto out;
}
@@ -314,12 +309,10 @@ again:
printf("Delete subvolume (%s): '%s/%s'\n",
commit_mode == COMMIT_EACH || (commit_mode == COMMIT_AFTER && 
cnt + 1 == argc)
? "commit" : "no-commit", dname, vname);
-   memset(, 0, sizeof(args));
-   strncpy_null(args.name, vname);
-   res = ioctl(fd, BTRFS_IOC_SNAP_DESTROY, );
-   if(res < 0 ){
-   error("cannot delete '%s/%s': %s", dname, vname,
-   strerror(errno));
+
+   err = btrfs_util_f_delete_subvolume(fd, vname, 0);
+   if (err) {
+   error_btrfs_util(err);
ret = 1;
goto out;
}
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 07/26] libbtrfsutil: add btrfs_util_subvolume_info()

[Omar Sandoval]

From: Omar Sandoval 

This gets the the information in `btrfs subvolume show` from the root
item.

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h| 109 
 libbtrfsutil/python/btrfsutilpy.h   |   3 +
 libbtrfsutil/python/module.c|  14 +++
 libbtrfsutil/python/subvolume.c | 113 +
 libbtrfsutil/python/tests/test_subvolume.py |  50 ++
 libbtrfsutil/subvolume.c| 149 
 6 files changed, 438 insertions(+)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 20029f18..41d5c1e7 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -20,8 +20,10 @@
 #ifndef BTRFS_UTIL_H
 #define BTRFS_UTIL_H
 
+#include 
 #include 
 #include 
+#include 
 
 #ifdef __cplusplus
 extern "C" {
@@ -118,6 +120,113 @@ enum btrfs_util_error btrfs_util_subvolume_path(const 
char *path, uint64_t id,
 enum btrfs_util_error btrfs_util_f_subvolume_path(int fd, uint64_t id,
  char **path_ret);
 
+/**
+ * struct btrfs_util_subvolume_info - Information about a Btrfs subvolume.
+ */
+struct btrfs_util_subvolume_info {
+   /** @id: ID of this subvolume, unique across the filesystem. */
+   uint64_t id;
+
+   /**
+* @parent_id: ID of the subvolume which contains this subvolume, or
+* zero for the root subvolume (BTRFS_FS_TREE_OBJECTID) or orphaned
+* subvolumes (i.e., subvolumes which have been deleted but not yet
+* cleaned up).
+*/
+   uint64_t parent_id;
+
+   /**
+* @dir_id: Inode number of the directory containing this subvolume in
+* the parent subvolume, or zero for the root subvolume
+* (BTRFS_FS_TREE_OBJECTID) or orphaned subvolumes.
+*/
+   uint64_t dir_id;
+
+   /** @flags: On-disk root item flags. */
+   uint64_t flags;
+
+   /** @uuid: UUID of this subvolume. */
+   uint8_t uuid[16];
+
+   /**
+* @parent_uuid: UUID of the subvolume this subvolume is a snapshot of,
+* or all zeroes if this subvolume is not a snapshot.
+*/
+   uint8_t parent_uuid[16];
+
+   /**
+* @received_uuid: UUID of the subvolume this subvolume was received
+* from, or all zeroes if this subvolume was not received. Note that
+* this field, @stransid, @rtransid, @stime, and @rtime are set manually
+* by userspace after a subvolume is received.
+*/
+   uint8_t received_uuid[16];
+
+   /** @generation: Transaction ID of the subvolume root. */
+   uint64_t generation;
+
+   /**
+* @ctransid: Transaction ID when an inode in this subvolume was last
+* changed.
+*/
+   uint64_t ctransid;
+
+   /** @otransid: Transaction ID when this subvolume was created. */
+   uint64_t otransid;
+
+   /**
+* @stransid: Transaction ID of the sent subvolume this subvolume was
+* received from, or zero if this subvolume was not received. See the
+* note on @received_uuid.
+*/
+   uint64_t stransid;
+
+   /**
+* @rtransid: Transaction ID when this subvolume was received, or zero
+* if this subvolume was not received. See the note on @received_uuid.
+*/
+   uint64_t rtransid;
+
+   /** @ctime: Time when an inode in this subvolume was last changed. */
+   struct timespec ctime;
+
+   /** @otime: Time when this subvolume was created. */
+   struct timespec otime;
+
+   /**
+* @stime: Not well-defined, usually zero unless it was set otherwise.
+* See the note on @received_uuid.
+*/
+   struct timespec stime;
+
+   /**
+* @rtime: Time when this subvolume was received, or zero if this
+* subvolume was not received. See the note on @received_uuid.
+*/
+   struct timespec rtime;
+};
+
+/**
+ * btrfs_util_subvolume_info() - Get information about a subvolume.
+ * @path: Path in a Btrfs filesystem. This may be any path in the filesystem; 
it
+ * does not have to refer to a subvolume unless @id is zero.
+ * @id: ID of subvolume to get information about. If zero is given, the
+ * subvolume ID of @path is used.
+ * @subvol: Returned subvolume information. This can be %NULL if you just want
+ * to check whether the subvolume exists; %BTRFS_UTIL_ERROR_SUBVOLUME_NOT_FOUND
+ * will be returned if it does not.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_subvolume_info(const char *path, uint64_t id,
+   struct 
btrfs_util_subvolume_info *subvol);
+
+/**
+ * btrfs_util_f_subvolume_info() - See btrfs_util_subvolume_info().
+ */
+enum btrfs_util_error btrfs_util_f_subvolume_info(int fd, uint64_t id,
+ 

[PATCH 04/26] libbtrfsutil: add btrfs_util_is_subvolume() and btrfs_util_subvolume_id()

[Omar Sandoval]

From: Omar Sandoval 

These are the most trivial helpers in the library and will be used to
implement several of the more involved functions.

Signed-off-by: Omar Sandoval 
---
 Makefile|   2 +-
 libbtrfsutil/btrfsutil.h|  33 +++
 libbtrfsutil/python/btrfsutilpy.h   |   3 +
 libbtrfsutil/python/module.c|  12 +++
 libbtrfsutil/python/setup.py|   1 +
 libbtrfsutil/python/subvolume.c |  73 +++
 libbtrfsutil/python/tests/__init__.py   |  66 ++
 libbtrfsutil/python/tests/test_subvolume.py |  57 
 libbtrfsutil/subvolume.c| 137 
 9 files changed, 383 insertions(+), 1 deletion(-)
 create mode 100644 libbtrfsutil/python/subvolume.c
 create mode 100644 libbtrfsutil/python/tests/test_subvolume.py
 create mode 100644 libbtrfsutil/subvolume.c

diff --git a/Makefile b/Makefile
index 02b03e81..48a558a9 100644
--- a/Makefile
+++ b/Makefile
@@ -123,7 +123,7 @@ libbtrfs_headers = send-stream.h send-utils.h send.h 
kernel-lib/rbtree.h btrfs-l
   kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
   extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
 libbtrfsutil_version := 0.1
-libbtrfsutil_objects = libbtrfsutil/errors.o
+libbtrfsutil_objects = libbtrfsutil/errors.o libbtrfsutil/subvolume.o
 convert_objects = convert/main.o convert/common.o convert/source-fs.o \
  convert/source-ext2.o convert/source-reiserfs.o
 mkfs_objects = mkfs/main.o mkfs/common.o
diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index fe1091ca..dff6599d 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -20,6 +20,8 @@
 #ifndef BTRFS_UTIL_H
 #define BTRFS_UTIL_H
 
+#include 
+
 #ifdef __cplusplus
 extern "C" {
 #endif
@@ -65,6 +67,37 @@ enum btrfs_util_error {
  */
 const char *btrfs_util_strerror(enum btrfs_util_error err);
 
+/**
+ * btrfs_util_is_subvolume() - Return whether a given path is a Btrfs 
subvolume.
+ * @path: Path to check.
+ *
+ * Return: %BTRFS_UTIL_OK if @path is a Btrfs subvolume,
+ * %BTRFS_UTIL_ERROR_NOT_BTRFS if @path is not on a Btrfs filesystem,
+ * %BTRFS_UTIL_ERROR_NOT_SUBVOLUME if @path is not a subvolume, non-zero error
+ * code on any other failure.
+ */
+enum btrfs_util_error btrfs_util_is_subvolume(const char *path);
+
+/**
+ * btrfs_util_f_is_subvolume() - See btrfs_util_is_subvolume().
+ */
+enum btrfs_util_error btrfs_util_f_is_subvolume(int fd);
+
+/**
+ * btrfs_util_subvolume_id() - Get the ID of the subvolume containing a path.
+ * @path: Path on a Btrfs filesystem.
+ * @id_ret: Returned subvolume ID.
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_subvolume_id(const char *path,
+ uint64_t *id_ret);
+
+/**
+ * btrfs_util_f_subvolume_id() - See btrfs_util_subvolume_id().
+ */
+enum btrfs_util_error btrfs_util_f_subvolume_id(int fd, uint64_t *id_ret);
+
 #ifdef __cplusplus
 }
 #endif
diff --git a/libbtrfsutil/python/btrfsutilpy.h 
b/libbtrfsutil/python/btrfsutilpy.h
index 6d82f7e1..9a04fda7 100644
--- a/libbtrfsutil/python/btrfsutilpy.h
+++ b/libbtrfsutil/python/btrfsutilpy.h
@@ -52,6 +52,9 @@ void SetFromBtrfsUtilErrorWithPaths(enum btrfs_util_error err,
struct path_arg *path1,
struct path_arg *path2);
 
+PyObject *is_subvolume(PyObject *self, PyObject *args, PyObject *kwds);
+PyObject *subvolume_id(PyObject *self, PyObject *args, PyObject *kwds);
+
 void add_module_constants(PyObject *m);
 
 #endif /* BTRFSUTILPY_H */
diff --git a/libbtrfsutil/python/module.c b/libbtrfsutil/python/module.c
index d7398808..d492cbc7 100644
--- a/libbtrfsutil/python/module.c
+++ b/libbtrfsutil/python/module.c
@@ -132,6 +132,18 @@ void path_cleanup(struct path_arg *path)
 }
 
 static PyMethodDef btrfsutil_methods[] = {
+   {"is_subvolume", (PyCFunction)is_subvolume,
+METH_VARARGS | METH_KEYWORDS,
+"is_subvolume(path) -> bool\n\n"
+"Get whether a file is a subvolume.\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor"},
+   {"subvolume_id", (PyCFunction)subvolume_id,
+METH_VARARGS | METH_KEYWORDS,
+"subvolume_id(path) -> int\n\n"
+"Get the ID of the subvolume containing a file.\n\n"
+"Arguments:\n"
+"path -- string, bytes, path-like object, or open file descriptor"},
{},
 };
 
diff --git a/libbtrfsutil/python/setup.py b/libbtrfsutil/python/setup.py
index 3dc778ab..be973a34 100755
--- a/libbtrfsutil/python/setup.py
+++ b/libbtrfsutil/python/setup.py
@@ -79,6 +79,7 @@ module = Extension(
 'constants.c',
 'error.c',
 'module.c',
+'subvolume.c',
 ],
 

[PATCH 02/26] Add libbtrfsutil

[Omar Sandoval]

From: Omar Sandoval 

Currently, users wishing to manage Btrfs filesystems programatically
have to shell out to btrfs-progs and parse the output. This isn't ideal.
The goal of libbtrfsutil is to provide a library version of as many of
the operations of btrfs-progs as possible and to migrate btrfs-progs to
use it.

Rather than simply refactoring the existing btrfs-progs code, the code
has to be written from scratch for a couple of reasons:

* A lot of the btrfs-progs code was not designed with a nice library API
  in mind in terms of reusability, naming, and error reporting.
* libbtrfsutil is licensed under the LGPL, whereas btrfs-progs is under
  the GPL, which makes it dubious to directly copy or move the code.

Eventually, most of the low-level btrfs-progs code should either live in
libbtrfsutil or the shared kernel/userspace filesystem code, and
btrfs-progs will just be the CLI wrapper.

This first commit just includes the build system changes, license,
README, and error reporting helper.

Signed-off-by: Omar Sandoval 
---
 .gitignore  |   2 +
 Makefile|  47 +--
 libbtrfsutil/COPYING| 674 
 libbtrfsutil/COPYING.LESSER | 165 +++
 libbtrfsutil/README.md  |  32 +++
 libbtrfsutil/btrfsutil.h|  72 +
 libbtrfsutil/errors.c   |  55 
 7 files changed, 1031 insertions(+), 16 deletions(-)
 create mode 100644 libbtrfsutil/COPYING
 create mode 100644 libbtrfsutil/COPYING.LESSER
 create mode 100644 libbtrfsutil/README.md
 create mode 100644 libbtrfsutil/btrfsutil.h
 create mode 100644 libbtrfsutil/errors.c

diff --git a/.gitignore b/.gitignore
index 8e607f6e..272d53e4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -43,6 +43,8 @@ libbtrfs.so.0.1
 library-test
 library-test-static
 /fssum
+/libbtrfsutil.so*
+/libbtrfsutil.a
 
 /tests/*-tests-results.txt
 /tests/test-console.txt
diff --git a/Makefile b/Makefile
index 6369e8f4..062f7f3c 100644
--- a/Makefile
+++ b/Makefile
@@ -73,6 +73,7 @@ CFLAGS = $(SUBST_CFLAGS) \
 -fPIC \
 -I$(TOPDIR) \
 -I$(TOPDIR)/kernel-lib \
+-I$(TOPDIR)/libbtrfsutil \
 $(EXTRAWARN_CFLAGS) \
 $(DEBUG_CFLAGS_INTERNAL) \
 $(EXTRA_CFLAGS)
@@ -121,12 +122,14 @@ libbtrfs_headers = send-stream.h send-utils.h send.h 
kernel-lib/rbtree.h btrfs-l
   kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
   kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
   extent-cache.h extent_io.h ioctl.h ctree.h btrfsck.h version.h
+libbtrfsutil_version := 0.1
+libbtrfsutil_objects = libbtrfsutil/errors.o
 convert_objects = convert/main.o convert/common.o convert/source-fs.o \
  convert/source-ext2.o convert/source-reiserfs.o
 mkfs_objects = mkfs/main.o mkfs/common.o
 image_objects = image/main.o image/sanitize.o
 all_objects = $(objects) $(cmds_objects) $(libbtrfs_objects) 
$(convert_objects) \
- $(mkfs_objects) $(image_objects)
+ $(mkfs_objects) $(image_objects) $(libbtrfsutil_objects)
 
 TESTS = fsck-tests.sh convert-tests.sh
 
@@ -248,10 +251,10 @@ static_convert_objects = $(patsubst %.o, %.static.o, 
$(convert_objects))
 static_mkfs_objects = $(patsubst %.o, %.static.o, $(mkfs_objects))
 static_image_objects = $(patsubst %.o, %.static.o, $(image_objects))
 
-libs_shared = libbtrfs.so.0.1
-libs_static = libbtrfs.a
+libs_shared = libbtrfs.so.0.1 libbtrfsutil.so.$(libbtrfsutil_version)
+libs_static = libbtrfs.a libbtrfsutil.a
 libs = $(libs_shared) $(libs_static)
-lib_links = libbtrfs.so.0 libbtrfs.so
+lib_links = libbtrfs.so.0 libbtrfs.so libbtrfsutil.so.0 libbtrfsutil.so
 headers = $(libbtrfs_headers)
 
 # make C=1 to enable sparse
@@ -289,7 +292,7 @@ endif
$(Q)$(CC) $(STATIC_CFLAGS) -c $< -o $@ $($(subst 
-,_,$(@:%.static.o=%)-cflags)) \
$($(subst -,_,btrfs-$(@:%/$(notdir $@)=%)-cflags))
 
-all: $(progs) libbtrfs $(BUILDDIRS)
+all: $(progs) $(libs) $(lib_links) $(BUILDDIRS)
 $(SUBDIRS): $(BUILDDIRS)
 $(BUILDDIRS):
@echo "Making all in $(patsubst build-%,%,$@)"
@@ -353,20 +356,31 @@ kernel-lib/tables.c:
@echo "[TABLE]  $@"
$(Q)./mktables > $@ || ($(RM) -f $@ && exit 1)
 
-libbtrfs: $(libs_shared) $(lib_links)
+libbtrfs.so.0.1: $(libbtrfs_objects)
+   @echo "[LD] $@"
+   $(Q)$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LIBBTRFS_LIBS) \
+   -shared -Wl,-soname,libbtrfs.so.0 -o $@
+
+libbtrfs.a: $(libbtrfs_objects)
+   @echo "[AR] $@"
+   $(Q)$(AR) cr $@ $^
+
+libbtrfs.so.0 libbtrfs.so: libbtrfs.so.0.1
+   @echo "[LN] $@"
+   $(Q)$(LN_S) -f $< $@
 
-$(libs_shared): $(libbtrfs_objects) $(lib_links) send.h
+libbtrfsutil.so.$(libbtrfsutil_version): $(libbtrfsutil_objects)
@echo "[LD] $@"
-   $(Q)$(CC) $(CFLAGS) $(libbtrfs_objects) $(LDFLAGS) $(LIBBTRFS_LIBS) \
-   -shared -Wl,-soname,libbtrfs.so.0 -o 

[PATCH 22/26] btrfs-progs: use libbtrfsutil for subvol sync

[Omar Sandoval]

From: Omar Sandoval 

btrfs_util_f_deleted_subvolumes() replaces enumerate_dead_subvols() and
btrfs_util_f_subvolume_info() replaces is_subvolume_cleaned().

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 217 ++-
 1 file changed, 21 insertions(+), 196 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index b969fc88..8dbb3e49 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -42,38 +42,11 @@
 #include "utils.h"
 #include "help.h"
 
-static int is_subvolume_cleaned(int fd, u64 subvolid)
+static int wait_for_subvolume_cleaning(int fd, size_t count, uint64_t *ids,
+  int sleep_interval)
 {
-   int ret;
-   struct btrfs_ioctl_search_args args;
-   struct btrfs_ioctl_search_key *sk = 
-
-   sk->tree_id = BTRFS_ROOT_TREE_OBJECTID;
-   sk->min_objectid = subvolid;
-   sk->max_objectid = subvolid;
-   sk->min_type = BTRFS_ROOT_ITEM_KEY;
-   sk->max_type = BTRFS_ROOT_ITEM_KEY;
-   sk->min_offset = 0;
-   sk->max_offset = (u64)-1;
-   sk->min_transid = 0;
-   sk->max_transid = (u64)-1;
-   sk->nr_items = 1;
-
-   ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, );
-   if (ret < 0)
-   return -errno;
-
-   if (sk->nr_items == 0)
-   return 1;
-
-   return 0;
-}
-
-static int wait_for_subvolume_cleaning(int fd, int count, u64 *ids,
-   int sleep_interval)
-{
-   int ret;
-   int i;
+   size_t i;
+   enum btrfs_util_error err;
 
while (1) {
int clean = 1;
@@ -81,16 +54,14 @@ static int wait_for_subvolume_cleaning(int fd, int count, 
u64 *ids,
for (i = 0; i < count; i++) {
if (!ids[i])
continue;
-   ret = is_subvolume_cleaned(fd, ids[i]);
-   if (ret < 0) {
-   error(
-   "cannot read status of dead subvolume %llu: %s",
-   (unsigned long long)ids[i], 
strerror(-ret));
-   return ret;
-   }
-   if (ret) {
-   printf("Subvolume id %llu is gone\n", ids[i]);
+   err = btrfs_util_f_subvolume_info(fd, ids[i], NULL);
+   if (err == BTRFS_UTIL_ERROR_SUBVOLUME_NOT_FOUND) {
+   printf("Subvolume id %" PRIu64 " is gone\n",
+  ids[i]);
ids[i] = 0;
+   } else if (err) {
+   error_btrfs_util(err);
+   return -errno;
} else {
clean = 0;
}
@@ -1031,160 +1002,15 @@ static const char * const cmd_subvol_sync_usage[] = {
NULL
 };
 
-#if 0
-/*
- * If we're looking for any dead subvolume, take a shortcut and look
- * for any ORPHAN_ITEMs in the tree root
- */
-static int fs_has_dead_subvolumes(int fd)
-{
-   int ret;
-   struct btrfs_ioctl_search_args args;
-   struct btrfs_ioctl_search_key *sk = 
-   struct btrfs_ioctl_search_header sh;
-   u64 min_subvolid = 0;
-
-again:
-   sk->tree_id = BTRFS_ROOT_TREE_OBJECTID;
-   sk->min_objectid = BTRFS_ORPHAN_OBJECTID;
-   sk->max_objectid = BTRFS_ORPHAN_OBJECTID;
-   sk->min_type = BTRFS_ORPHAN_ITEM_KEY;
-   sk->max_type = BTRFS_ORPHAN_ITEM_KEY;
-   sk->min_offset = min_subvolid;
-   sk->max_offset = (u64)-1;
-   sk->min_transid = 0;
-   sk->max_transid = (u64)-1;
-   sk->nr_items = 1;
-
-   ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, );
-   if (ret < 0)
-   return -errno;
-
-   if (!sk->nr_items)
-   return 0;
-
-   memcpy(, args.buf, sizeof(sh));
-   min_subvolid = sh.offset;
-
-   /*
-* Verify that the root item is really there and we haven't hit
-* a stale orphan
-*/
-   sk->tree_id = BTRFS_ROOT_TREE_OBJECTID;
-   sk->min_objectid = min_subvolid;
-   sk->max_objectid = min_subvolid;
-   sk->min_type = BTRFS_ROOT_ITEM_KEY;
-   sk->max_type = BTRFS_ROOT_ITEM_KEY;
-   sk->min_offset = 0;
-   sk->max_offset = (u64)-1;
-   sk->min_transid = 0;
-   sk->max_transid = (u64)-1;
-   sk->nr_items = 1;
-
-   ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, );
-   if (ret < 0)
-   return -errno;
-
-   /*
-* Stale orphan, try the next one
-*/
-   if (!sk->nr_items) {
-   min_subvolid++;
-   goto again;
-   }
-
-   return 1;
-}
-#endif
-
-#define SUBVOL_ID_BATCH1024
-
-/*
- * Enumerate all dead subvolumes that exist in the filesystem.
- * Fill @ids and reallocate to bigger size if needed.
- */
-static int 

[PATCH 18/26] btrfs-progs: use libbtrfsutil for get-default

[Omar Sandoval]

From: Omar Sandoval 

The only thing of note here is the "top level" column. This used to mean
something else, but for a long time it has been equal to the parent ID.
I preserved this for backwards compatability.

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 55 ++-
 1 file changed, 26 insertions(+), 29 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index aaf88af8..557108c1 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -14,6 +14,7 @@
  * Boston, MA 021110-1307, USA.
  */
 
+#include 
 #include 
 #include 
 #include 
@@ -788,32 +789,25 @@ static const char * const cmd_subvol_get_default_usage[] 
= {
 static int cmd_subvol_get_default(int argc, char **argv)
 {
int fd = -1;
-   int ret;
-   char *subvol;
-   struct btrfs_list_filter_set *filter_set;
-   u64 default_id;
+   int ret = 1;
+   uint64_t default_id;
DIR *dirstream = NULL;
+   enum btrfs_util_error err;
+   struct btrfs_util_subvolume_info subvol;
+   char *path;
 
clean_args_no_options(argc, argv, cmd_subvol_get_default_usage);
 
if (check_argc_exact(argc - optind, 1))
usage(cmd_subvol_get_default_usage);
 
-   subvol = argv[1];
-   fd = btrfs_open_dir(subvol, , 1);
+   fd = btrfs_open_dir(argv[1], , 1);
if (fd < 0)
return 1;
 
-   ret = btrfs_list_get_default_subvolume(fd, _id);
-   if (ret) {
-   error("failed to look up default subvolume: %s",
-   strerror(errno));
-   goto out;
-   }
-
-   ret = 1;
-   if (default_id == 0) {
-   error("'default' dir item not found");
+   err = btrfs_util_f_get_default_subvolume(fd, _id);
+   if (err) {
+   error_btrfs_util(err);
goto out;
}
 
@@ -824,24 +818,27 @@ static int cmd_subvol_get_default(int argc, char **argv)
goto out;
}
 
-   filter_set = btrfs_list_alloc_filter_set();
-   btrfs_list_setup_filter(_set, BTRFS_LIST_FILTER_ROOTID,
-   default_id);
+   err = btrfs_util_f_subvolume_info(fd, default_id, );
+   if (err) {
+   error_btrfs_util(err);
+   goto out;
+   }
 
-   /* by default we shall print the following columns*/
-   btrfs_list_setup_print_column(BTRFS_LIST_OBJECTID);
-   btrfs_list_setup_print_column(BTRFS_LIST_GENERATION);
-   btrfs_list_setup_print_column(BTRFS_LIST_TOP_LEVEL);
-   btrfs_list_setup_print_column(BTRFS_LIST_PATH);
+   err = btrfs_util_f_subvolume_path(fd, default_id, );
+   if (err) {
+   error_btrfs_util(err);
+   goto out;
+   }
 
-   ret = btrfs_list_subvols_print(fd, filter_set, NULL,
-   BTRFS_LIST_LAYOUT_DEFAULT, 1, NULL);
+   printf("ID %" PRIu64 " gen %" PRIu64 " top level %" PRIu64 " path %s\n",
+  subvol.id, subvol.generation, subvol.parent_id, path);
 
-   if (filter_set)
-   free(filter_set);
+   free(path);
+
+   ret = 0;
 out:
close_file_or_dir(fd, dirstream);
-   return !!ret;
+   return ret;
 }
 
 static const char * const cmd_subvol_set_default_usage[] = {
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 21/26] btrfs-progs: use libbtrfsutil for subvol show

[Omar Sandoval]

From: Omar Sandoval 

Now implemented with btrfs_util_subvolume_path(),
btrfs_util_subvolume_info(), and subvolume iterators.

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 150 ---
 utils.c  | 118 ---
 utils.h  |   5 --
 3 files changed, 99 insertions(+), 174 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index c5e03011..b969fc88 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -825,19 +825,20 @@ static const char * const cmd_subvol_show_usage[] = {
 
 static int cmd_subvol_show(int argc, char **argv)
 {
-   struct root_info get_ri;
-   struct btrfs_list_filter_set *filter_set = NULL;
char tstr[256];
char uuidparse[BTRFS_UUID_UNPARSED_SIZE];
char *fullpath = NULL;
-   char raw_prefix[] = "\t\t\t\t";
int fd = -1;
int ret = 1;
DIR *dirstream1 = NULL;
int by_rootid = 0;
int by_uuid = 0;
-   u64 rootid_arg;
+   u64 rootid_arg = 0;
u8 uuid_arg[BTRFS_UUID_SIZE];
+   struct btrfs_util_subvolume_iterator *iter;
+   struct btrfs_util_subvolume_info subvol;
+   char *subvol_path = NULL;
+   enum btrfs_util_error err;
 
while (1) {
int c;
@@ -874,7 +875,6 @@ static int cmd_subvol_show(int argc, char **argv)
usage(cmd_subvol_show_usage);
}
 
-   memset(_ri, 0, sizeof(get_ri));
fullpath = realpath(argv[optind], NULL);
if (!fullpath) {
error("cannot find real path for '%s': %s",
@@ -882,89 +882,137 @@ static int cmd_subvol_show(int argc, char **argv)
goto out;
}
 
-   if (by_rootid) {
-   ret = get_subvol_info_by_rootid(fullpath, _ri, rootid_arg);
-   } else if (by_uuid) {
-   ret = get_subvol_info_by_uuid(fullpath, _ri, uuid_arg);
-   } else {
-   ret = get_subvol_info(fullpath, _ri);
+   fd = open_file_or_dir(fullpath, );
+   if (fd < 0) {
+   error("can't access '%s'", fullpath);
+   goto out;
}
 
-   if (ret) {
-   if (ret < 0) {
-   error("Failed to get subvol info %s: %s",
-   fullpath, strerror(-ret));
-   } else {
-   error("Failed to get subvol info %s: %d",
-   fullpath, ret);
+   if (by_uuid) {
+   err = btrfs_util_f_create_subvolume_iterator(fd,
+
BTRFS_FS_TREE_OBJECTID,
+0, );
+   if (err) {
+   error_btrfs_util(err);
+   goto out;
+   }
+
+   for (;;) {
+   err = btrfs_util_subvolume_iterator_next_info(iter,
+ 
_path,
+ );
+   if (err == BTRFS_UTIL_ERROR_STOP_ITERATION) {
+   uuid_unparse(uuid_arg, uuidparse);
+   error("can't find uuid '%s' on '%s'", uuidparse,
+ fullpath);
+   btrfs_util_destroy_subvolume_iterator(iter);
+   goto out;
+   } else if (err) {
+   error_btrfs_util(err);
+   btrfs_util_destroy_subvolume_iterator(iter);
+   goto out;
+   }
+
+   if (uuid_compare(subvol.uuid, uuid_arg) == 0)
+   break;
+
+   free(subvol_path);
+   }
+   btrfs_util_destroy_subvolume_iterator(iter);
+   } else {
+   /*
+* If !by_rootid, rootid_arg = 0, which means find the
+* subvolume ID of the fd and use that.
+*/
+   err = btrfs_util_f_subvolume_info(fd, rootid_arg,
+ );
+   if (err) {
+   error_btrfs_util(err);
+   goto out;
+   }
+
+   err = btrfs_util_f_subvolume_path(fd, subvol.id, _path);
+   if (err) {
+   error_btrfs_util(err);
+   goto out;
}
-   return ret;
+
}
 
/* print the info */
-   printf("%s\n", get_ri.full_path);
-   printf("\tName: \t\t\t%s\n", get_ri.name);
+   printf("%s\n", subvol.id == BTRFS_FS_TREE_OBJECTID ? "/" : subvol_path);
+   printf("\tName: \t\t\t%s\n",
+  (subvol.id == BTRFS_FS_TREE_OBJECTID ? "" :
+   

[PATCH 10/26] libbtrfsutil: add subvolume iterator helpers

[Omar Sandoval]

From: Omar Sandoval 

This is how we can implement stuff like `btrfs subvol list`. Rather than
producing the entire list upfront, the iterator approach uses less
memory in the common case where the whole list is not stored (O(max
subvolume path length)). It supports both pre-order traversal (useful
for, e.g, recursive snapshot) and post-order traversal (useful for
recursive delete).

Signed-off-by: Omar Sandoval 
---
 libbtrfsutil/btrfsutil.h|  92 
 libbtrfsutil/python/btrfsutilpy.h   |   1 +
 libbtrfsutil/python/module.c|   8 +
 libbtrfsutil/python/subvolume.c | 211 ++
 libbtrfsutil/python/tests/test_subvolume.py |  56 +
 libbtrfsutil/subvolume.c| 322 
 6 files changed, 690 insertions(+)

diff --git a/libbtrfsutil/btrfsutil.h b/libbtrfsutil/btrfsutil.h
index 1643ea7b..3f78a34d 100644
--- a/libbtrfsutil/btrfsutil.h
+++ b/libbtrfsutil/btrfsutil.h
@@ -332,6 +332,98 @@ enum btrfs_util_error btrfs_util_f_create_subvolume(int 
parent_fd,
uint64_t *async_transid,
struct 
btrfs_util_qgroup_inherit *qgroup_inherit);
 
+struct btrfs_util_subvolume_iterator;
+
+/**
+ * BTRFS_UTIL_SUBVOLUME_ITERATOR_POST_ORDER - Iterate post-order. The default
+ * behavior is pre-order, e.g., foo will be yielded before foo/bar. If this 
flag
+ * is specified, foo/bar will be yielded before foo.
+ */
+#define BTRFS_UTIL_SUBVOLUME_ITERATOR_POST_ORDER (1 << 0)
+#define BTRFS_UTIL_SUBVOLUME_ITERATOR_MASK ((1 << 1) - 1)
+
+/**
+ * btrfs_util_create_subvolume_iterator() - Create an iterator over subvolumes
+ * in a Btrfs filesystem.
+ * @path: Path in a Btrfs filesystem. This may be any path in the filesystem; 
it
+ * does not have to refer to a subvolume unless @top is zero.
+ * @top: List subvolumes beneath (but not including) the subvolume with this 
ID.
+ * If zero is given, the subvolume ID of @path is used. To list all subvolumes,
+ * pass %BTRFS_FS_TREE_OBJECTID (i.e., 5). The returned paths are relative to
+ * the subvolume with this ID.
+ * @flags: Bitmask of BTRFS_UTIL_SUBVOLUME_ITERATOR_* flags.
+ * @ret: Returned iterator.
+ *
+ * The returned iterator must be freed with
+ * btrfs_util_destroy_subvolume_iterator().
+ *
+ * Return: %BTRFS_UTIL_OK on success, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_create_subvolume_iterator(const char *path,
+  uint64_t top,
+  int flags,
+  struct 
btrfs_util_subvolume_iterator **ret);
+
+/**
+ * btrfs_util_f_create_subvolume_iterator() - See
+ * btrfs_util_create_subvolume_iterator().
+ */
+enum btrfs_util_error btrfs_util_f_create_subvolume_iterator(int fd, uint64_t 
top,
+int flags,
+struct 
btrfs_util_subvolume_iterator **ret);
+
+/**
+ * btrfs_util_destroy_subvolume_iterator() - Destroy a subvolume iterator
+ * previously created by btrfs_util_create_subvolume_iterator().
+ * @iter: Iterator to destroy.
+ */
+void btrfs_util_destroy_subvolume_iterator(struct 
btrfs_util_subvolume_iterator *iter);
+
+/**
+ * btrfs_util_subvolume_iterator_fd() - Get the file descriptor associated with
+ * a subvolume iterator.
+ * @iter: Iterator to get.
+ *
+ * This can be used to get the file descriptor opened by
+ * btrfs_util_create_subvolume_iterator() in order to use it for other
+ * functions.
+ *
+ * Return: File descriptor.
+ */
+int btrfs_util_subvolume_iterator_fd(const struct 
btrfs_util_subvolume_iterator *iter);
+
+/**
+ * btrfs_util_subvolume_iterator_next() - Get the next subvolume from a
+ * subvolume iterator.
+ * @iter: Subvolume iterator.
+ * @path_ret: Returned subvolume path, relative to the subvolume ID used to
+ * create the iterator. May be %NULL.
+ * Must be freed with free().
+ * @id_ret: Returned subvolume ID. May be %NULL.
+ *
+ * Return: %BTRFS_UTIL_OK on success, %BTRFS_UTIL_ERROR_STOP_ITERATION if there
+ * are no more subvolumes, non-zero error code on failure.
+ */
+enum btrfs_util_error btrfs_util_subvolume_iterator_next(struct 
btrfs_util_subvolume_iterator *iter,
+char **path_ret,
+uint64_t *id_ret);
+
+/**
+ * btrfs_util_subvolume_iterator_next_info() - Get information about the next
+ * subvolume for a subvolume iterator.
+ * @iter: Subvolume iterator.
+ * @path_ret: See btrfs_util_subvolume_iterator_next().
+ * @subvol: Returned subvolume information.
+ *
+ * This convenience function basically combines
+ * btrfs_util_subvolume_iterator_next() and btrfs_util_subvolume_info().
+ *
+ * 

[PATCH 26/26] btrfs-progs: use libbtrfsutil for subvolume list

[Omar Sandoval]

From: Omar Sandoval 

This is the most involved conversion because it replaces the libbtrfs
implementation entirely. I took care to preserve the existing behavior
in all cases, warts and all.

Signed-off-by: Omar Sandoval 
---
 btrfs-list.c | 953 +--
 btrfs-list.h |   7 +-
 2 files changed, 268 insertions(+), 692 deletions(-)

diff --git a/btrfs-list.c b/btrfs-list.c
index 267aef98..d9aef8fc 100644
--- a/btrfs-list.c
+++ b/btrfs-list.c
@@ -16,6 +16,7 @@
  * Boston, MA 021110-1307, USA.
  */
 
+#include 
 #include 
 #include 
 #include 
@@ -32,18 +33,20 @@
 #include "ioctl.h"
 #include 
 #include "btrfs-list.h"
-#include "rbtree-utils.h"
 
 #include 
 
 #define BTRFS_LIST_NFILTERS_INCREASE   (2 * BTRFS_LIST_FILTER_MAX)
 #define BTRFS_LIST_NCOMPS_INCREASE (2 * BTRFS_LIST_COMP_MAX)
 
-/* we store all the roots we find in an rbtree so that we can
- * search for them later.
- */
-struct root_lookup {
-   struct rb_root root;
+struct listed_subvol {
+   struct btrfs_util_subvolume_info info;
+   char *path;
+};
+
+struct subvol_list {
+   size_t num;
+   struct listed_subvol subvols[];
 };
 
 static struct {
@@ -126,15 +129,15 @@ void btrfs_list_setup_print_column(enum 
btrfs_list_column_enum column)
btrfs_list_columns[i].need_print = 1;
 }
 
-static int comp_entry_with_rootid(struct root_info *entry1,
- struct root_info *entry2,
+static int comp_entry_with_rootid(const struct listed_subvol *entry1,
+ const struct listed_subvol *entry2,
  int is_descending)
 {
int ret;
 
-   if (entry1->root_id > entry2->root_id)
+   if (entry1->info.id > entry2->info.id)
ret = 1;
-   else if (entry1->root_id < entry2->root_id)
+   else if (entry1->info.id < entry2->info.id)
ret = -1;
else
ret = 0;
@@ -142,15 +145,15 @@ static int comp_entry_with_rootid(struct root_info 
*entry1,
return is_descending ? -ret : ret;
 }
 
-static int comp_entry_with_gen(struct root_info *entry1,
-  struct root_info *entry2,
+static int comp_entry_with_gen(const struct listed_subvol *entry1,
+  const struct listed_subvol *entry2,
   int is_descending)
 {
int ret;
 
-   if (entry1->gen > entry2->gen)
+   if (entry1->info.generation > entry2->info.generation)
ret = 1;
-   else if (entry1->gen < entry2->gen)
+   else if (entry1->info.generation < entry2->info.generation)
ret = -1;
else
ret = 0;
@@ -158,15 +161,15 @@ static int comp_entry_with_gen(struct root_info *entry1,
return is_descending ? -ret : ret;
 }
 
-static int comp_entry_with_ogen(struct root_info *entry1,
-   struct root_info *entry2,
+static int comp_entry_with_ogen(const struct listed_subvol *entry1,
+   const struct listed_subvol *entry2,
int is_descending)
 {
int ret;
 
-   if (entry1->ogen > entry2->ogen)
+   if (entry1->info.otransid > entry2->info.otransid)
ret = 1;
-   else if (entry1->ogen < entry2->ogen)
+   else if (entry1->info.otransid < entry2->info.otransid)
ret = -1;
else
ret = 0;
@@ -174,15 +177,15 @@ static int comp_entry_with_ogen(struct root_info *entry1,
return is_descending ? -ret : ret;
 }
 
-static int comp_entry_with_path(struct root_info *entry1,
-   struct root_info *entry2,
+static int comp_entry_with_path(const struct listed_subvol *entry1,
+   const struct listed_subvol *entry2,
int is_descending)
 {
int ret;
 
-   if (strcmp(entry1->full_path, entry2->full_path) > 0)
+   if (strcmp(entry1->path, entry2->path) > 0)
ret = 1;
-   else if (strcmp(entry1->full_path, entry2->full_path) < 0)
+   else if (strcmp(entry1->path, entry2->path) < 0)
ret = -1;
else
ret = 0;
@@ -272,16 +275,14 @@ static int btrfs_list_setup_comparer(struct 
btrfs_list_comparer_set **comp_set,
return 0;
 }
 
-static int sort_comp(struct root_info *entry1, struct root_info *entry2,
-struct btrfs_list_comparer_set *set)
+static int subvol_comp(const void *entry1, const void *entry2, void *arg)
 {
+   const struct btrfs_list_comparer_set * const set = arg;
int rootid_compared = 0;
-   int i, ret = 0;
-
-   if (!set || !set->ncomps)
-   return comp_entry_with_rootid(entry1, entry2, 0);
+   int ret;
+   int i;
 
-   for (i = 0; i < set->ncomps; i++) {
+   for (i = 0; set && i < set->ncomps; i++) {
   

[PATCH 23/26] btrfs-progs: replace test_issubvolume() with btrfs_util_is_subvolume()

[Omar Sandoval]

From: Omar Sandoval 

This gets the remaining occurrences that weren't covered by previous
conversions.

Signed-off-by: Omar Sandoval 
---
 cmds-qgroup.c| 13 ++---
 cmds-subvolume.c | 11 ---
 utils.c  | 34 +-
 utils.h  |  1 -
 4 files changed, 15 insertions(+), 44 deletions(-)

diff --git a/cmds-qgroup.c b/cmds-qgroup.c
index fac692d4..16ba6f86 100644
--- a/cmds-qgroup.c
+++ b/cmds-qgroup.c
@@ -20,6 +20,8 @@
 #include 
 #include 
 
+#include 
+
 #include "ctree.h"
 #include "ioctl.h"
 
@@ -431,6 +433,7 @@ static int cmd_qgroup_limit(int argc, char **argv)
int compressed = 0;
int exclusive = 0;
DIR *dirstream = NULL;
+   enum btrfs_util_error err;
 
while (1) {
int c = getopt(argc, argv, "ce");
@@ -471,13 +474,9 @@ static int cmd_qgroup_limit(int argc, char **argv)
if (argc - optind == 2) {
args.qgroupid = 0;
path = argv[optind + 1];
-   ret = test_issubvolume(path);
-   if (ret < 0) {
-   error("cannot access '%s': %s", path, strerror(-ret));
-   return 1;
-   }
-   if (!ret) {
-   error("'%s' is not a subvolume", path);
+   err = btrfs_util_is_subvolume(path);
+   if (err) {
+   error_btrfs_util(err);
return 1;
}
/*
diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index 8dbb3e49..5c75799c 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -748,6 +748,7 @@ static int cmd_subvol_find_new(int argc, char **argv)
char *subvol;
u64 last_gen;
DIR *dirstream = NULL;
+   enum btrfs_util_error err;
 
clean_args_no_options(argc, argv, cmd_subvol_find_new_usage);
 
@@ -757,13 +758,9 @@ static int cmd_subvol_find_new(int argc, char **argv)
subvol = argv[optind];
last_gen = arg_strtou64(argv[optind + 1]);
 
-   ret = test_issubvolume(subvol);
-   if (ret < 0) {
-   error("cannot access subvolume %s: %s", subvol, strerror(-ret));
-   return 1;
-   }
-   if (!ret) {
-   error("not a subvolume: %s", subvol);
+   err = btrfs_util_is_subvolume(subvol);
+   if (err) {
+   error_btrfs_util(err);
return 1;
}
 
diff --git a/utils.c b/utils.c
index 6a667907..bdb389cf 100644
--- a/utils.c
+++ b/utils.c
@@ -40,6 +40,8 @@
 #include 
 #include 
 
+#include 
+
 #include "kerncompat.h"
 #include "radix-tree.h"
 #include "ctree.h"
@@ -1451,6 +1453,7 @@ u64 parse_qgroupid(const char *p)
char *s = strchr(p, '/');
const char *ptr_src_end = p + strlen(p);
char *ptr_parse_end = NULL;
+   enum btrfs_util_error err;
u64 level;
u64 id;
int fd;
@@ -1478,8 +1481,8 @@ u64 parse_qgroupid(const char *p)
 
 path:
/* Path format like subv at 'my_subvol' is the fallback case */
-   ret = test_issubvolume(p);
-   if (ret < 0 || !ret)
+   err = btrfs_util_is_subvolume(p);
+   if (err)
goto err;
fd = open(p, O_RDONLY);
if (fd < 0)
@@ -2443,33 +2446,6 @@ int test_issubvolname(const char *name)
strcmp(name, ".") && strcmp(name, "..");
 }
 
-/*
- * Test if path is a subvolume
- * Returns:
- *   0 - path exists but it is not a subvolume
- *   1 - path exists and it is  a subvolume
- * < 0 - error
- */
-int test_issubvolume(const char *path)
-{
-   struct stat st;
-   struct statfs stfs;
-   int res;
-
-   res = stat(path, );
-   if (res < 0)
-   return -errno;
-
-   if (st.st_ino != BTRFS_FIRST_FREE_OBJECTID || !S_ISDIR(st.st_mode))
-   return 0;
-
-   res = statfs(path, );
-   if (res < 0)
-   return -errno;
-
-   return (int)stfs.f_type == BTRFS_SUPER_MAGIC;
-}
-
 const char *subvol_strip_mountpoint(const char *mnt, const char *full_path)
 {
int len = strlen(mnt);
diff --git a/utils.h b/utils.h
index 4a95bfb5..4bfbf63e 100644
--- a/utils.h
+++ b/utils.h
@@ -147,7 +147,6 @@ u64 disk_size(const char *path);
 u64 get_partition_size(const char *dev);
 
 int test_issubvolname(const char *name);
-int test_issubvolume(const char *path);
 int test_isdir(const char *path);
 
 const char *subvol_strip_mountpoint(const char *mnt, const char *full_path);
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 25/26] btrfs-progs: deprecate libbtrfs helpers with libbtrfsutil equivalents

[Omar Sandoval]

From: Omar Sandoval 

The old libbtrfs defines some helpers which do the same thing as some
libbtrfsutil helpers. Reimplement the libbtrfs helpers in terms of the
libbtrfsutil APIs and mark the libbtrfs versions as deprecated, which we
could ideally get rid of eventually.

Signed-off-by: Omar Sandoval 
---
 Makefile |   4 +-
 btrfs-list.c | 252 +--
 btrfs-list.h |  15 ++--
 cmds-inspect.c   |   9 +-
 cmds-receive.c   |  12 ++-
 cmds-subvolume.c |  10 ++-
 send-utils.c |  25 --
 7 files changed, 146 insertions(+), 181 deletions(-)

diff --git a/Makefile b/Makefile
index 8bbe1c5e..b37d1ae9 100644
--- a/Makefile
+++ b/Makefile
@@ -372,10 +372,10 @@ kernel-lib/tables.c:
@echo "[TABLE]  $@"
$(Q)./mktables > $@ || ($(RM) -f $@ && exit 1)
 
-libbtrfs.so.0.1: $(libbtrfs_objects)
+libbtrfs.so.0.1: $(libbtrfs_objects) libbtrfsutil.so
@echo "[LD] $@"
$(Q)$(CC) $(CFLAGS) $^ $(LDFLAGS) $(LIBBTRFS_LIBS) \
-   -shared -Wl,-soname,libbtrfs.so.0 -o $@
+   -shared -Wl,-soname,libbtrfs.so.0 -o $@ -L. -lbtrfsutil
 
 libbtrfs.a: $(libbtrfs_objects)
@echo "[AR] $@"
diff --git a/btrfs-list.c b/btrfs-list.c
index b6d76585..267aef98 100644
--- a/btrfs-list.c
+++ b/btrfs-list.c
@@ -34,6 +34,8 @@
 #include "btrfs-list.h"
 #include "rbtree-utils.h"
 
+#include 
+
 #define BTRFS_LIST_NFILTERS_INCREASE   (2 * BTRFS_LIST_FILTER_MAX)
 #define BTRFS_LIST_NCOMPS_INCREASE (2 * BTRFS_LIST_COMP_MAX)
 
@@ -907,55 +909,14 @@ build:
 
 int btrfs_list_get_default_subvolume(int fd, u64 *default_id)
 {
-   struct btrfs_ioctl_search_args args;
-   struct btrfs_ioctl_search_key *sk = 
-   struct btrfs_ioctl_search_header *sh;
-   u64 found = 0;
-   int ret;
-
-   memset(, 0, sizeof(args));
-
-   /*
-* search for a dir item with a name 'default' in the tree of
-* tree roots, it should point us to a default root
-*/
-   sk->tree_id = BTRFS_ROOT_TREE_OBJECTID;
-
-   /* don't worry about ancient format and request only one item */
-   sk->nr_items = 1;
-
-   sk->max_objectid = BTRFS_ROOT_TREE_DIR_OBJECTID;
-   sk->min_objectid = BTRFS_ROOT_TREE_DIR_OBJECTID;
-   sk->max_type = BTRFS_DIR_ITEM_KEY;
-   sk->min_type = BTRFS_DIR_ITEM_KEY;
-   sk->max_offset = (u64)-1;
-   sk->max_transid = (u64)-1;
-
-   ret = ioctl(fd, BTRFS_IOC_TREE_SEARCH, );
-   if (ret < 0)
-   return ret;
-
-   /* the ioctl returns the number of items it found in nr_items */
-   if (sk->nr_items == 0)
-   goto out;
+   enum btrfs_util_error err;
+   uint64_t id;
 
-   sh = (struct btrfs_ioctl_search_header *)args.buf;
+   err = btrfs_util_f_get_default_subvolume(fd, );
+   if (err)
+   return -1;
 
-   if (btrfs_search_header_type(sh) == BTRFS_DIR_ITEM_KEY) {
-   struct btrfs_dir_item *di;
-   int name_len;
-   char *name;
-
-   di = (struct btrfs_dir_item *)(sh + 1);
-   name_len = btrfs_stack_dir_name_len(di);
-   name = (char *)(di + 1);
-
-   if (!strncmp("default", name, name_len))
-   found = btrfs_disk_key_objectid(>location);
-   }
-
-out:
-   *default_id = found;
+   *default_id = id;
return 0;
 }
 
@@ -1518,10 +1479,11 @@ int btrfs_list_subvols_print(int fd, struct 
btrfs_list_filter_set *filter_set,
int ret = 0;
u64 top_id = 0;
 
-   if (full_path)
-   ret = btrfs_list_get_path_rootid(fd, _id);
-   if (ret)
-   return ret;
+   /*
+* full_path hasn't done anything since 4f5ebb3ef553 ("Btrfs-progs: fix
+* to make list specified directory's subvolumes work"). See
+* https://www.spinics.net/lists/linux-btrfs/msg69820.html
+*/
 
ret = btrfs_list_subvols(fd, _lookup);
if (ret)
@@ -1535,83 +1497,84 @@ int btrfs_list_subvols_print(int fd, struct 
btrfs_list_filter_set *filter_set,
return 0;
 }
 
-static char *strdup_or_null(const char *s)
+static void ri_from_subvolume_info(struct root_info *ri,
+  const struct btrfs_util_subvolume_info 
*subvol)
 {
-   if (!s)
-   return NULL;
-   return strdup(s);
+   ri->root_id = subvol->id;
+   /*
+* struct btrfs_util_subvolume_info doesn't have the root item key
+* offset, but it is equal to otransid for snapshots and zero otherwise.
+*/
+   if (uuid_is_null(subvol->parent_uuid))
+   ri->root_offset = 0;
+   else
+   ri->root_offset = subvol->otransid;
+   ri->flags = subvol->flags;
+   ri->ref_tree = subvol->parent_id;
+   ri->dir_id = subvol->dir_id;
+   ri->top_id = subvol->parent_id;
+   ri->gen = subvol->generation;
+   

[PATCH 19/26] btrfs-progs: use libbtrfsutil for subvol create and snapshot

[Omar Sandoval]

From: Omar Sandoval 

These become trivial calls to the libbtrfsutil helpers, and we can get
rid of the qgroup inherit code in progs.

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 225 ++-
 qgroup.c |  64 
 qgroup.h |   2 -
 3 files changed, 58 insertions(+), 233 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index 557108c1..6b474fe4 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -103,6 +103,35 @@ static int wait_for_subvolume_cleaning(int fd, int count, 
u64 *ids,
return 0;
 }
 
+static int qgroup_inherit_add_group(struct btrfs_util_qgroup_inherit **inherit,
+   const char *arg)
+{
+   enum btrfs_util_error err;
+   u64 qgroupid;
+
+   if (!*inherit) {
+   err = btrfs_util_create_qgroup_inherit(0, inherit);
+   if (err) {
+   error_btrfs_util(err);
+   return -1;
+   }
+   }
+
+   qgroupid = parse_qgroupid(optarg);
+   if (qgroupid == 0) {
+   error("invalid qgroup specification, qgroupid must not be 0");
+   return -1;
+   }
+
+   err = btrfs_util_qgroup_inherit_add_group(inherit, qgroupid);
+   if (err) {
+   error_btrfs_util(err);
+   return -1;
+   }
+
+   return 0;
+}
+
 static const char * const subvolume_cmd_group_usage[] = {
"btrfs subvolume  ",
NULL
@@ -121,15 +150,9 @@ static const char * const cmd_subvol_create_usage[] = {
 
 static int cmd_subvol_create(int argc, char **argv)
 {
-   int retval, res, len;
-   int fddst = -1;
-   char*dupname = NULL;
-   char*dupdir = NULL;
-   char*newname;
-   char*dstdir;
-   char*dst;
-   struct btrfs_qgroup_inherit *inherit = NULL;
-   DIR *dirstream = NULL;
+   struct btrfs_util_qgroup_inherit *inherit = NULL;
+   enum btrfs_util_error err;
+   int retval = 1;
 
while (1) {
int c = getopt(argc, argv, "i:");
@@ -138,11 +161,8 @@ static int cmd_subvol_create(int argc, char **argv)
 
switch (c) {
case 'i':
-   res = qgroup_inherit_add_group(, optarg);
-   if (res) {
-   retval = res;
+   if (qgroup_inherit_add_group(, optarg) == -1)
goto out;
-   }
break;
default:
usage(cmd_subvol_create_usage);
@@ -152,70 +172,18 @@ static int cmd_subvol_create(int argc, char **argv)
if (check_argc_exact(argc - optind, 1))
usage(cmd_subvol_create_usage);
 
-   dst = argv[optind];
-
-   retval = 1; /* failure */
-   res = test_isdir(dst);
-   if (res < 0 && res != -ENOENT) {
-   error("cannot access %s: %s", dst, strerror(-res));
-   goto out;
-   }
-   if (res >= 0) {
-   error("target path already exists: %s", dst);
-   goto out;
-   }
-
-   dupname = strdup(dst);
-   newname = basename(dupname);
-   dupdir = strdup(dst);
-   dstdir = dirname(dupdir);
-
-   if (!test_issubvolname(newname)) {
-   error("invalid subvolume name: %s", newname);
-   goto out;
-   }
-
-   len = strlen(newname);
-   if (len == 0 || len >= BTRFS_VOL_NAME_MAX) {
-   error("subvolume name too long: %s", newname);
-   goto out;
-   }
-
-   fddst = btrfs_open_dir(dstdir, , 1);
-   if (fddst < 0)
-   goto out;
-
-   printf("Create subvolume '%s/%s'\n", dstdir, newname);
-   if (inherit) {
-   struct btrfs_ioctl_vol_args_v2  args;
-
-   memset(, 0, sizeof(args));
-   strncpy_null(args.name, newname);
-   args.flags |= BTRFS_SUBVOL_QGROUP_INHERIT;
-   args.size = qgroup_inherit_size(inherit);
-   args.qgroup_inherit = inherit;
-
-   res = ioctl(fddst, BTRFS_IOC_SUBVOL_CREATE_V2, );
-   } else {
-   struct btrfs_ioctl_vol_args args;
-
-   memset(, 0, sizeof(args));
-   strncpy_null(args.name, newname);
-
-   res = ioctl(fddst, BTRFS_IOC_SUBVOL_CREATE, );
-   }
+   printf("Create subvolume '%s'\n", argv[optind]);
 
-   if (res < 0) {
-   error("cannot create subvolume: %s", strerror(errno));
+   err = btrfs_util_create_subvolume(argv[optind], 0, NULL, inherit);
+   if (err) {
+   error_btrfs_util(err);
goto out;
}
 
-   retval = 0; /* success */
+   retval = 0;
 out:
-   close_file_or_dir(fddst, dirstream);
-   free(inherit);
-   free(dupname);
-   free(dupdir);
+ 

[PATCH 01/26] btrfs-progs: get rid of undocumented qgroup inheritance options

[Omar Sandoval]

From: Omar Sandoval 

The -c option to subvol create and the -c and -x options to subvol
snapshot have never been documented (and -x doesn't actually work
because it's not in the getopt option string). The functionality is
dubious and the kernel interface is being removed, so get rid of these.

Signed-off-by: Omar Sandoval 
---
 cmds-subvolume.c | 25 ++---
 qgroup.c | 42 --
 qgroup.h |  2 --
 3 files changed, 2 insertions(+), 67 deletions(-)

diff --git a/cmds-subvolume.c b/cmds-subvolume.c
index dc626a64..f8e34bc8 100644
--- a/cmds-subvolume.c
+++ b/cmds-subvolume.c
@@ -129,18 +129,11 @@ static int cmd_subvol_create(int argc, char **argv)
DIR *dirstream = NULL;
 
while (1) {
-   int c = getopt(argc, argv, "c:i:");
+   int c = getopt(argc, argv, "i:");
if (c < 0)
break;
 
switch (c) {
-   case 'c':
-   res = qgroup_inherit_add_copy(, optarg, 0);
-   if (res) {
-   retval = res;
-   goto out;
-   }
-   break;
case 'i':
res = qgroup_inherit_add_group(, optarg);
if (res) {
@@ -665,18 +658,11 @@ static int cmd_subvol_snapshot(int argc, char **argv)
 
memset(, 0, sizeof(args));
while (1) {
-   int c = getopt(argc, argv, "c:i:r");
+   int c = getopt(argc, argv, "i:r");
if (c < 0)
break;
 
switch (c) {
-   case 'c':
-   res = qgroup_inherit_add_copy(, optarg, 0);
-   if (res) {
-   retval = res;
-   goto out;
-   }
-   break;
case 'i':
res = qgroup_inherit_add_group(, optarg);
if (res) {
@@ -687,13 +673,6 @@ static int cmd_subvol_snapshot(int argc, char **argv)
case 'r':
readonly = 1;
break;
-   case 'x':
-   res = qgroup_inherit_add_copy(, optarg, 1);
-   if (res) {
-   retval = res;
-   goto out;
-   }
-   break;
default:
usage(cmd_subvol_snapshot_usage);
}
diff --git a/qgroup.c b/qgroup.c
index 156825fd..e9158a26 100644
--- a/qgroup.c
+++ b/qgroup.c
@@ -1310,45 +1310,3 @@ int qgroup_inherit_add_group(struct btrfs_qgroup_inherit 
**inherit, char *arg)
 
return 0;
 }
-
-int qgroup_inherit_add_copy(struct btrfs_qgroup_inherit **inherit, char *arg,
-   int type)
-{
-   int ret;
-   u64 qgroup_src;
-   u64 qgroup_dst;
-   char *p;
-   int pos = 0;
-
-   p = strchr(arg, ':');
-   if (!p) {
-bad:
-   error("invalid copy specification, missing separator :");
-   return -EINVAL;
-   }
-   *p = 0;
-   qgroup_src = parse_qgroupid(arg);
-   qgroup_dst = parse_qgroupid(p + 1);
-   *p = ':';
-
-   if (!qgroup_src || !qgroup_dst)
-   goto bad;
-
-   if (*inherit)
-   pos = (*inherit)->num_qgroups +
- (*inherit)->num_ref_copies * 2 * type;
-
-   ret = qgroup_inherit_realloc(inherit, 2, pos);
-   if (ret)
-   return ret;
-
-   (*inherit)->qgroups[pos++] = qgroup_src;
-   (*inherit)->qgroups[pos++] = qgroup_dst;
-
-   if (!type)
-   ++(*inherit)->num_ref_copies;
-   else
-   ++(*inherit)->num_excl_copies;
-
-   return 0;
-}
diff --git a/qgroup.h b/qgroup.h
index 875fbdf3..bb6610d7 100644
--- a/qgroup.h
+++ b/qgroup.h
@@ -92,7 +92,5 @@ int btrfs_qgroup_setup_comparer(struct 
btrfs_qgroup_comparer_set **comp_set,
int is_descending);
 int qgroup_inherit_size(struct btrfs_qgroup_inherit *p);
 int qgroup_inherit_add_group(struct btrfs_qgroup_inherit **inherit, char *arg);
-int qgroup_inherit_add_copy(struct btrfs_qgroup_inherit **inherit, char *arg,
-   int type);
 
 #endif
-- 
2.16.1

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 03/26] libbtrfsutil: add Python bindings

[Omar Sandoval]

From: Omar Sandoval 

The C libbtrfsutil library isn't very useful for scripting, so we also
want bindings for Python. Writing unit tests in Python is also much
easier than doing so in C. Only Python 3 is supported; if someone really
wants Python 2 support, they can write their own bindings. This commit
is just the scaffolding.

Signed-off-by: Omar Sandoval 
---
 INSTALL   |   4 +
 Makefile  |  36 ++
 Makefile.inc.in   |   2 +
 configure.ac  |  15 +++
 libbtrfsutil/README.md|   5 +-
 libbtrfsutil/python/.gitignore|   7 ++
 libbtrfsutil/python/btrfsutilpy.h |  57 ++
 libbtrfsutil/python/error.c   | 202 ++
 libbtrfsutil/python/module.c  | 166 
 libbtrfsutil/python/setup.py  | 100 +
 libbtrfsutil/python/tests/__init__.py |   0
 11 files changed, 593 insertions(+), 1 deletion(-)
 create mode 100644 libbtrfsutil/python/.gitignore
 create mode 100644 libbtrfsutil/python/btrfsutilpy.h
 create mode 100644 libbtrfsutil/python/error.c
 create mode 100644 libbtrfsutil/python/module.c
 create mode 100755 libbtrfsutil/python/setup.py
 create mode 100644 libbtrfsutil/python/tests/__init__.py

diff --git a/INSTALL b/INSTALL
index 819b92ea..24d6e24f 100644
--- a/INSTALL
+++ b/INSTALL
@@ -41,6 +41,10 @@ To build from the released tarballs:
 $ make
 $ make install
 
+To install the libbtrfsutil Python bindings:
+
+$ make install_python
+
 You may disable building some parts like documentation, btrfs-convert or
 backtrace support. See ./configure --help for more.
 
diff --git a/Makefile b/Makefile
index 062f7f3c..02b03e81 100644
--- a/Makefile
+++ b/Makefile
@@ -144,8 +144,10 @@ endif
 
 ifeq ($(BUILD_VERBOSE),1)
   Q =
+  SETUP_PY_Q =
 else
   Q = @
+  SETUP_PY_Q = -q
 endif
 
 ifeq ("$(origin D)", "command line")
@@ -293,6 +295,9 @@ endif
$($(subst -,_,btrfs-$(@:%/$(notdir $@)=%)-cflags))
 
 all: $(progs) $(libs) $(lib_links) $(BUILDDIRS)
+ifeq ($(PYTHON_BINDINGS),1)
+all: libbtrfsutil_python
+endif
 $(SUBDIRS): $(BUILDDIRS)
 $(BUILDDIRS):
@echo "Making all in $(patsubst build-%,%,$@)"
@@ -336,6 +341,16 @@ test-inst: all
 
 test: test-fsck test-mkfs test-convert test-misc test-fuzz test-cli
 
+ifeq ($(PYTHON_BINDINGS),1)
+test-libbtrfsutil: libbtrfsutil_python
+   $(Q)cd libbtrfsutil/python; \
+   LD_LIBRARY_PATH=../.. $(PYTHON) -m unittest discover -v tests
+
+.PHONY: test-libbtrfsutil
+
+test: test-libbtrfsutil
+endif
+
 #
 # NOTE: For static compiles, you need to have all the required libs
 #  static equivalent available
@@ -382,6 +397,15 @@ libbtrfsutil.so.0 libbtrfsutil.so: 
libbtrfsutil.so.$(libbtrfsutil_version)
@echo "[LN] $@"
$(Q)$(LN_S) -f $< $@
 
+ifeq ($(PYTHON_BINDINGS),1)
+libbtrfsutil_python: libbtrfsutil.so
+   @echo "[PY] libbtrfsutil"
+   $(Q)cd libbtrfsutil/python; \
+   CFLAGS= LDFLAGS= $(PYTHON) setup.py $(SETUP_PY_Q) build_ext -i 
build
+
+.PHONY: libbtrfsutil_python
+endif
+
 # keep intermediate files from the below implicit rules around
 .PRECIOUS: $(addsuffix .o,$(progs))
 
@@ -565,6 +589,10 @@ clean: $(CLEANDIRS)
  $(libs) $(lib_links) \
  $(progs_static) $(progs_extra) \
  libbtrfsutil/*.o libbtrfsutil/*.o.d
+ifeq ($(PYTHON_BINDINGS),1)
+   $(Q)cd libbtrfsutil/python; \
+   $(PYTHON) setup.py $(SETUP_PY_Q) clean -a
+endif
 
 clean-doc:
@echo "Cleaning Documentation"
@@ -599,6 +627,14 @@ ifneq ($(udevdir),)
$(INSTALL) -m644 $(udev_rules) $(DESTDIR)$(udevruledir)
 endif
 
+ifeq ($(PYTHON_BINDINGS),1)
+install_python: libbtrfsutil_python
+   $(Q)cd libbtrfsutil/python; \
+   $(PYTHON) setup.py install --skip-build $(if $(DESTDIR),--root 
$(DESTDIR)) --prefix $(prefix)
+
+.PHONY: install_python
+endif
+
 install-static: $(progs_static) $(INSTALLDIRS)
$(INSTALL) -m755 -d $(DESTDIR)$(bindir)
$(INSTALL) $(progs_static) $(DESTDIR)$(bindir)
diff --git a/Makefile.inc.in b/Makefile.inc.in
index 56271903..408e00d2 100644
--- a/Makefile.inc.in
+++ b/Makefile.inc.in
@@ -14,6 +14,8 @@ DISABLE_BTRFSCONVERT = @DISABLE_BTRFSCONVERT@
 BTRFSCONVERT_EXT2 = @BTRFSCONVERT_EXT2@
 BTRFSCONVERT_REISERFS = @BTRFSCONVERT_REISERFS@
 BTRFSRESTORE_ZSTD = @BTRFSRESTORE_ZSTD@
+PYTHON_BINDINGS = @PYTHON_BINDINGS@
+PYTHON = @PYTHON@
 
 SUBST_CFLAGS = @CFLAGS@
 SUBST_LDFLAGS = @LDFLAGS@
diff --git a/configure.ac b/configure.ac
index 290dc1d5..5e2905d5 100644
--- a/configure.ac
+++ b/configure.ac
@@ -199,6 +199,19 @@ fi
 AS_IF([test "x$enable_zstd" = xyes], [BTRFSRESTORE_ZSTD=1], 
[BTRFSRESTORE_ZSTD=0])
 AC_SUBST(BTRFSRESTORE_ZSTD)
 
+AC_ARG_ENABLE([python],
+   AS_HELP_STRING([--disable-python], [do not build libbtrfsutil Python 
bindings]),
+   [], 

[PATCH 00/26] btrfs-progs: introduce libbtrfsutil, "btrfs-progs as a library"

[Omar Sandoval]

From: Omar Sandoval 

Hello,

One of the features requests I get most often is a library to do the
sorts of operations that we do with btrfs-progs. We can shell out to
btrfs-progs, but the output format isn't always easily parsasble, and
shelling out isn't always ideal. There's libbtrfs, but it's very
incomplete, doesn't have a well thought out API, and is licensed under
the GPL, making it hard to use for many projects.

libbtrfsutil is a new library written from scratch to address these
issues. The implementation is completely independent of the existing
btrfs-progs code, including kerncompat.h, and has a clean API and naming
scheme. It is licensed under the LGPL. It also includes Python bindings
by default. I will maintain the library code.

Patch 1 is a preparation cleanup which can go in independently. Patch 2
adds the build system stuff for the library, and patch 3 does the same
for the Python bindings. Patches 4-14 implement the library helpers,
currently subvolume helpers and the sync ioctls. Patches 15-26 replace
the btrfs-progs and libbtrfs code to use libbtrfsutil instead. I took
care to preserve backwards-compatibility. `btrfs subvol list` in
particular had some buggy behaviors for -o and -a that I emulated in the
new code, see the comments in the code.

These patches are also available on my GitHub:
https://github.com/osandov/btrfs-progs/tree/libbtrfsutil. That branch
will rebase as I update this series.

Please share feedback regarding the API, implementation, or anything
else.

Thanks!

Omar Sandoval (26):
  btrfs-progs: get rid of undocumented qgroup inheritance options
  Add libbtrfsutil
  libbtrfsutil: add Python bindings
  libbtrfsutil: add btrfs_util_is_subvolume() and
btrfs_util_subvolume_id()
  libbtrfsutil: add qgroup inheritance helpers
  libbtrfsutil: add btrfs_util_create_subvolume()
  libbtrfsutil: add btrfs_util_subvolume_info()
  libbtrfsutil: add btrfs_util_[gs]et_read_only()
  libbtrfsutil: add btrfs_util_[gs]et_default_subvolume()
  libbtrfsutil: add subvolume iterator helpers
  libbtrfsutil: add btrfs_util_create_snapshot()
  libbtrfsutil: add btrfs_util_delete_subvolume()
  libbtrfsutil: add btrfs_util_deleted_subvolumes()
  libbtrfsutil: add filesystem sync helpers
  btrfs-progs: use libbtrfsutil for read-only property
  btrfs-progs: use libbtrfsutil for sync ioctls
  btrfs-progs: use libbtrfsutil for set-default
  btrfs-progs: use libbtrfsutil for get-default
  btrfs-progs: use libbtrfsutil for subvol create and snapshot
  btrfs-progs: use libbtrfsutil for subvol delete
  btrfs-progs: use libbtrfsutil for subvol show
  btrfs-progs: use libbtrfsutil for subvol sync
  btrfs-progs: replace test_issubvolume() with btrfs_util_is_subvolume()
  btrfs-progs: add recursive snapshot/delete using libbtrfsutil
  btrfs-progs: deprecate libbtrfs helpers with libbtrfsutil equivalents
  btrfs-progs: use libbtrfsutil for subvolume list

 .gitignore   |2 +
 Documentation/btrfs-subvolume.asciidoc   |   14 +-
 INSTALL  |4 +
 Makefile |   84 +-
 Makefile.inc.in  |2 +
 btrfs-list.c | 1201 +++---
 btrfs-list.h |   22 +-
 cmds-filesystem.c|   20 +-
 cmds-inspect.c   |9 +-
 cmds-qgroup.c|   20 +-
 cmds-receive.c   |   12 +-
 cmds-subvolume.c |  807 +--
 configure.ac |   15 +
 libbtrfsutil/COPYING |  674 +
 libbtrfsutil/COPYING.LESSER  |  165 +++
 libbtrfsutil/README.md   |   35 +
 libbtrfsutil/btrfsutil.h |  621 
 libbtrfsutil/errors.c|   55 +
 libbtrfsutil/filesystem.c|  103 ++
 libbtrfsutil/internal.h  |   36 +
 libbtrfsutil/python/.gitignore   |7 +
 libbtrfsutil/python/btrfsutilpy.h|   84 ++
 libbtrfsutil/python/error.c  |  202 
 libbtrfsutil/python/filesystem.c |   94 ++
 libbtrfsutil/python/module.c |  321 ++
 libbtrfsutil/python/qgroup.c |  141 +++
 libbtrfsutil/python/setup.py |  103 ++
 libbtrfsutil/python/subvolume.c  |  665 +
 libbtrfsutil/python/tests/__init__.py|   66 ++
 libbtrfsutil/python/tests/test_filesystem.py |   73 ++
 libbtrfsutil/python/tests/test_qgroup.py |   57 ++
 libbtrfsutil/python/tests/test_subvolume.py  |  383 +++
 libbtrfsutil/qgroup.c|   86 ++
 libbtrfsutil/subvolume.c | 1383 ++
 messages.h   |   14 +
 props.c   

Re: [PATCH] Btrfs: fix unexpected -EEXIST when creating new inode

[Nikolay Borisov]

On 25.01.2018 20:02, Liu Bo wrote:
> The highest objectid, which is assigned to new inode, is decided at
> the time of initializing fs roots.  However, in cases where log replay
> gets processed, the btree which fs root owns might be changed, so we
> have to search it again for the highest objectid, otherwise creating
> new inode would end up with -EEXIST.
> 
> cc:  v4.4-rc6+
> Fixes: f32e48e92596 ("Btrfs: Initialize btrfs_root->highest_objectid when 
> loading tree root and subvolume roots")
> Signed-off-by: Liu Bo 
> ---
>  fs/btrfs/tree-log.c | 19 +++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
> index a7e6235..646cdbf 100644
> --- a/fs/btrfs/tree-log.c
> +++ b/fs/btrfs/tree-log.c
> @@ -28,6 +28,7 @@
>  #include "hash.h"
>  #include "compression.h"
>  #include "qgroup.h"
> +#include "inode-map.h"
>  
>  /* magic values for the inode_only field in btrfs_log_inode:
>   *
> @@ -5715,6 +5716,24 @@ int btrfs_recover_log_trees(struct btrfs_root 
> *log_root_tree)
> path);
>   }
>  
> + if (!ret && wc.stage == LOG_WALK_REPLAY_ALL) {
> + struct btrfs_root *root = wc.replay_dest;
> +
> + btrfs_release_path(path);
> +
> + /*
> +  * We have just replayed everything, and the highest
> +  * objectid of fs roots probably has changed in case
> +  * some inode_item's got replayed.
> +  */
> + /*

nit: No need to start a new multiline comment

> +  * root->objectid_mutex is not acquired as log replay
> +  * could only happen during mount.
> +  */
> + ret = btrfs_find_highest_objectid(root,
> +   >highest_objectid);
> + }
> +
>   key.offset = found_key.offset - 1;
>   wc.replay_dest->log_root = NULL;
>   free_extent_buffer(log->node);
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[RFC PATCH] btrfs: Gracefully handle errors in btrfs_qgroup_trace_extent_post

[Nikolay Borisov]

Running generic/019 with qgroups on the scratch device enabled is
almost guaranteed to trigger the BUG_ON in btrfs_free_tree_block. It's
supposed to trigger only on -ENOMEM, in reality, however, it's possible
to get -EIO from btrfs_qgroup_trace_extent_post. This function just
finds the roots of the extent being tracked and sets the qrecord->old_roots
list. If this operation fails nothing critical happens except the
quota accounting can be considered wrong. In such case just set the
INCONSISTENT flag for the quota and be done with it.

Signed-off-by: Nikolay Borisov 
---

Qu, 

This fixes the crash for me, however I'm not entirely sure it's really the 
best fix since it's leaking the usage of INCONSISTENT out of qgroup.c. Ideally
I'd like to avoid this. Since you have more experience with qgroups and also you
introduced the extent tracking code in add_delayed_tree_ref how does 
look to you? 


 fs/btrfs/delayed-ref.c | 9 +++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c
index a1a40cf382e3..1e9aa18cc0d8 100644
--- a/fs/btrfs/delayed-ref.c
+++ b/fs/btrfs/delayed-ref.c
@@ -820,8 +820,13 @@ int btrfs_add_delayed_tree_ref(struct btrfs_fs_info 
*fs_info,
 num_bytes, parent, ref_root, level, action);
spin_unlock(_refs->lock);
 
-   if (qrecord_inserted)
-   return btrfs_qgroup_trace_extent_post(fs_info, record);
+   if (qrecord_inserted) {
+   int ret = btrfs_qgroup_trace_extent_post(fs_info, record);
+   if (ret != -ENOMEM)
+   fs_info->qgroup_flags |= 
BTRFS_QGROUP_STATUS_FLAG_INCONSISTENT;
+   else
+   return -ENOMEM;
+   }
return 0;
 
 free_head_ref:
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs check: backref lost, mismatch with its hash -- can't repair

[^m'e]

On Fri, Jan 26, 2018 at 12:16 PM, Qu Wenruo  wrote:
> Branch updated, problem in 1399 should be fixed.
>
> Seems the remaining problems are less and less now.
>
> Thanks,
> Qu
>

Great! The fix worked, and repair goes throught at last :-) though
we're still left with some inconsistencies at check:
(1st run)
--
checking fs roots
ERROR: root 1385 INODE[30039323] is orphan item
ERROR: root 1385 INODE[18446744073709551361] is orphan item
ERROR: root 1399 DIR INODE [30039322] size 99 not equal to 96
ERROR: root 1399 INODE[18446744073709551361] is orphan item
ERROR: errors found in fs roots
...
--

repair (1st run)
--
Fixed 0 roots.
checking extents
checking free space cache
checking fs roots
warning line 4682
checking csums
checking root refs
enabling repair mode
Checking filesystem on /dev/sdb3
UUID: de1723e2-150c-4448-bb36-be14d7d96093
No device size related problem found
cache and super generation don't match, space cache will be invalidated
reset isize for dir 30039322 root 1399
found 104509362176 bytes used, no error found
total csum bytes: 99690492
total tree bytes: 2394259456
total fs tree bytes: 2173468672
total extent tree bytes: 87900160
btree space waste bytes: 574264314
file data blocks allocated: 190189735936
 referenced 150679924736
--

check (2nd run)
--
checking fs roots
ERROR: root 1385 INODE[30039323] is orphan item
ERROR: root 1385 INODE[18446744073709551361] is orphan item
ERROR: root 1399 INODE[18446744073709551361] is orphan item
ERROR: errors found in fs roots
Checking filesystem on /dev/sdb3
UUID: de1723e2-150c-4448-bb36-be14d7d96093
cache and super generation don't match, space cache will be invalidated
found 104509362176 bytes used, error(s) found
total csum bytes: 99690492
total tree bytes: 14561427456
total fs tree bytes: 14340636672
total extent tree bytes: 87900160
btree space waste bytes: 3163391144
file data blocks allocated: 362059718656
 referenced 315586768896
--

repair (2nd run)
--
Fixed 0 roots.
checking extents
checking free space cache
checking fs roots
checking csums
checking root refs
enabling repair mode
Checking filesystem on /dev/sdb3
UUID: de1723e2-150c-4448-bb36-be14d7d96093
No device size related problem found
cache and super generation don't match, space cache will be invalidated
found 104509362176 bytes used, no error found
total csum bytes: 99690492
total tree bytes: 2394259456
total fs tree bytes: 2173468672
total extent tree bytes: 87900160
btree space waste bytes: 574264314
file data blocks allocated: 190189735936
 referenced 150679924736
--

It looks like it didn't detect anything bad...
As a side note, check in "original" mode doesn't detect the issues as
in runs reported above.

Debugging:
--
# ./btrfs.static inspect dump-tree -t 1385 /dev/sdb3 | grep -C 20
18446744073709551361
item 45 key (47329988 INODE_REF 256) itemoff 9768 itemsize 20
index 28 namelen 10 name: lost+found
item 46 key (47329988 DIR_ITEM 2438219243) itemoff 9726 itemsize 42
location key (30039324 INODE_ITEM 0) type FILE
transid 0 data_len 0 name_len 12
name: metadata.xml
item 47 key (47329988 DIR_INDEX 2) itemoff 9684 itemsize 42
location key (30039324 INODE_ITEM 0) type FILE
transid 0 data_len 0 name_len 12
name: metadata.xml
item 48 key (MULTIPLE INODE_ITEM 0) itemoff 9524 itemsize 160
generation 461639 transid 0 size 0 nbytes 0
block group 0 mode 100700 links 0 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1516888573.0 (2018-01-25 13:56:13)
ctime 1516888573.0 (2018-01-25 13:56:13)
mtime 1516888573.0 (2018-01-25 13:56:13)
otime 0.0 (1970-01-01 00:00:00)
item 49 key (ORPHAN ORPHAN_ITEM 30039323) itemoff 9524 itemsize 0
orphan item
item 50 key (ORPHAN ORPHAN_ITEM 18446744073709551361) itemoff 9524
itemsize 0
orphan item
total bytes 247335313408
bytes used 104509362176
uuid de1723e2-150c-4448-bb36-be14d7d96093
--

--
# ./btrfs.static inspect dump-tree -t 1399 /dev/sdb3 | grep -C 20
18446744073709551361
item 17 key (47343871 INODE_REF 6928733) itemoff 13096 itemsize 21
index 2746 namelen 11 name: cron.hourly
item 18 

Re: degraded permanent mount option

[Austin S. Hemmelgarn]

On 2018-01-26 09:47, Christophe Yayon wrote:

Hi Austin,

Thanks for your answer. It was my opinion too as the "degraded" seems to be flagged as 
"Mostly OK" on btrfs wiki status page. I am running Archlinux with recent kernel on all 
my servers (because of use of btrfs as my main filesystem, i need a recent kernel).

Your idea to add a separate entry in grub.cfg with rootflags=degraded is 
attractive, i will do this...

Just a last question, i thank that it was necessary to add "degraded" option in 
grub.cfg AND fstab to allow boot in degraded mode. I am not sure that only grub.cfg is 
sufficient...
Yesterday, i have done some test and boot a a system with only 1 of 2 drive in 
my root raid1 array. No problem with systemd, but i added rootflags and fstab 
option. I didn't test with only rootflags.
Hmm...  I'm pretty sure that you only need degraded in rootflags for a 
degraded boot without systemd involved.  Not sure about with systemd 
involved, though the fact that it worked with systemd at all is 
interesting, as last I knew systemd doesn't do any special casing for 
BTRFS and just looks at whether all the devices are registered with the 
kernel or not.


Also, as far as I know, `degraded` in the mount options won't cause any 
change in behavior if there is no device missing, so you're not really 
going to be running 'degraded' if you've got all your devices (though 
depending on how long it takes to scan devices, you may end up with some 
issues during boot when they're technically all present and working).

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: degraded permanent mount option

[Christophe Yayon]

Hi Austin,

Thanks for your answer. It was my opinion too as the "degraded" seems to be 
flagged as "Mostly OK" on btrfs wiki status page. I am running Archlinux with 
recent kernel on all my servers (because of use of btrfs as my main filesystem, 
i need a recent kernel).

Your idea to add a separate entry in grub.cfg with rootflags=degraded is 
attractive, i will do this...

Just a last question, i thank that it was necessary to add "degraded" option in 
grub.cfg AND fstab to allow boot in degraded mode. I am not sure that only 
grub.cfg is sufficient... 
Yesterday, i have done some test and boot a a system with only 1 of 2 drive in 
my root raid1 array. No problem with systemd, but i added rootflags and fstab 
option. I didn't test with only rootflags.

Thanks. 


-- 
  Christophe Yayon
  cyayon-l...@nbux.org

On Fri, Jan 26, 2018, at 15:18, Austin S. Hemmelgarn wrote:
> On 2018-01-26 09:02, Christophe Yayon wrote:
> > Hi all,
> > 
> > I don't know if it the right place to ask. Sorry it's not...
> No, it's just fine to ask here.  Questions like this are part of why the 
> mailing list exists.
> > 
> > Just a little question about "degraded" mount option. Is it a good idea to 
> > add this option (permanent) in fstab and grub rootflags for raid1/10 array 
> > ? Just to allow the system to boot again if a single hdd fail.
> Some people will disagree with me on this, but I would personally 
> suggest not doing this.  I'm of the opinion that running an array 
> degraded for any period of time beyond the bare minimum required to fix 
> it is a bad idea, given that:
> * It's not a widely tested configuration, so you are statistically more 
> likely to run into previously unknown bugs.  Even aside from that, there 
> are probably some edge cases that people have not yet found.
> * There are some issues with older kernel versions trying to access the 
> array after it's been mounted writable and degraded when it's only two 
> devices in raid1 mode.  This in turn is a good example of the above 
> point about not being widely tested, as it took quite a while for this 
> problem to come up on the mailing list.
> * Running degraded is liable to be slower, because the filesystem has to 
> account for the fact that the missing device might reappear at any 
> moment.  This is actually true of any replication system, not just BTRFS.
> * For a 2 device raid1 volume, there is no functional advantage to 
> running degraded with one device compared to converting to just use a 
> single device (this is only true of BTRFS because of the fact that it's 
> trivial to convert things, while for MD and LVM it is extremely 
> complicated to do so online).
> 
> Additionally, adding the `degraded` mount option won't actually let you 
> mount the root filesystem if you're using systemd as an init system, 
> because systemd refuses to mount BTRFS volumes which have devices missing.
> 
> Assuming that the systemd thing isn't an issue for you, I would suggest 
> instead creating a separate GRUB entry with the option set in rootflags. 
>   This will allow you to manually boot the system if the array is 
> degraded, but will make sure you notice during boot (in my case, I don't 
> even do that, but I'm also reasonably used to tweaking kernel parameters 
> from GRUB prior to booting the system that it would end up just wasting 
> space).
> > 
> > Of course, i have some cron jobs to check my array health.
> It's good to hear that you're taking the initiative to monitor things, 
> however this fact doesn't really change my assessment above.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] btrfs: tree-checker: Replace root parameter with fs_info

[Josef Bacik]

On Thu, Jan 25, 2018 at 02:56:18PM +0800, Qu Wenruo wrote:
> When inspecting the error message with real corruption, the "root=%llu"
> always shows "1" (root tree), instead of correct owner.
> 
> The problem is that we are getting @root from page->mapping->host, which
> points the same btree inode, so we will always get the same root.
> 
> This makes the root owner output meaningless, and harder to port
> tree-checker to btrfs-progs.
> 
> So get rid of the false and meaningless @root parameter and replace it
> with @fs_info.
> To get the owner, we can only rely on btrfs_header_owner() now.
> 
> Signed-off-by: Qu Wenruo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] btrfs: handle failure of add_pending_csums

[Josef Bacik]

On Mon, Jan 08, 2018 at 10:59:43AM +0200, Nikolay Borisov wrote:
> add_pending_csums was added as part of the new data=ordered implementation in
> e6dcd2dc9c48 ("Btrfs: New data=ordered implementation"). Even back then it
> called the btrfs_csum_file_blocks which can fail but it never bothered 
> handling
> the failure. In ENOMEM situation this could lead to the filesystem failing to
> write the checksums for a particular extent and not detect this. On read this
> could lead to the filesystem erroring out due to crc mismatch. Fix it by
> propagating failure from add_pending_csums and handling them
> 
> Signed-off-by: Nikolay Borisov 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix unexpected -EEXIST when creating new inode

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:56AM -0700, Liu Bo wrote:
> The highest objectid, which is assigned to new inode, is decided at
> the time of initializing fs roots.  However, in cases where log replay
> gets processed, the btree which fs root owns might be changed, so we
> have to search it again for the highest objectid, otherwise creating
> new inode would end up with -EEXIST.
> 
> cc:  v4.4-rc6+
> Fixes: f32e48e92596 ("Btrfs: Initialize btrfs_root->highest_objectid when 
> loading tree root and subvolume roots")
> Signed-off-by: Liu Bo 
> ---

Eesh that's bad.

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: do not check inode's runtime flags under root->orphan_lock

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:55AM -0700, Liu Bo wrote:
> It's not necessary to hold ->orphan_lock when checking inode's runtime
> flags.
> 
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix use-after-free on root->orphan_block_rsv

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:54AM -0700, Liu Bo wrote:
> I got these from running generic/475,
> 
> WARNING: CPU: 0 PID: 26384 at fs/btrfs/inode.c:3326 
> btrfs_orphan_commit_root+0x1ac/0x2b0 [btrfs]
> BUG: unable to handle kernel NULL pointer dereference at 0010
> IP: btrfs_block_rsv_release+0x1c/0x70 [btrfs]
> Call Trace:
>   btrfs_orphan_release_metadata+0x9f/0x200 [btrfs]
>   btrfs_orphan_del+0x10d/0x170 [btrfs]
>   btrfs_setattr+0x500/0x640 [btrfs]
>   notify_change+0x7ae/0x870
>   do_truncate+0xca/0x130
>   vfs_truncate+0x2ee/0x3d0
>   do_sys_truncate+0xaf/0xf0
>   SyS_truncate+0xe/0x10
>   entry_SYSCALL_64_fastpath+0x1f/0x96
> 
> The race is between btrfs_orphan_commit_root and btrfs_orphan_del,
> t1t2
> btrfs_orphan_commit_root btrfs_orphan_del
>spin_lock
>check (>orphan_inodes)
>root->orphan_block_rsv = NULL;
>spin_unlock
>  atomic_dec(>orphan_inodes);
>  access root->orphan_block_rsv
> 
> Accessing root->orphan_block_rsv must be done before decreasing
> root->orphan_inodes.
> 
> cc:  v3.12+
> Fixes: 703c88e03524 ("Btrfs: fix tracking of orphan inode count")
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: degraded permanent mount option

[Austin S. Hemmelgarn]

On 2018-01-26 09:02, Christophe Yayon wrote:

Hi all,

I don't know if it the right place to ask. Sorry it's not...
No, it's just fine to ask here.  Questions like this are part of why the 
mailing list exists.


Just a little question about "degraded" mount option. Is it a good idea to add 
this option (permanent) in fstab and grub rootflags for raid1/10 array ? Just to allow 
the system to boot again if a single hdd fail.
Some people will disagree with me on this, but I would personally 
suggest not doing this.  I'm of the opinion that running an array 
degraded for any period of time beyond the bare minimum required to fix 
it is a bad idea, given that:
* It's not a widely tested configuration, so you are statistically more 
likely to run into previously unknown bugs.  Even aside from that, there 
are probably some edge cases that people have not yet found.
* There are some issues with older kernel versions trying to access the 
array after it's been mounted writable and degraded when it's only two 
devices in raid1 mode.  This in turn is a good example of the above 
point about not being widely tested, as it took quite a while for this 
problem to come up on the mailing list.
* Running degraded is liable to be slower, because the filesystem has to 
account for the fact that the missing device might reappear at any 
moment.  This is actually true of any replication system, not just BTRFS.
* For a 2 device raid1 volume, there is no functional advantage to 
running degraded with one device compared to converting to just use a 
single device (this is only true of BTRFS because of the fact that it's 
trivial to convert things, while for MD and LVM it is extremely 
complicated to do so online).


Additionally, adding the `degraded` mount option won't actually let you 
mount the root filesystem if you're using systemd as an init system, 
because systemd refuses to mount BTRFS volumes which have devices missing.


Assuming that the systemd thing isn't an issue for you, I would suggest 
instead creating a separate GRUB entry with the option set in rootflags. 
 This will allow you to manually boot the system if the array is 
degraded, but will make sure you notice during boot (in my case, I don't 
even do that, but I'm also reasonably used to tweaking kernel parameters 
from GRUB prior to booting the system that it would end up just wasting 
space).


Of course, i have some cron jobs to check my array health.
It's good to hear that you're taking the initiative to monitor things, 
however this fact doesn't really change my assessment above.

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix btrfs_evict_inode to handle abnormal inodes correctly

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:53AM -0700, Liu Bo wrote:
> This regression is introduced in
> commit 3d48d9810de4 ("btrfs: Handle uninitialised inode eviction").
> 
> There are two problems,
> 
> a) it is ->destroy_inode() that does the final free on inode, not
>->evict_inode(),
> b) clear_inode() must be called before ->evict_inode() returns.
> 
> This could end up hitting BUG_ON(inode->i_state != (I_FREEING | I_CLEAR));
> in evict() because I_CLEAR is set in clear_inode().
> 
> Fixes: commit 3d48d9810de4 ("btrfs: Handle uninitialised inode eviction")
> Cc:  # v4.7-rc6+
> Signed-off-by: Liu Bo 
> ---

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix extent state leak from tree log

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:52AM -0700, Liu Bo wrote:
> It's possible that btrfs_sync_log() bails out after one of the two
> btrfs_write_marked_extents() which convert extent state's state bit into
> EXTENT_NEED_WAIT from EXTENT_DIRTY/EXTENT_NEW, however only EXTENT_DIRTY
> and EXTENT_NEW are searched by free_log_tree() so that those extent states
> with EXTENT_NEED_WAIT lead to memory leak.
> 
> cc: 
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix crash due to not cleaning up tree log block's dirty bits

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:51AM -0700, Liu Bo wrote:
> In cases that the whole fs flips into readonly status due to failures in
> critical sections, then log tree's blocks are still dirty, and this leads
> to a crash during umount time, the crash is about use-after-free,
> 
> umount
>  -> close_ctree
> -> stop workers
> -> iput(btree_inode)
>-> iput_final
>   -> write_inode_now
>-> ...
>  -> queue job on stop'd workers
> 
> cc:  v3.12+
> Fixes: 681ae50917df ("Btrfs: cleanup reserved space when freeing tree log on 
> error")
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix unexpected cow in run_delalloc_nocow

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:49AM -0700, Liu Bo wrote:
> Fstests generic/475 provides a way to fail metadata reads while
> checking if checksum exists for the inode inside run_delalloc_nocow(),
> and csum_exist_in_range() interprets error (-EIO) as inode having
> checksum and makes its caller enters the cow path.
> 
> In case of free space inode, this ends up with a warning in
> cow_file_range().
> 
> The same problem applies for btrfs_cross_ref_exist() since it may also
> read metadata in between.
> 
> With this, run_delalloc_nocow() bails out when errors occur at the two
> places.
> 
> cc:  v2.6.28+
> Fixes: 17d217fe970d ("Btrfs: fix nodatasum handling in balancing code")
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

degraded permanent mount option

[Christophe Yayon]

Hi all,

I don't know if it the right place to ask. Sorry it's not...

Just a little question about "degraded" mount option. Is it a good idea to add 
this option (permanent) in fstab and grub rootflags for raid1/10 array ? Just 
to allow the system to boot again if a single hdd fail.

Of course, i have some cron jobs to check my array health.

Thanks.

-- 
  Christophe Yayon
  cyayon-l...@nbux.org
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: fix deadlock in run_delalloc_nocow

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:50AM -0700, Liu Bo wrote:
> @cur_offset is not set back to what it should be (@cow_start) if
> btrfs_next_leaf() returns something wrong, and the range [cow_start,
> cur_offset) remains locked forever.
> 
> cc: 
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Btrfs: enhance leak debug checker for extent state and extent buffer

[Josef Bacik]

On Thu, Jan 25, 2018 at 11:02:48AM -0700, Liu Bo wrote:
> This prints out eb->bflags since it contains some useful information,
> e.g. whether eb is dirty.
> 
> Signed-off-by: Liu Bo 

Reviewed-by: Josef Bacik 

Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs check: backref lost, mismatch with its hash -- can't repair

[Qu Wenruo]

Branch updated, problem in 1399 should be fixed.

Seems the remaining problems are less and less now.

Thanks,
Qu

On 2018年01月26日 19:19, ^m'e wrote:
> On Thu, Jan 25, 2018 at 11:30 PM, Qu Wenruo  wrote:
>>
>> Please provide dump for this:
>>
>> # btrfs inspect dump-tree -t 1399  | grep -C 20 18446744073709551615
>>
> 
> This gives nothing.
> 
>> And
>>
>> # btrfs inspect dump-tree -t 1399  | grep -C 20 30039322
>>
> 
> This gives:
> ---
> location key (30037910 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 3
> name: bam
> item 52 key (30037720 DIR_ITEM 508462201) itemoff 14104 itemsize 40
> location key (30039832 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 10
> name: suse-build
> item 53 key (30037720 DIR_ITEM 541125215) itemoff 14070 itemsize 34
> location key (30038354 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 4
> name: cram
> item 54 key (30037720 DIR_ITEM 543235706) itemoff 14035 itemsize 35
> location key (30039133 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 5
> name: lsuio
> item 55 key (30037720 DIR_ITEM 586823170) itemoff 14000 itemsize 35
> location key (30038846 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 5
> name: geany
> item 56 key (30037720 DIR_ITEM 603413733) itemoff 13938 itemsize 62
> location key (30039322 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 32
> name: obs-service-download_src_package
> item 57 key (30037720 DIR_ITEM 623694194) itemoff 13903 itemsize 35
> location key (30038092 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 5
> name: byacc
> item 58 key (30037720 DIR_ITEM 637448305) itemoff 13868 itemsize 35
> location key (43374420 INODE_ITEM 0) type DIR
> transid 200308 data_len 0 name_len 5
> name: vpuml
> item 59 key (30037720 DIR_ITEM 660989717) itemoff 13828 itemsize 40
> location key (30038283 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 10
> name: comparator
> item 60 key (30037720 DIR_ITEM 666000672) itemoff 13782 itemsize 46
> location key (30039257 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 16
> name: molecule-plugins
> item 61 key (30037720 DIR_ITEM 679217690) itemoff 13749 itemsize 33
> location key (36281336 INODE_ITEM 0) type DIR
> --
> location key (30039292 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 15
> name: nvidia-cuda-sdk
> item 73 key (30037720 DIR_INDEX 238) itemoff 13448 itemsize 49
> location key (30039299 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 19
> name: nvidia-cuda-toolkit
> item 74 key (30037720 DIR_INDEX 239) itemoff 13411 itemsize 37
> location key (30039309 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 7
> name: objconv
> item 75 key (30037720 DIR_INDEX 240) itemoff 13361 itemsize 50
> location key (30039314 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 20
> name: obs-service-cpanspec
> item 76 key (30037720 DIR_INDEX 241) itemoff 13305 itemsize 56
> location key (30039318 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 26
> name: obs-service-download_files
> item 77 key (30037720 DIR_INDEX 242) itemoff 13243 itemsize 62
> location key (30039322 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 32
> name: obs-service-download_src_package
> item 78 key (30037720 DIR_INDEX 243) itemoff 13189 itemsize 54
> location key (30039326 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 24
> name: obs-service-download_url
> item 79 key (30037720 DIR_INDEX 244) itemoff 13135 itemsize 54
> location key (30039330 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 24
> name: obs-service-extract_file
> item 80 key (30037720 DIR_INDEX 245) itemoff 13077 itemsize 58
> location key (30039334 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 28
> name: obs-service-format_spec_file
> item 81 key (30037720 DIR_INDEX 246) itemoff 13007 itemsize 70
> location key (30039338 INODE_ITEM 0) type DIR
> transid 136248 data_len 0 name_len 40
> name: obs-service-generator_driver_update_disk
> item 82 key (30037720 DIR_INDEX 247) itemoff 12953 itemsize 54
> location key (30039342 INODE_ITEM 0) type DIR
> --
> mtime 1504685599.188061317 (2017-09-06 08:13:19)
> otime 1504685599.188061317 (2017-09-06 08:13:19)
> item 73 key (30039320 INODE_REF 30039318) itemoff 5278 itemsize 22
>   

Re: [PATCH] Btrfs: fix btrfs_evict_inode to handle abnormal inodes correctly

[Nikolay Borisov]

On 25.01.2018 20:02, Liu Bo wrote:
> This regression is introduced in
> commit 3d48d9810de4 ("btrfs: Handle uninitialised inode eviction").
> 
> There are two problems,
> 
> a) it is ->destroy_inode() that does the final free on inode, not
>->evict_inode(),
> b) clear_inode() must be called before ->evict_inode() returns.
> 
> This could end up hitting BUG_ON(inode->i_state != (I_FREEING | I_CLEAR));
> in evict() because I_CLEAR is set in clear_inode().
> 

Oops, It seems I've missed that when I wrote the original patch.

Reviewed-by: Nikolay Borisov 

> Fixes: commit 3d48d9810de4 ("btrfs: Handle uninitialised inode eviction")
> Cc:  # v4.7-rc6+
> Signed-off-by: Liu Bo 
> ---
>  fs/btrfs/inode.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index 281a250..bc6ef73 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -5286,7 +5286,7 @@ void btrfs_evict_inode(struct inode *inode)
>   trace_btrfs_inode_evict(inode);
>  
>   if (!root) {
> - kmem_cache_free(btrfs_inode_cachep, BTRFS_I(inode));
> + clear_inode(inode);
>   return;
>   }
>  
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs check: backref lost, mismatch with its hash -- can't repair

[^m'e]

On Thu, Jan 25, 2018 at 11:30 PM, Qu Wenruo  wrote:
>
> Please provide dump for this:
>
> # btrfs inspect dump-tree -t 1399  | grep -C 20 18446744073709551615
>

This gives nothing.

> And
>
> # btrfs inspect dump-tree -t 1399  | grep -C 20 30039322
>

This gives:
---
location key (30037910 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 3
name: bam
item 52 key (30037720 DIR_ITEM 508462201) itemoff 14104 itemsize 40
location key (30039832 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 10
name: suse-build
item 53 key (30037720 DIR_ITEM 541125215) itemoff 14070 itemsize 34
location key (30038354 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 4
name: cram
item 54 key (30037720 DIR_ITEM 543235706) itemoff 14035 itemsize 35
location key (30039133 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 5
name: lsuio
item 55 key (30037720 DIR_ITEM 586823170) itemoff 14000 itemsize 35
location key (30038846 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 5
name: geany
item 56 key (30037720 DIR_ITEM 603413733) itemoff 13938 itemsize 62
location key (30039322 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 32
name: obs-service-download_src_package
item 57 key (30037720 DIR_ITEM 623694194) itemoff 13903 itemsize 35
location key (30038092 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 5
name: byacc
item 58 key (30037720 DIR_ITEM 637448305) itemoff 13868 itemsize 35
location key (43374420 INODE_ITEM 0) type DIR
transid 200308 data_len 0 name_len 5
name: vpuml
item 59 key (30037720 DIR_ITEM 660989717) itemoff 13828 itemsize 40
location key (30038283 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 10
name: comparator
item 60 key (30037720 DIR_ITEM 666000672) itemoff 13782 itemsize 46
location key (30039257 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 16
name: molecule-plugins
item 61 key (30037720 DIR_ITEM 679217690) itemoff 13749 itemsize 33
location key (36281336 INODE_ITEM 0) type DIR
--
location key (30039292 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 15
name: nvidia-cuda-sdk
item 73 key (30037720 DIR_INDEX 238) itemoff 13448 itemsize 49
location key (30039299 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 19
name: nvidia-cuda-toolkit
item 74 key (30037720 DIR_INDEX 239) itemoff 13411 itemsize 37
location key (30039309 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 7
name: objconv
item 75 key (30037720 DIR_INDEX 240) itemoff 13361 itemsize 50
location key (30039314 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 20
name: obs-service-cpanspec
item 76 key (30037720 DIR_INDEX 241) itemoff 13305 itemsize 56
location key (30039318 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 26
name: obs-service-download_files
item 77 key (30037720 DIR_INDEX 242) itemoff 13243 itemsize 62
location key (30039322 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 32
name: obs-service-download_src_package
item 78 key (30037720 DIR_INDEX 243) itemoff 13189 itemsize 54
location key (30039326 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 24
name: obs-service-download_url
item 79 key (30037720 DIR_INDEX 244) itemoff 13135 itemsize 54
location key (30039330 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 24
name: obs-service-extract_file
item 80 key (30037720 DIR_INDEX 245) itemoff 13077 itemsize 58
location key (30039334 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 28
name: obs-service-format_spec_file
item 81 key (30037720 DIR_INDEX 246) itemoff 13007 itemsize 70
location key (30039338 INODE_ITEM 0) type DIR
transid 136248 data_len 0 name_len 40
name: obs-service-generator_driver_update_disk
item 82 key (30037720 DIR_INDEX 247) itemoff 12953 itemsize 54
location key (30039342 INODE_ITEM 0) type DIR
--
mtime 1504685599.188061317 (2017-09-06 08:13:19)
otime 1504685599.188061317 (2017-09-06 08:13:19)
item 73 key (30039320 INODE_REF 30039318) itemoff 5278 itemsize 22
index 3 namelen 12 name: metadata.xml
item 74 key (30039320 EXTENT_DATA 0) itemoff 4809 itemsize 469
generation 136248 type 0 (inline)
inline extent data size 448 ram_bytes 448 compression 0 (none)
item 75 key (30039321 INODE_ITEM 0) itemoff 4649 itemsize 160
generation 136248 transid 202216 size 213 nbytes 213
 

Re: [PATCH] Btrfs: delete dead code in btrfs_orphan_add()

[Nikolay Borisov]

On 26.01.2018 01:56, Omar Sandoval wrote:
> From: Omar Sandoval 
> 
> btrfs_orphan_add() has had this case commented out since it was first
> introduced in commit d68fc57b7e32 ("Btrfs: Metadata reservation for
> orphan inodes"). Most of the orphan cleanup code has been rewritten
> since then, so it's safe to say that this code isn't needed.

I've tried removing this a couple of times but David objected due to
missing historical context and whether we might need it. I'm totally
fine with killing it.

Reviewed-by: Nikolay Borisov 

> 
> Signed-off-by: Omar Sandoval 
> ---
>  fs/btrfs/inode.c | 22 +-
>  1 file changed, 1 insertion(+), 21 deletions(-)
> 
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index e1a7f3cb5be9..d4e3a09c0ea4 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -3334,17 +3334,6 @@ int btrfs_orphan_add(struct btrfs_trans_handle *trans,
>  
>   if (!test_and_set_bit(BTRFS_INODE_HAS_ORPHAN_ITEM,
> >runtime_flags)) {
> -#if 0
> - /*
> -  * For proper ENOSPC handling, we should do orphan
> -  * cleanup when mounting. But this introduces backward
> -  * compatibility issue.
> -  */
> - if (!xchg(>orphan_item_inserted, 1))
> - insert = 2;
> - else
> - insert = 1;
> -#endif
>   insert = 1;
>   atomic_inc(>orphan_inodes);
>   }
> @@ -3370,7 +3359,7 @@ int btrfs_orphan_add(struct btrfs_trans_handle *trans,
>   }
>  
>   /* insert an orphan item to track this unlinked/truncated file */
> - if (insert >= 1) {
> + if (insert) {
>   ret = btrfs_insert_orphan_item(trans, root, btrfs_ino(inode));
>   if (ret) {
>   atomic_dec(>orphan_inodes);
> @@ -3389,15 +3378,6 @@ int btrfs_orphan_add(struct btrfs_trans_handle *trans,
>   ret = 0;
>   }
>  
> - /* insert an orphan item to track subvolume contains orphan files */
> - if (insert >= 2) {
> - ret = btrfs_insert_orphan_item(trans, fs_info->tree_root,
> -root->root_key.objectid);
> - if (ret && ret != -EEXIST) {
> - btrfs_abort_transaction(trans, ret);
> - return ret;
> - }
> - }
>   return 0;
>  }
>  
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 15/15] btrfs-progs: fsck-tests: add image for original and lowmem check

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> This image have two cases mixed:
> 1) Both filetypes of dir_item and dir_index about inode 258
>are corrupted.
> 2) inode item 258 is missing.

It would be better to provide some debug tree output in commit message.

Something like:
--
item 6 key (257 DIR_ITEM 4128386376) itemoff 15834 itemsize 35
location key (258 INODE_ITEM 0) type DIR_ITEM.100
transid 7 data_len 0 name_len 5
name: file1
item 7 key (257 DIR_INDEX 2) itemoff 15799 itemsize 35
location key (258 INODE_ITEM 0) type DIR_ITEM.34
transid 7 data_len 0 name_len 5
name: file1
item 8 key (258 INODE_REF 257) itemoff 15784 itemsize 15
index 2 namelen 5 name: file1
--

would explain the problem easier.

And I think the coverage is not that good.

The image will fall into the fallback case as no reliable filetype.

But it would be better to have another image with regular/prealloc file
extent to allow us to exam the filetype guess/find code.

Thanks,
Qu
> 
> Signed-off-by: Su Yue 
> ---
>  .../029-mismatched-filetype-no-inode/default_case.img| Bin 0 -> 3072 
> bytes
>  1 file changed, 0 insertions(+), 0 deletions(-)
>  create mode 100644 
> tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img
> 
> diff --git 
> a/tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img 
> b/tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img
> new file mode 100644
> index 
> ..57fc55b13c18bb9e5098a2fc4cb78f44fcf4abd9
> GIT binary patch
> literal 3072
> zcmeHIdr;F?7EXCTRG?Bxc@>Nzl**%3o;ExN5K61)k|sa|c?E0?0tS<45}pcFF|3HG
> zP(T(C5ekF^)P$A90ufn+LPC@zJVi+;1Oh=If!~6@ywljzjJ23d(ZjK
> zojF%g#i^6~k^cn!$27LiPu{I9Fuxu2_TKVAAg%4)oBa+Gw*w%~J8ZGd_-G#$_^81D
> zqXPJmONab%?&_8cU%hU@e3P9%Ckpzm)Ff$v)Wc@$>`BOZ#`Pp<)M#>*a!#6gR-n6^
> z)1=c4bzkSqAeVbqNE#^6(OCUuf%ng!<@)45b#aP846yZF4X-R_(Y-V^`b{r7%mwmG
> zSWwtole>bWF5})+N6j;AXb0Gxy74YnHE}t5`;DOxH;HrQz7u)mWsTfIuMAkZ<(_L<
> zC!F*GIJRKTD$r+#ynp(^1;1avE`FkKNIg+L5PnOuanrCw zs4O>R`tt$)P*2%T!%ObT%-uhb?}*?n%}JRS;pn;z#C$w4a`Mn{yX|Ga9U}x~oOl
> zPdy^u@GIhKy*>SRG86A0MnN+-sb03Tf_vZ5-E+liKXMd_+MktgGVy
> zvel#Bjd}T{tkT9P=zgwDEa(bJDo>c|p2U#nuEy7Oq}9Yu^C{YMx%Hc`+EvW8NX3^E
> zk5b)CX?>IaMfBjmD-m+Yq}J{ZNNVX0?AQ<#|9zRx-J>V|%LlTp#^N=X)8>mh-)=f9
> zEYlDVvLL+$VG(Ui8HDAoY(D;gCVKkJ-@HePR*lull}K3i@%ZJsPNw}D=_Eh9%JD@h
> z?Erk8B2lV5KN*FmZmG~q(|r;G^8W)cQe$Qm!0sCQG}HAsPz|4CwFrA_^+e2sR_0gZ
> zT6?nGG7Cq$kNKf{KyPloT`ZqIZM2Cu7Dyak38$c#V`;-zUb>Sns-7=GO_RFo*VdgR
> 

Re: [PATCH 13/15] btrfs-progs: check: find inode filetype in create_inode_item()

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> Now, find_file_type() doesn't return 0 if mismatched filetype is in a
> backref.
> 
> Let create_inode_item() first call find_file_type() to get filetype.
> If it failed, then guess filetype.
> 
> Signed-off-by: Su Yue 
> ---
>  cmds-check.c | 29 ++---
>  1 file changed, 22 insertions(+), 7 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index a83f0a92f48b..6091c7ef3442 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -3140,6 +3140,8 @@ static int create_inode_item_lowmem(struct 
> btrfs_trans_handle *trans,
>   return __create_inode_item(trans, root, ino, 0, 0, 0, mode);
>  }
>  
> +static u32 btrfs_type_to_imode(u8 type);
> +static int find_file_type(struct inode_record *rec, u8 *type);

Nice to see code shared between original and lowmem mode.

>  static int create_inode_item(struct btrfs_root *root,
>struct inode_record *rec, int root_dir)
>  {
> @@ -3148,14 +3150,19 @@ static int create_inode_item(struct btrfs_root *root,
>   u32 mode = 0;
>   u64 size = 0;
>   int ret;
> + u8 type = 0;
>  
> - trans = btrfs_start_transaction(root, 1);
> - if (IS_ERR(trans)) {
> - ret = PTR_ERR(trans);
> - return ret;
> + nlink = root_dir ? 1 : rec->found_link;
> + ret = find_file_type(rec, );
> + if (!ret) {
> + mode = btrfs_type_to_imode(type) | 0755;
> + if (type == BTRFS_FT_REG_FILE)
> + size = rec->found_size;
> + else
> + size = rec->extent_end;
> + goto create_inode;
>   }

Just a nicpick, here we could use simple if else branch to avoid extra tag:

if (!ret) {
...
} else if (rec->found_dir_item) {
...
} else if (!rec->found_dir_item) {
...
}

Despite that, feel free to add my tag:

Reviewed-by: Qu Wenruo 

Thanks,
Qu

>  
> - nlink = root_dir ? 1 : rec->found_link;
>   if (rec->found_dir_item) {
>   if (rec->found_file_extent)
>   fprintf(stderr, "root %llu inode %llu has both a dir "
> @@ -3167,7 +3174,14 @@ static int create_inode_item(struct btrfs_root *root,
>   size = rec->found_size;
>   } else if (!rec->found_dir_item) {
>   size = rec->extent_end;
> - mode =  S_IFREG | 0755;
> + mode = S_IFREG | 0755;
> + }
> +
> +create_inode:
> + trans = btrfs_start_transaction(root, 1);
> + if (IS_ERR(trans)) {
> + ret = PTR_ERR(trans);
> + return ret;
>   }
>  
>   ret = __create_inode_item(trans, root, rec->ino, size, rec->nbytes,
> @@ -3307,7 +3321,8 @@ static int find_file_type(struct inode_record *rec, u8 
> *type)
>   }
>  
>   list_for_each_entry(backref, >backrefs, list) {
> - if (backref->found_dir_index || backref->found_dir_item) {
> + if ((backref->found_dir_index || backref->found_dir_item) &&
> + (!(backref->errors & REF_ERR_FILETYPE_UNMATCH))) {
>   *type = backref->filetype;
>   return 0;
>   }
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 09/15] btrfs-progs: lowmem check: change logic of leaf process if repair

[Su Yue]

On 01/26/2018 06:01 PM, Qu Wenruo wrote:



On 2018年01月26日 16:35, Su Yue wrote:

In lowmem check without repair, process_one_leaf_v2() will process one
entire leaf and inner check_inode_item() leads path point next
leaf.
In the beginning, process_one_leaf_v2() will let path point fist inode
item or first position where inode id changed.

However, in lowmem repair, process_one_leaf_v2() will be interrupted
to process one leaf because repair will CoW the leaf. Then some items
unprocessed is skipped.
Since repair may also delete some items, we can't use tricks like
record last checked key.

So, only for lowmem repair:
1. check_inode_item is responsible for handle case missing inode item.


I think the idea to use inode item as the indicator is a good idea.
And since only check_inode_item() can delete inode item, let it to
handle the path is reasonable.


2. process_one_leaf_v2() do not modify path manually, and check_inode()
promise that @path points last checked item.
Only when something are fixed, process_one_leaf_v2() will continue
to check in next round.

Signed-off-by: Su Yue 
---
  cmds-check.c | 68 
  1 file changed, 59 insertions(+), 9 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index e57eea4e61c9..ae0a9e146399 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -2007,6 +2007,8 @@ static int process_one_leaf_v2(struct btrfs_root *root, 
struct btrfs_path *path,
  
  	cur_bytenr = cur->start;
  
+	if (repair)

+   goto again;
/* skip to first inode item or the first inode number change */
nritems = btrfs_header_nritems(cur);
for (i = 0; i < nritems; i++) {
@@ -2033,9 +2035,12 @@ again:
goto out;
  
  	/* still have inode items in thie leaf */

-   if (cur->start == cur_bytenr)
+   if (cur->start == cur_bytenr) {
+   ret = btrfs_next_item(root, path);
+   if (ret > 0)
+   goto out;
goto again;
-
+   }
/*
 * we have switched to another leaf, above nodes may
 * have changed, here walk down the path, if a node
@@ -5721,6 +5726,8 @@ static int repair_dir_item(struct btrfs_root *root, 
struct btrfs_key *key,
  ino);
}
}
+   } else {
+   true_filetype = filetype;


This modification doesn't seems related to this patch.

Maybe it's better to move it 4th patch?


Yep.

}
  
  	/*

@@ -6489,6 +6496,45 @@ out:
return ret;
  }
  
+/*

+ * Try insert new inode item frist.
+ * If failed, jump to next inode item.
+ */
+static int handle_inode_item_missing(struct btrfs_root *root,
+struct btrfs_path *path)
+{
+   struct btrfs_key key;
+   int ret;
+
+   btrfs_item_key_to_cpu(path->nodes[0], , path->slots[0]);
+
+   ret = repair_inode_item_missing(root, key.objectid, 0);
+   if (!ret) {
+   btrfs_release_path(path);
+   ret = btrfs_search_slot(NULL, root, , path, 0, 0);
+   if (ret)
+   goto next_inode;
+   else
+   goto out;
+   }
+
+next_inode:
+   error("inode item[%llu] is missing, skip to check next inode",
+ key.objectid);
+   while (1) {
+   ret = btrfs_next_item(root, path);
+   if (ret > 0)
+   goto out;


ret < 0 case is not handled.


+   btrfs_item_key_to_cpu(path->nodes[0], , path->slots[0]);
+   if (key.type == BTRFS_INODE_ITEM_KEY) {
+   ret = 0;
+   break;
+   }
+   }
+out:
+   return ret;
+}
+
  /*
   * Check INODE_ITEM and related ITEMs (the same inode number)
   * 1. check link count
@@ -6536,6 +6582,13 @@ static int check_inode_item(struct btrfs_root *root, 
struct btrfs_path *path,
err |= LAST_ITEM;
return err;
}
+   if (key.type != BTRFS_INODE_ITEM_KEY && repair) {
+   ret = handle_inode_item_missing(root, path);
+   if (ret > 0)
+   err |= LAST_ITEM;
+   if (ret)
+   return err;
+   }
  
  	ii = btrfs_item_ptr(node, slot, struct btrfs_inode_item);

isize = btrfs_inode_size(node, ii);
@@ -6561,7 +6614,6 @@ static int check_inode_item(struct btrfs_root *root, 
struct btrfs_path *path,
btrfs_item_key_to_cpu(node, , slot);
if (key.objectid != inode_id)
goto out;
-
switch (key.type) {
case BTRFS_INODE_REF_KEY:
ret = check_inode_ref(root, , path, namebuf,
@@ -6608,12 +6660,10 @@ static int check_inode_item(struct btrfs_root *root, 
struct btrfs_path *path,
}
  
  out:

-   if (err & LAST_ITEM) {

Re: [PATCH 12/15] btrfs-progs: check: increase counter error in check_inode_recs()

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> Counter @error decides return values of check_inode_recs().
> 
> Previously, @error won't be incremented even repair_inode_recs() failed.
> It causes 'btrfs check --repair' prints some error information but
> returns 0.
> 
> So, if root dir is missing and repair is disabled, @error should be
> incremented.
> And after repair_inode_recs(), increase @error if any errors in inodes
> and backrefs.
> 
> Signed-off-by: Su Yue 

Reviewed-by: Qu Wenruo 

Thanks,
Qu

> ---
>  cmds-check.c | 6 --
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index b23a4493b12b..a83f0a92f48b 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -4137,6 +4137,7 @@ static int check_inode_recs(struct btrfs_root *root,
>   return -EAGAIN;
>   }
>  
> + error++;
>   fprintf(stderr, "root %llu root dir %llu not found\n",
>   (unsigned long long)root->root_key.objectid,
>   (unsigned long long)root_dirid);
> @@ -4176,10 +4177,9 @@ static int check_inode_recs(struct btrfs_root *root,
>   free_inode_rec(rec);
>   continue;
>   }
> - ret = 0;
>   }
>  
> - if (!(repair && ret == 0))
> + if (rec->errors)
>   error++;
>   print_inode_error(root, rec);
>   list_for_each_entry(backref, >backrefs, list) {
> @@ -4189,6 +4189,8 @@ static int check_inode_recs(struct btrfs_root *root,
>   backref->errors |= REF_ERR_NO_DIR_INDEX;
>   if (!backref->found_inode_ref)
>   backref->errors |= REF_ERR_NO_INODE_REF;
> + if (backref->errors)
> + error++;
>   fprintf(stderr, "\tunresolved ref dir %llu index %llu"
>   " namelen %u name %s filetype %d errors %x",
>   (unsigned long long)backref->dir,
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 10/15] btrfs-progs: check: clear I_ERR_FILE_EXTENT_DISCOUNT after repair

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> Original check forgets to clear bit I_ERR_FILE_EXTENT_DISCOUNT
> in rec->errors after repair.
> 
> So just do it.
> 
> Signed-off-by: Su Yue 

Reviewed-by: Qu Wenruo 

And a test case for this would help us to avoid further regression.

Thanks,
Qu

> ---
>  cmds-check.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index ae0a9e146399..d8d9a3227c06 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -3978,6 +3978,7 @@ static int repair_inode_discount_extent(struct 
> btrfs_trans_handle *trans,
>   if (ret < 0)
>   goto out;
>   }
> + rec->errors &= ~I_ERR_FILE_EXTENT_DISCOUNT;
>   printf("Fixed discount file extents for inode: %llu in root: %llu\n",
>  rec->ino, root->objectid);
>  out:
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 09/15] btrfs-progs: lowmem check: change logic of leaf process if repair

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> In lowmem check without repair, process_one_leaf_v2() will process one
> entire leaf and inner check_inode_item() leads path point next
> leaf.
> In the beginning, process_one_leaf_v2() will let path point fist inode
> item or first position where inode id changed.
> 
> However, in lowmem repair, process_one_leaf_v2() will be interrupted
> to process one leaf because repair will CoW the leaf. Then some items
> unprocessed is skipped.
> Since repair may also delete some items, we can't use tricks like
> record last checked key.
> 
> So, only for lowmem repair:
> 1. check_inode_item is responsible for handle case missing inode item.

I think the idea to use inode item as the indicator is a good idea.
And since only check_inode_item() can delete inode item, let it to
handle the path is reasonable.

> 2. process_one_leaf_v2() do not modify path manually, and check_inode()
>promise that @path points last checked item.
>Only when something are fixed, process_one_leaf_v2() will continue
>to check in next round.
> 
> Signed-off-by: Su Yue 
> ---
>  cmds-check.c | 68 
> 
>  1 file changed, 59 insertions(+), 9 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index e57eea4e61c9..ae0a9e146399 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -2007,6 +2007,8 @@ static int process_one_leaf_v2(struct btrfs_root *root, 
> struct btrfs_path *path,
>  
>   cur_bytenr = cur->start;
>  
> + if (repair)
> + goto again;
>   /* skip to first inode item or the first inode number change */
>   nritems = btrfs_header_nritems(cur);
>   for (i = 0; i < nritems; i++) {
> @@ -2033,9 +2035,12 @@ again:
>   goto out;
>  
>   /* still have inode items in thie leaf */
> - if (cur->start == cur_bytenr)
> + if (cur->start == cur_bytenr) {
> + ret = btrfs_next_item(root, path);
> + if (ret > 0)
> + goto out;
>   goto again;
> -
> + }
>   /*
>* we have switched to another leaf, above nodes may
>* have changed, here walk down the path, if a node
> @@ -5721,6 +5726,8 @@ static int repair_dir_item(struct btrfs_root *root, 
> struct btrfs_key *key,
> ino);
>   }
>   }
> + } else {
> + true_filetype = filetype;

This modification doesn't seems related to this patch.

Maybe it's better to move it 4th patch?

>   }
>  
>   /*
> @@ -6489,6 +6496,45 @@ out:
>   return ret;
>  }
>  
> +/*
> + * Try insert new inode item frist.
> + * If failed, jump to next inode item.
> + */
> +static int handle_inode_item_missing(struct btrfs_root *root,
> +  struct btrfs_path *path)
> +{
> + struct btrfs_key key;
> + int ret;
> +
> + btrfs_item_key_to_cpu(path->nodes[0], , path->slots[0]);
> +
> + ret = repair_inode_item_missing(root, key.objectid, 0);
> + if (!ret) {
> + btrfs_release_path(path);
> + ret = btrfs_search_slot(NULL, root, , path, 0, 0);
> + if (ret)
> + goto next_inode;
> + else
> + goto out;
> + }
> +
> +next_inode:
> + error("inode item[%llu] is missing, skip to check next inode",
> +   key.objectid);
> + while (1) {
> + ret = btrfs_next_item(root, path);
> + if (ret > 0)
> + goto out;

ret < 0 case is not handled.

> + btrfs_item_key_to_cpu(path->nodes[0], , path->slots[0]);
> + if (key.type == BTRFS_INODE_ITEM_KEY) {
> + ret = 0;
> + break;
> + }
> + }
> +out:
> + return ret;
> +}
> +
>  /*
>   * Check INODE_ITEM and related ITEMs (the same inode number)
>   * 1. check link count
> @@ -6536,6 +6582,13 @@ static int check_inode_item(struct btrfs_root *root, 
> struct btrfs_path *path,
>   err |= LAST_ITEM;
>   return err;
>   }
> + if (key.type != BTRFS_INODE_ITEM_KEY && repair) {
> + ret = handle_inode_item_missing(root, path);
> + if (ret > 0)
> + err |= LAST_ITEM;
> + if (ret)
> + return err;
> + }
>  
>   ii = btrfs_item_ptr(node, slot, struct btrfs_inode_item);
>   isize = btrfs_inode_size(node, ii);
> @@ -6561,7 +6614,6 @@ static int check_inode_item(struct btrfs_root *root, 
> struct btrfs_path *path,
>   btrfs_item_key_to_cpu(node, , slot);
>   if (key.objectid != inode_id)
>   goto out;
> -
>   switch (key.type) {
>   case BTRFS_INODE_REF_KEY:
>   ret = check_inode_ref(root, , path, namebuf,
> @@ -6608,12 +6660,10 @@ static int check_inode_item(struct btrfs_root *root, 

Re: [PATCH 06/15] btrfs-progs: lowmem check: let check_dir_item() return if repaired

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> If repair_dir_item deleted the item, goto last checked then returns
> instead of searching di_key again then returns -ENOENT.
> 
> Signed-off-by: Su Yue 
> ---
>  cmds-check.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index eb65a18fe64b..4ce6139b3ab1 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -5931,7 +5931,7 @@ begin:
>   path->slots[0]--;

Well, not a problem of this patch, but I found previous lines have problem:
--
ret = btrfs_search_slot(NULL, root, di_key, path, 0, 0);
/* the item was deleted, let path point the last checked item */
if (ret > 0) {
if (path->slots[0] == 0)
btrfs_prev_leaf(root, path);
else
path->slots[0]--;
}
--

Here return value of btrfs_prev_leaf() is ignored, which can return
error or >1.

Thanks,
Qu

>   }
>   if (ret)
> - goto out;
> + return err;>}
>  
>   node = path->nodes[0];
> @@ -6040,7 +6040,7 @@ next:
>   break;
>   }
>   }
> -out:
> +
>   /* research path */
>   btrfs_release_path(path);
>   ret = btrfs_search_slot(NULL, root, di_key, path, 0, 0);
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 07/15] btrfs-progs: lowmem check: find_dir_item by di_key in check_dir_item()

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> In check_dir_item, we are going to search corresponding
> dir_item/index.
> @key shouldn't be used here. It should be @di_key.
> 
> Signed-off-by: Su Yue 

Reviewed-by: Qu Wenruo 

Thanks,
Qu

> ---
>  cmds-check.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index 4ce6139b3ab1..caac71a67472 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -6001,7 +6001,7 @@ begin:
>  
>   /* check relative INDEX/ITEM */
>   key.objectid = di_key->objectid;
> - if (key.type == BTRFS_DIR_ITEM_KEY) {
> + if (di_key->type == BTRFS_DIR_ITEM_KEY) {
>   key.type = BTRFS_DIR_INDEX_KEY;
>   key.offset = index;
>   } else {
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 05/15] btrfs-progs: lowmem check: let check_dir_item() continue if find wrong inode_item

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> check_dir_item can handle missed/mismatched inode item well.
> Let it continue to check corresponding dir_item/index and inode_ref.
> 
> Signed-off-by: Su Yue 

Reviewed-by: Qu Wenruo 

Thanks,
Qu

> ---
>  cmds-check.c | 15 ++-
>  1 file changed, 6 insertions(+), 9 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index e33dd7db0048..eb65a18fe64b 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -5984,15 +5984,12 @@ begin:
>   ret = btrfs_search_slot(NULL, root, , path, 0, 0);
>   if (ret) {
>   tmp_err |= INODE_ITEM_MISSING;
> - goto next;
> - }
> -
> - ii = btrfs_item_ptr(path->nodes[0], path->slots[0],
> - struct btrfs_inode_item);
> - mode = btrfs_inode_mode(path->nodes[0], ii);
> - if (imode_to_type(mode) != filetype) {
> - tmp_err |= INODE_ITEM_MISMATCH;
> - goto next;
> + } else {
> + ii = btrfs_item_ptr(path->nodes[0], path->slots[0],
> + struct btrfs_inode_item);
> + mode = btrfs_inode_mode(path->nodes[0], ii);
> + if (imode_to_type(mode) != filetype)
> + tmp_err |= INODE_ITEM_MISMATCH;
>   }
>  
>   /* Check relative INODE_REF/INODE_EXTREF */
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 02/15] btrfs-progs: lowmem check: find and guess inode filetype

[Qu Wenruo]

On 2018年01月26日 17:31, Su Yue wrote:
> 
> 
> On 01/26/2018 05:14 PM, Qu Wenruo wrote:
>>
>>
>> On 2018年01月26日 16:35, Su Yue wrote:
>>> Introduce find_file_type_lowmem() and guess_file_type_lowmem().
>>>
>>> find_file_type_lowmem() gets filetypes from inode_item, dir_item and
>>> dir_index. If two of three's filetype are same and valid, it thinks
>>> the value is correct.
>>>
>>> guess_file_type_lowmem() searches file_extent and dir_item/index then
>>> returns with filetype.
>>>
>>> Signed-off-by: Su Yue 
>>> ---
>>>   cmds-check.c | 193
>>> +++
>>>   1 file changed, 193 insertions(+)
>>>
>>> diff --git a/cmds-check.c b/cmds-check.c
>>> index e3505a7f9d6b..b200fdccf0e5 100644
>>> --- a/cmds-check.c
>>> +++ b/cmds-check.c
>>> @@ -3306,6 +3306,199 @@ static int find_file_type(struct inode_record
>>> *rec, u8 *type)
>>>   return -ENOENT;
>>>   }
>>>   +/*
>>> + * Fetch filetype from exited completed dir_item, dir_index and
>>> inode_item.
>>    ^^
>>    existing?
> 
> Oh.. a Typo.
>>> + * If two of tree items'filetype are same, we think the type is
>>> trusted.
>>> + *
>>> + * Return 0 if file type is found and BTRFS_FT_* is stored into type.
>>> + * Return <0 if file type is not found.
>>
>> This also includes extra error like -EIO from btrfs_search_slot().
>>
>>> + */
>>> +static int find_file_type_lowmem(struct btrfs_root *root, u64 ino,
>>> u8 *type)
>>> +{
>>> +    struct btrfs_key key;
>>> +    struct btrfs_path path;
>>> +    struct btrfs_path path2;
>>> +    struct btrfs_inode_ref *iref;
>>> +    struct btrfs_dir_item *dir_item;
>>> +    struct btrfs_dir_item *dir_index;
>>> +    struct extent_buffer *eb;
>>> +    u64 dir;
>>> +    u64 index;
>>> +    char namebuf[BTRFS_NAME_LEN] = {0};
>>> +    u32 namelen;
>>> +    u8 inode_filetype = BTRFS_FT_UNKNOWN;
>>> +    u8 dir_item_filetype;
>>> +    u8 dir_index_filetype;
>>> +    u8 true_file_type;
>>> +    int slot;
>>> +    int ret;
>>> +
>>> +    key.objectid = ino;
>>> +    key.type = BTRFS_INODE_ITEM_KEY;
>>> +    key.offset = 0;
>>> +
>>> +    btrfs_init_path();
>>> +    ret = btrfs_search_slot(NULL, root, , , 0, 0);
>>> +    if (ret < 0)
>>> +    goto out;
>>> +    if (!ret) {
>>> +    struct btrfs_inode_item *ii;
>>> +
>>> +    ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
>>> +    struct btrfs_inode_item);
>>> +    inode_filetype = imode_to_type(btrfs_inode_mode(path.nodes[0],
>>> +    ii));
>>> +    }
>>> +
>>> +    key.objectid = ino;
>>> +    key.type = BTRFS_INODE_REF_KEY;
>>> +    key.offset = (u64)-1;
>>> +
>>> +    btrfs_release_path();
>>> +    ret = btrfs_search_slot(NULL, root, , , 0, 0);
>>> +    if (ret < 0)
>>> +    goto out;
>>> +    if (!ret) {
>>> +    ret = -EIO;
>>> +    goto out;
>>> +    }
>>> +
>>> +    btrfs_init_path();
>>> +next:
>>> +    btrfs_release_path()> +    ret = btrfs_previous_item(root,
>>> , ino, BTRFS_INODE_REF_KEY);
>>> +    if (ret) {
>>> +    ret = -ENOENT;
>>
>> For ret < 0 case, return value is overwritten.
>>
> Will fix it.
>>
>>> +    goto out;
>>> +    }
>>> +
>>> +    eb = path.nodes[0];
>>> +    slot = path.slots[0];
>>> +    btrfs_item_key_to_cpu(eb, , slot);
>>> +    dir = key.offset;
>>> +    iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
>>> +    index = btrfs_inode_ref_index(eb, iref);
>>> +    namelen = btrfs_inode_ref_name_len(eb, iref);
>>> +    read_extent_buffer(eb, namebuf, (unsigned long)(iref + 1),
>>> namelen);
>>> +
>>> +    dir_index = btrfs_lookup_dir_index(NULL, root, , dir,
>>> namebuf,
>>> +   namelen, index, 0);
>>> +    if (!dir_index)
>>> +    goto next;
>>> +    dir_index_filetype = btrfs_dir_type(path2.nodes[0], dir_index);
>>> +    btrfs_release_path();
>>> +    if (dir_index_filetype == inode_filetype) {
>>> +    true_file_type = inode_filetype;
>>> +    goto found;
>>> +    }
>>> +
>>> +    dir_item = btrfs_lookup_dir_item(NULL, root, , dir, namebuf,
>>> + namelen, 0);
>>> +    if (!dir_item)
>>> +    goto next;
>>> +    dir_item_filetype = btrfs_dir_type(path2.nodes[0], dir_item);
>>> +    btrfs_release_path();
>>> +    if (dir_item_filetype == inode_filetype) {
>>> +    true_file_type = inode_filetype;
>>> +    goto found;
>>> +    }
>>> +
>>> +    if (dir_index_filetype == dir_item_filetype) {
>>> +    true_file_type = dir_index_filetype;
>>> +    goto found;
>>> +    }
>>> +    goto next;
>>> +found:
>>> +    /* rare case, two of three items are both corrupted */
>>> +    if (true_file_type == BTRFS_FT_UNKNOWN ||
>>> +    true_file_type >= BTRFS_FT_MAX)
>>> +    goto next;
>>> +    *type = true_file_type;
>>> +    ret = 0;
>>> +out:
>>> +    btrfs_release_path();
>>> +    return ret;
>>> +}
>>> +
>>> +static int find_normal_file_extent(struct btrfs_root *root, u64 

Re: [PATCH 04/15] btrfs-progs: lowmem check: repair complex cases in repair_dir_item()

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> If inode item is missing or filetype is corrupted maybe, and filetypes
> of dir_item and dir_index are corrupted too, lowmem repair may insert
> wrong inode item and dir_item/index.
> 
> First, find and guess filetype of inode item, if failed, use
> BTRFS_REG_FILE as fallback for insertion.
> If filetype is not available, just delete current dir_item/index.
> 
> And repair_dir_item also calls repair_inode_item_mismatch() now.
> 
> Signed-off-by: Su Yue 

Looks good.

Reviewed-by: Qu Wenruo 

Thanks,
Qu
> ---
>  cmds-check.c | 95 
> +---
>  1 file changed, 78 insertions(+), 17 deletions(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index 08a2662e603c..e33dd7db0048 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -2811,7 +2811,7 @@ static int walk_down_tree_v2(struct btrfs_trans_handle 
> *trans,
>  
>   ret = check_child_node(cur, path->slots[*level], next);
>   err |= ret;
> - if (ret < 0) 
> + if (ret < 0)
>   break;
>  
>   if (btrfs_is_leaf(next))
> @@ -5697,29 +5697,91 @@ static void print_dir_item_err(struct btrfs_root 
> *root, struct btrfs_key *key,
>  /*
>   * Call repair_inode_item_missing and repair_ternary_lowmem to repair
>   *
> + * @filetype:filetype of the dir_item/index
> + *
>   * Returns error after repair
>   */
> -static int repair_dir_item(struct btrfs_root *root, u64 dirid, u64 ino,
> -u64 index, u8 filetype, char *namebuf, u32 name_len,
> -int err)
> +static int repair_dir_item(struct btrfs_root *root, struct btrfs_key *key,
> +u64 ino, u64 index, u8 filetype, char *namebuf,
> +u32 name_len, int err)
>  {
>   int ret;
> + u64 dirid = key->objectid;
> + u8 true_filetype;
>  
> - if (err & INODE_ITEM_MISSING) {
> - ret = repair_inode_item_missing(root, ino, filetype);
> - if (!ret)
> - err &= ~(INODE_ITEM_MISMATCH | INODE_ITEM_MISSING);
> + if (err & (INODE_ITEM_MISMATCH | INODE_ITEM_MISSING)) {
> + ret = find_file_type_lowmem(root, ino, _filetype);
> + if (ret) {
> + ret = guess_file_type_lowmem(root, ino,
> +  _filetype);
> + if (ret) {
> + true_filetype = BTRFS_FT_REG_FILE;
> + error(
> + "can't get file type for inode %llu, using FILE as fallback",
> +   ino);
> + }
> + }
>   }
>  
> - if (err & ~(INODE_ITEM_MISMATCH | INODE_ITEM_MISSING)) {
> - ret = repair_ternary_lowmem(root, dirid, ino, index, namebuf,
> - name_len, filetype, err);
> + /*
> +  * Case: the dir_item has corresponding inode_ref but
> +  * mismatch/missed inode_item and mismatch/missed another
> +  * dir_item/index.
> +  * repair_ternary_lowmem prefer to change another dir_item/index with
> +  * wrong filetype. So delete the item here.
> +  */
> + if (filetype != true_filetype &&
> + (err & (DIR_ITEM_MISMATCH | DIR_ITEM_MISSING) ||
> +  err & (DIR_INDEX_MISMATCH | DIR_INDEX_MISSING))) {
> + struct btrfs_trans_handle *trans;
> + struct btrfs_path *path;
> +
> + path = btrfs_alloc_path();
> + if (!path)
> + goto out;
> + trans = btrfs_start_transaction(root, 0);
> + ret = btrfs_search_slot(trans, root, key, path, -1, 1);
> + if (ret) {
> + btrfs_commit_transaction(trans, root);
> + btrfs_release_path(path);
> + goto out;
> + }
> + ret = btrfs_del_item(trans, root, path);
>   if (!ret) {
> - err &= ~(DIR_INDEX_MISMATCH | DIR_INDEX_MISSING);
> - err &= ~(DIR_ITEM_MISMATCH | DIR_ITEM_MISSING);
> - err &= ~(INODE_REF_MISSING);
> + err = 0;
> + printf(
> + "Deleted dir_item[%llu %u %llu]root %llu name %s filetype %u\n",
> +key->objectid, key->type, key->offset,
> +root->objectid, namebuf, filetype);
>   }
> + btrfs_commit_transaction(trans, root);
> + btrfs_release_path(path);
> + /*
> +  * Leave remains to check_inode_item() and check_inode_ref().
> +  */
> + goto out;
> + }
> +
> + ret = repair_ternary_lowmem(root, dirid, ino, index, namebuf,
> + name_len, true_filetype, err);
> + if (!ret) {
> +  

Re: [PATCH 02/15] btrfs-progs: lowmem check: find and guess inode filetype

[Su Yue]

On 01/26/2018 05:14 PM, Qu Wenruo wrote:



On 2018年01月26日 16:35, Su Yue wrote:

Introduce find_file_type_lowmem() and guess_file_type_lowmem().

find_file_type_lowmem() gets filetypes from inode_item, dir_item and
dir_index. If two of three's filetype are same and valid, it thinks
the value is correct.

guess_file_type_lowmem() searches file_extent and dir_item/index then
returns with filetype.

Signed-off-by: Su Yue 
---
  cmds-check.c | 193 +++
  1 file changed, 193 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index e3505a7f9d6b..b200fdccf0e5 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3306,6 +3306,199 @@ static int find_file_type(struct inode_record *rec, u8 
*type)
return -ENOENT;
  }
  
+/*

+ * Fetch filetype from exited completed dir_item, dir_index and inode_item.

   ^^
   existing?


Oh.. a Typo.

+ * If two of tree items'filetype are same, we think the type is trusted.
+ *
+ * Return 0 if file type is found and BTRFS_FT_* is stored into type.
+ * Return <0 if file type is not found.


This also includes extra error like -EIO from btrfs_search_slot().


+ */
+static int find_file_type_lowmem(struct btrfs_root *root, u64 ino, u8 *type)
+{
+   struct btrfs_key key;
+   struct btrfs_path path;
+   struct btrfs_path path2;
+   struct btrfs_inode_ref *iref;
+   struct btrfs_dir_item *dir_item;
+   struct btrfs_dir_item *dir_index;
+   struct extent_buffer *eb;
+   u64 dir;
+   u64 index;
+   char namebuf[BTRFS_NAME_LEN] = {0};
+   u32 namelen;
+   u8 inode_filetype = BTRFS_FT_UNKNOWN;
+   u8 dir_item_filetype;
+   u8 dir_index_filetype;
+   u8 true_file_type;
+   int slot;
+   int ret;
+
+   key.objectid = ino;
+   key.type = BTRFS_INODE_ITEM_KEY;
+   key.offset = 0;
+
+   btrfs_init_path();
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret < 0)
+   goto out;
+   if (!ret) {
+   struct btrfs_inode_item *ii;
+
+   ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
+   struct btrfs_inode_item);
+   inode_filetype = imode_to_type(btrfs_inode_mode(path.nodes[0],
+   ii));
+   }
+
+   key.objectid = ino;
+   key.type = BTRFS_INODE_REF_KEY;
+   key.offset = (u64)-1;
+
+   btrfs_release_path();
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret < 0)
+   goto out;
+   if (!ret) {
+   ret = -EIO;
+   goto out;
+   }
+
+   btrfs_init_path();
+next:
+   btrfs_release_path()> +ret = btrfs_previous_item(root, , 
ino, BTRFS_INODE_REF_KEY);
+   if (ret) {
+   ret = -ENOENT;


For ret < 0 case, return value is overwritten.


Will fix it.



+   goto out;
+   }
+
+   eb = path.nodes[0];
+   slot = path.slots[0];
+   btrfs_item_key_to_cpu(eb, , slot);
+   dir = key.offset;
+   iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
+   index = btrfs_inode_ref_index(eb, iref);
+   namelen = btrfs_inode_ref_name_len(eb, iref);
+   read_extent_buffer(eb, namebuf, (unsigned long)(iref + 1), namelen);
+
+   dir_index = btrfs_lookup_dir_index(NULL, root, , dir, namebuf,
+  namelen, index, 0);
+   if (!dir_index)
+   goto next;
+   dir_index_filetype = btrfs_dir_type(path2.nodes[0], dir_index);
+   btrfs_release_path();
+   if (dir_index_filetype == inode_filetype) {
+   true_file_type = inode_filetype;
+   goto found;
+   }
+
+   dir_item = btrfs_lookup_dir_item(NULL, root, , dir, namebuf,
+namelen, 0);
+   if (!dir_item)
+   goto next;
+   dir_item_filetype = btrfs_dir_type(path2.nodes[0], dir_item);
+   btrfs_release_path();
+   if (dir_item_filetype == inode_filetype) {
+   true_file_type = inode_filetype;
+   goto found;
+   }
+
+   if (dir_index_filetype == dir_item_filetype) {
+   true_file_type = dir_index_filetype;
+   goto found;
+   }
+   goto next;
+found:
+   /* rare case, two of three items are both corrupted */
+   if (true_file_type == BTRFS_FT_UNKNOWN ||
+   true_file_type >= BTRFS_FT_MAX)
+   goto next;
+   *type = true_file_type;
+   ret = 0;
+out:
+   btrfs_release_path();
+   return ret;
+}
+
+static int find_normal_file_extent(struct btrfs_root *root, u64 ino);
+/*
+ * Try to determine inode type if type not found.
+ *
+ * For found regular file extent, it must be FILE.
+ * For found dir_item/index, it must be DIR.
+ *
+ * Return 0 if file type is confirmed and 

Re: [PATCH 03/15] btrfs-progs: lowmem check: find filetype in repair_inode_missing()

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> If parameter @filetype is 0, repair_inode_missing will find filetype
> automatically.
> 
> And let it return -EEXIST instead of 0 if inode item is existed.
> 
> Signed-off-by: Su Yue 

Reviewed-by: Qu Wenruo 

Thanks,
Qu

> ---
>  cmds-check.c | 20 +++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index b200fdccf0e5..08a2662e603c 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -5161,6 +5161,9 @@ out:
>  /*
>   * Insert the missing inode item.
>   *
> + * @filetype: if 0, find file type automatically.
> + *  if find nothing, set inode as regular file.
> + *
>   * Returns 0 means success.
>   * Returns <0 means error.
>   */
> @@ -5176,6 +5179,19 @@ static int repair_inode_item_missing(struct btrfs_root 
> *root, u64 ino,
>   key.type = BTRFS_INODE_ITEM_KEY;
>   key.offset = 0;
>  
> + if (!filetype) {
> + ret = find_file_type_lowmem(root, ino, );
> + if (ret) {
> + ret = guess_file_type_lowmem(root, ino, );
> + if (ret) {
> + filetype = BTRFS_FT_REG_FILE;
> + error(
> + "can't get file type for inode %llu, using FILE as fallback",
> +   ino);
> + }
> + }
> + }
> +
>   btrfs_init_path();
>   trans = btrfs_start_transaction(root, 1);
>   if (IS_ERR(trans)) {
> @@ -5184,7 +5200,9 @@ static int repair_inode_item_missing(struct btrfs_root 
> *root, u64 ino,
>   }
>  
>   ret = btrfs_search_slot(trans, root, , , 1, 1);
> - if (ret < 0 || !ret)
> + if (!ret)
> + ret = -EEXIST;
> + if (ret < 0)
>   goto fail;
>  
>   /* insert inode item */
> 



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 02/15] btrfs-progs: lowmem check: find and guess inode filetype

[Qu Wenruo]

On 2018年01月26日 17:14, Qu Wenruo wrote:
> 
> 
> On 2018年01月26日 16:35, Su Yue wrote:
>> Introduce find_file_type_lowmem() and guess_file_type_lowmem().
>>
>> find_file_type_lowmem() gets filetypes from inode_item, dir_item and
>> dir_index. If two of three's filetype are same and valid, it thinks
>> the value is correct.
>>
>> guess_file_type_lowmem() searches file_extent and dir_item/index then
>> returns with filetype.
>>
>> Signed-off-by: Su Yue 
>> ---
>>  cmds-check.c | 193 
>> +++
>>  1 file changed, 193 insertions(+)
>>
>> diff --git a/cmds-check.c b/cmds-check.c
>> index e3505a7f9d6b..b200fdccf0e5 100644
>> --- a/cmds-check.c
>> +++ b/cmds-check.c
>> @@ -3306,6 +3306,199 @@ static int find_file_type(struct inode_record *rec, 
>> u8 *type)
>>  return -ENOENT;
>>  }
>>  
>> +/*
>> + * Fetch filetype from exited completed dir_item, dir_index and inode_item.
>   ^^
>   existing?
>> + * If two of tree items'filetype are same, we think the type is trusted.
>> + *
>> + * Return 0 if file type is found and BTRFS_FT_* is stored into type.
>> + * Return <0 if file type is not found.
> 
> This also includes extra error like -EIO from btrfs_search_slot().
> 
>> + */
>> +static int find_file_type_lowmem(struct btrfs_root *root, u64 ino, u8 *type)
>> +{
>> +struct btrfs_key key;
>> +struct btrfs_path path;
>> +struct btrfs_path path2;
>> +struct btrfs_inode_ref *iref;
>> +struct btrfs_dir_item *dir_item;
>> +struct btrfs_dir_item *dir_index;
>> +struct extent_buffer *eb;
>> +u64 dir;
>> +u64 index;
>> +char namebuf[BTRFS_NAME_LEN] = {0};
>> +u32 namelen;
>> +u8 inode_filetype = BTRFS_FT_UNKNOWN;
>> +u8 dir_item_filetype;
>> +u8 dir_index_filetype;
>> +u8 true_file_type;
>> +int slot;
>> +int ret;
>> +
>> +key.objectid = ino;
>> +key.type = BTRFS_INODE_ITEM_KEY;
>> +key.offset = 0;
>> +
>> +btrfs_init_path();
>> +ret = btrfs_search_slot(NULL, root, , , 0, 0);
>> +if (ret < 0)
>> +goto out;
>> +if (!ret) {
>> +struct btrfs_inode_item *ii;
>> +
>> +ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
>> +struct btrfs_inode_item);
>> +inode_filetype = imode_to_type(btrfs_inode_mode(path.nodes[0],
>> +ii));
>> +}
>> +
>> +key.objectid = ino;
>> +key.type = BTRFS_INODE_REF_KEY;
>> +key.offset = (u64)-1;
>> +
>> +btrfs_release_path();
>> +ret = btrfs_search_slot(NULL, root, , , 0, 0);
>> +if (ret < 0)
>> +goto out;
>> +if (!ret) {
>> +ret = -EIO;
>> +goto out;
>> +}
>> +
>> +btrfs_init_path();
>> +next:
>> +btrfs_release_path()> +   ret = btrfs_previous_item(root, , 
>> ino, BTRFS_INODE_REF_KEY);
>> +if (ret) {
>> +ret = -ENOENT;
> 
> For ret < 0 case, return value is overwritten.
> 
> 
>> +goto out;
>> +}
>> +
>> +eb = path.nodes[0];
>> +slot = path.slots[0];
>> +btrfs_item_key_to_cpu(eb, , slot);
>> +dir = key.offset;
>> +iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
>> +index = btrfs_inode_ref_index(eb, iref);
>> +namelen = btrfs_inode_ref_name_len(eb, iref);
>> +read_extent_buffer(eb, namebuf, (unsigned long)(iref + 1), namelen);
>> +
>> +dir_index = btrfs_lookup_dir_index(NULL, root, , dir, namebuf,
>> +   namelen, index, 0);
>> +if (!dir_index)
>> +goto next;
>> +dir_index_filetype = btrfs_dir_type(path2.nodes[0], dir_index);
>> +btrfs_release_path();
>> +if (dir_index_filetype == inode_filetype) {
>> +true_file_type = inode_filetype;
>> +goto found;
>> +}
>> +
>> +dir_item = btrfs_lookup_dir_item(NULL, root, , dir, namebuf,
>> + namelen, 0);
>> +if (!dir_item)
>> +goto next;
>> +dir_item_filetype = btrfs_dir_type(path2.nodes[0], dir_item);
>> +btrfs_release_path();
>> +if (dir_item_filetype == inode_filetype) {
>> +true_file_type = inode_filetype;
>> +goto found;
>> +}
>> +
>> +if (dir_index_filetype == dir_item_filetype) {
>> +true_file_type = dir_index_filetype;
>> +goto found;
>> +}
>> +goto next;
>> +found:
>> +/* rare case, two of three items are both corrupted */
>> +if (true_file_type == BTRFS_FT_UNKNOWN ||
>> +true_file_type >= BTRFS_FT_MAX)
>> +goto next;
>> +*type = true_file_type;
>> +ret = 0;
>> +out:
>> +btrfs_release_path();
>> +return ret;
>> +}
>> +
>> +static int find_normal_file_extent(struct btrfs_root *root, u64 ino);
>> +/*
>> + * Try to determine inode type if type not found.
>> + *
>> + * For found regular 

Re: [PATCH 02/15] btrfs-progs: lowmem check: find and guess inode filetype

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> Introduce find_file_type_lowmem() and guess_file_type_lowmem().
> 
> find_file_type_lowmem() gets filetypes from inode_item, dir_item and
> dir_index. If two of three's filetype are same and valid, it thinks
> the value is correct.
> 
> guess_file_type_lowmem() searches file_extent and dir_item/index then
> returns with filetype.
> 
> Signed-off-by: Su Yue 
> ---
>  cmds-check.c | 193 
> +++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index e3505a7f9d6b..b200fdccf0e5 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -3306,6 +3306,199 @@ static int find_file_type(struct inode_record *rec, 
> u8 *type)
>   return -ENOENT;
>  }
>  
> +/*
> + * Fetch filetype from exited completed dir_item, dir_index and inode_item.
  ^^
  existing?
> + * If two of tree items'filetype are same, we think the type is trusted.
> + *
> + * Return 0 if file type is found and BTRFS_FT_* is stored into type.
> + * Return <0 if file type is not found.

This also includes extra error like -EIO from btrfs_search_slot().

> + */
> +static int find_file_type_lowmem(struct btrfs_root *root, u64 ino, u8 *type)
> +{
> + struct btrfs_key key;
> + struct btrfs_path path;
> + struct btrfs_path path2;
> + struct btrfs_inode_ref *iref;
> + struct btrfs_dir_item *dir_item;
> + struct btrfs_dir_item *dir_index;
> + struct extent_buffer *eb;
> + u64 dir;
> + u64 index;
> + char namebuf[BTRFS_NAME_LEN] = {0};
> + u32 namelen;
> + u8 inode_filetype = BTRFS_FT_UNKNOWN;
> + u8 dir_item_filetype;
> + u8 dir_index_filetype;
> + u8 true_file_type;
> + int slot;
> + int ret;
> +
> + key.objectid = ino;
> + key.type = BTRFS_INODE_ITEM_KEY;
> + key.offset = 0;
> +
> + btrfs_init_path();
> + ret = btrfs_search_slot(NULL, root, , , 0, 0);
> + if (ret < 0)
> + goto out;
> + if (!ret) {
> + struct btrfs_inode_item *ii;
> +
> + ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
> + struct btrfs_inode_item);
> + inode_filetype = imode_to_type(btrfs_inode_mode(path.nodes[0],
> + ii));
> + }
> +
> + key.objectid = ino;
> + key.type = BTRFS_INODE_REF_KEY;
> + key.offset = (u64)-1;
> +
> + btrfs_release_path();
> + ret = btrfs_search_slot(NULL, root, , , 0, 0);
> + if (ret < 0)
> + goto out;
> + if (!ret) {
> + ret = -EIO;
> + goto out;
> + }
> +
> + btrfs_init_path();
> +next:
> + btrfs_release_path()> +   ret = btrfs_previous_item(root, , 
> ino, BTRFS_INODE_REF_KEY);
> + if (ret) {
> + ret = -ENOENT;

For ret < 0 case, return value is overwritten.


> + goto out;
> + }
> +
> + eb = path.nodes[0];
> + slot = path.slots[0];
> + btrfs_item_key_to_cpu(eb, , slot);
> + dir = key.offset;
> + iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
> + index = btrfs_inode_ref_index(eb, iref);
> + namelen = btrfs_inode_ref_name_len(eb, iref);
> + read_extent_buffer(eb, namebuf, (unsigned long)(iref + 1), namelen);
> +
> + dir_index = btrfs_lookup_dir_index(NULL, root, , dir, namebuf,
> +namelen, index, 0);
> + if (!dir_index)
> + goto next;
> + dir_index_filetype = btrfs_dir_type(path2.nodes[0], dir_index);
> + btrfs_release_path();
> + if (dir_index_filetype == inode_filetype) {
> + true_file_type = inode_filetype;
> + goto found;
> + }
> +
> + dir_item = btrfs_lookup_dir_item(NULL, root, , dir, namebuf,
> +  namelen, 0);
> + if (!dir_item)
> + goto next;
> + dir_item_filetype = btrfs_dir_type(path2.nodes[0], dir_item);
> + btrfs_release_path();
> + if (dir_item_filetype == inode_filetype) {
> + true_file_type = inode_filetype;
> + goto found;
> + }
> +
> + if (dir_index_filetype == dir_item_filetype) {
> + true_file_type = dir_index_filetype;
> + goto found;
> + }
> + goto next;
> +found:
> + /* rare case, two of three items are both corrupted */
> + if (true_file_type == BTRFS_FT_UNKNOWN ||
> + true_file_type >= BTRFS_FT_MAX)
> + goto next;
> + *type = true_file_type;
> + ret = 0;
> +out:
> + btrfs_release_path();
> + return ret;
> +}
> +
> +static int find_normal_file_extent(struct btrfs_root *root, u64 ino);
> +/*
> + * Try to determine inode type if type not found.
> + *
> + * For found regular file extent, it must be FILE.
> + * For found dir_item/index, it must be DIR.
> + *
> + * Return 0 if file type is 

Re: [PATCH 02/15] btrfs-progs: lowmem check: find and guess inode filetype

[Qu Wenruo]

On 2018年01月26日 16:35, Su Yue wrote:
> Introduce find_file_type_lowmem() and guess_file_type_lowmem().
> 
> find_file_type_lowmem() gets filetypes from inode_item, dir_item and
> dir_index. If two of three's filetype are same and valid, it thinks
> the value is correct.
> 
> guess_file_type_lowmem() searches file_extent and dir_item/index then
> returns with filetype.
> 
> Signed-off-by: Su Yue 

Looks good.

Reviewed-by: Qu Wenruo 

Thanks,
Qu
> ---
>  cmds-check.c | 193 
> +++
>  1 file changed, 193 insertions(+)
> 
> diff --git a/cmds-check.c b/cmds-check.c
> index e3505a7f9d6b..b200fdccf0e5 100644
> --- a/cmds-check.c
> +++ b/cmds-check.c
> @@ -3306,6 +3306,199 @@ static int find_file_type(struct inode_record *rec, 
> u8 *type)
>   return -ENOENT;
>  }
>  
> +/*
> + * Fetch filetype from exited completed dir_item, dir_index and inode_item.
> + * If two of tree items'filetype are same, we think the type is trusted.
> + *
> + * Return 0 if file type is found and BTRFS_FT_* is stored into type.
> + * Return <0 if file type is not found.
> + */
> +static int find_file_type_lowmem(struct btrfs_root *root, u64 ino, u8 *type)
> +{
> + struct btrfs_key key;
> + struct btrfs_path path;
> + struct btrfs_path path2;
> + struct btrfs_inode_ref *iref;
> + struct btrfs_dir_item *dir_item;
> + struct btrfs_dir_item *dir_index;
> + struct extent_buffer *eb;
> + u64 dir;
> + u64 index;
> + char namebuf[BTRFS_NAME_LEN] = {0};
> + u32 namelen;
> + u8 inode_filetype = BTRFS_FT_UNKNOWN;
> + u8 dir_item_filetype;
> + u8 dir_index_filetype;
> + u8 true_file_type;
> + int slot;
> + int ret;
> +
> + key.objectid = ino;
> + key.type = BTRFS_INODE_ITEM_KEY;
> + key.offset = 0;
> +
> + btrfs_init_path();
> + ret = btrfs_search_slot(NULL, root, , , 0, 0);
> + if (ret < 0)
> + goto out;
> + if (!ret) {
> + struct btrfs_inode_item *ii;
> +
> + ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
> + struct btrfs_inode_item);
> + inode_filetype = imode_to_type(btrfs_inode_mode(path.nodes[0],
> + ii));
> + }
> +
> + key.objectid = ino;
> + key.type = BTRFS_INODE_REF_KEY;
> + key.offset = (u64)-1;
> +
> + btrfs_release_path();
> + ret = btrfs_search_slot(NULL, root, , , 0, 0);
> + if (ret < 0)
> + goto out;
> + if (!ret) {
> + ret = -EIO;
> + goto out;
> + }
> +
> + btrfs_init_path();
> +next:
> + btrfs_release_path();
> + ret = btrfs_previous_item(root, , ino, BTRFS_INODE_REF_KEY);
> + if (ret) {
> + ret = -ENOENT;
> + goto out;
> + }
> +
> + eb = path.nodes[0];
> + slot = path.slots[0];
> + btrfs_item_key_to_cpu(eb, , slot);
> + dir = key.offset;
> + iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
> + index = btrfs_inode_ref_index(eb, iref);
> + namelen = btrfs_inode_ref_name_len(eb, iref);
> + read_extent_buffer(eb, namebuf, (unsigned long)(iref + 1), namelen);
> +
> + dir_index = btrfs_lookup_dir_index(NULL, root, , dir, namebuf,
> +namelen, index, 0);
> + if (!dir_index)
> + goto next;
> + dir_index_filetype = btrfs_dir_type(path2.nodes[0], dir_index);
> + btrfs_release_path();
> + if (dir_index_filetype == inode_filetype) {
> + true_file_type = inode_filetype;
> + goto found;
> + }
> +
> + dir_item = btrfs_lookup_dir_item(NULL, root, , dir, namebuf,
> +  namelen, 0);
> + if (!dir_item)
> + goto next;
> + dir_item_filetype = btrfs_dir_type(path2.nodes[0], dir_item);
> + btrfs_release_path();
> + if (dir_item_filetype == inode_filetype) {
> + true_file_type = inode_filetype;
> + goto found;
> + }
> +
> + if (dir_index_filetype == dir_item_filetype) {
> + true_file_type = dir_index_filetype;
> + goto found;
> + }
> + goto next;
> +found:
> + /* rare case, two of three items are both corrupted */
> + if (true_file_type == BTRFS_FT_UNKNOWN ||
> + true_file_type >= BTRFS_FT_MAX)
> + goto next;
> + *type = true_file_type;
> + ret = 0;
> +out:
> + btrfs_release_path();
> + return ret;
> +}
> +
> +static int find_normal_file_extent(struct btrfs_root *root, u64 ino);
> +/*
> + * Try to determine inode type if type not found.
> + *
> + * For found regular file extent, it must be FILE.
> + * For found dir_item/index, it must be DIR.
> + *
> + * Return 0 if file type is confirmed and BTRFS_FT_* is stored into type.
> + * Return <0 if file type is unknown.
> + */
> +static int 

[PATCH 06/15] btrfs-progs: lowmem check: let check_dir_item() return if repaired

[Su Yue]

If repair_dir_item deleted the item, goto last checked then returns
instead of searching di_key again then returns -ENOENT.

Signed-off-by: Su Yue 
---
 cmds-check.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index eb65a18fe64b..4ce6139b3ab1 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -5931,7 +5931,7 @@ begin:
path->slots[0]--;
}
if (ret)
-   goto out;
+   return err;
}
 
node = path->nodes[0];
@@ -6040,7 +6040,7 @@ next:
break;
}
}
-out:
+
/* research path */
btrfs_release_path(path);
ret = btrfs_search_slot(NULL, root, di_key, path, 0, 0);
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 07/15] btrfs-progs: lowmem check: find_dir_item by di_key in check_dir_item()

[Su Yue]

In check_dir_item, we are going to search corresponding
dir_item/index.
@key shouldn't be used here. It should be @di_key.

Signed-off-by: Su Yue 
---
 cmds-check.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cmds-check.c b/cmds-check.c
index 4ce6139b3ab1..caac71a67472 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -6001,7 +6001,7 @@ begin:
 
/* check relative INDEX/ITEM */
key.objectid = di_key->objectid;
-   if (key.type == BTRFS_DIR_ITEM_KEY) {
+   if (di_key->type == BTRFS_DIR_ITEM_KEY) {
key.type = BTRFS_DIR_INDEX_KEY;
key.offset = index;
} else {
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 04/15] btrfs-progs: lowmem check: repair complex cases in repair_dir_item()

[Su Yue]

If inode item is missing or filetype is corrupted maybe, and filetypes
of dir_item and dir_index are corrupted too, lowmem repair may insert
wrong inode item and dir_item/index.

First, find and guess filetype of inode item, if failed, use
BTRFS_REG_FILE as fallback for insertion.
If filetype is not available, just delete current dir_item/index.

And repair_dir_item also calls repair_inode_item_mismatch() now.

Signed-off-by: Su Yue 
---
 cmds-check.c | 95 +---
 1 file changed, 78 insertions(+), 17 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 08a2662e603c..e33dd7db0048 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -2811,7 +2811,7 @@ static int walk_down_tree_v2(struct btrfs_trans_handle 
*trans,
 
ret = check_child_node(cur, path->slots[*level], next);
err |= ret;
-   if (ret < 0) 
+   if (ret < 0)
break;
 
if (btrfs_is_leaf(next))
@@ -5697,29 +5697,91 @@ static void print_dir_item_err(struct btrfs_root *root, 
struct btrfs_key *key,
 /*
  * Call repair_inode_item_missing and repair_ternary_lowmem to repair
  *
+ * @filetype:  filetype of the dir_item/index
+ *
  * Returns error after repair
  */
-static int repair_dir_item(struct btrfs_root *root, u64 dirid, u64 ino,
-  u64 index, u8 filetype, char *namebuf, u32 name_len,
-  int err)
+static int repair_dir_item(struct btrfs_root *root, struct btrfs_key *key,
+  u64 ino, u64 index, u8 filetype, char *namebuf,
+  u32 name_len, int err)
 {
int ret;
+   u64 dirid = key->objectid;
+   u8 true_filetype;
 
-   if (err & INODE_ITEM_MISSING) {
-   ret = repair_inode_item_missing(root, ino, filetype);
-   if (!ret)
-   err &= ~(INODE_ITEM_MISMATCH | INODE_ITEM_MISSING);
+   if (err & (INODE_ITEM_MISMATCH | INODE_ITEM_MISSING)) {
+   ret = find_file_type_lowmem(root, ino, _filetype);
+   if (ret) {
+   ret = guess_file_type_lowmem(root, ino,
+_filetype);
+   if (ret) {
+   true_filetype = BTRFS_FT_REG_FILE;
+   error(
+   "can't get file type for inode %llu, using FILE as fallback",
+ ino);
+   }
+   }
}
 
-   if (err & ~(INODE_ITEM_MISMATCH | INODE_ITEM_MISSING)) {
-   ret = repair_ternary_lowmem(root, dirid, ino, index, namebuf,
-   name_len, filetype, err);
+   /*
+* Case: the dir_item has corresponding inode_ref but
+* mismatch/missed inode_item and mismatch/missed another
+* dir_item/index.
+* repair_ternary_lowmem prefer to change another dir_item/index with
+* wrong filetype. So delete the item here.
+*/
+   if (filetype != true_filetype &&
+   (err & (DIR_ITEM_MISMATCH | DIR_ITEM_MISSING) ||
+err & (DIR_INDEX_MISMATCH | DIR_INDEX_MISSING))) {
+   struct btrfs_trans_handle *trans;
+   struct btrfs_path *path;
+
+   path = btrfs_alloc_path();
+   if (!path)
+   goto out;
+   trans = btrfs_start_transaction(root, 0);
+   ret = btrfs_search_slot(trans, root, key, path, -1, 1);
+   if (ret) {
+   btrfs_commit_transaction(trans, root);
+   btrfs_release_path(path);
+   goto out;
+   }
+   ret = btrfs_del_item(trans, root, path);
if (!ret) {
-   err &= ~(DIR_INDEX_MISMATCH | DIR_INDEX_MISSING);
-   err &= ~(DIR_ITEM_MISMATCH | DIR_ITEM_MISSING);
-   err &= ~(INODE_REF_MISSING);
+   err = 0;
+   printf(
+   "Deleted dir_item[%llu %u %llu]root %llu name %s filetype %u\n",
+  key->objectid, key->type, key->offset,
+  root->objectid, namebuf, filetype);
}
+   btrfs_commit_transaction(trans, root);
+   btrfs_release_path(path);
+   /*
+* Leave remains to check_inode_item() and check_inode_ref().
+*/
+   goto out;
+   }
+
+   ret = repair_ternary_lowmem(root, dirid, ino, index, namebuf,
+   name_len, true_filetype, err);
+   if (!ret) {
+   err &= ~(DIR_INDEX_MISMATCH | DIR_INDEX_MISSING);
+   err &= ~(DIR_ITEM_MISMATCH | DIR_ITEM_MISSING);
+   err &= ~(INODE_REF_MISSING);
+   }
+
+   if 

[PATCH 02/15] btrfs-progs: lowmem check: find and guess inode filetype

[Su Yue]

Introduce find_file_type_lowmem() and guess_file_type_lowmem().

find_file_type_lowmem() gets filetypes from inode_item, dir_item and
dir_index. If two of three's filetype are same and valid, it thinks
the value is correct.

guess_file_type_lowmem() searches file_extent and dir_item/index then
returns with filetype.

Signed-off-by: Su Yue 
---
 cmds-check.c | 193 +++
 1 file changed, 193 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index e3505a7f9d6b..b200fdccf0e5 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3306,6 +3306,199 @@ static int find_file_type(struct inode_record *rec, u8 
*type)
return -ENOENT;
 }
 
+/*
+ * Fetch filetype from exited completed dir_item, dir_index and inode_item.
+ * If two of tree items'filetype are same, we think the type is trusted.
+ *
+ * Return 0 if file type is found and BTRFS_FT_* is stored into type.
+ * Return <0 if file type is not found.
+ */
+static int find_file_type_lowmem(struct btrfs_root *root, u64 ino, u8 *type)
+{
+   struct btrfs_key key;
+   struct btrfs_path path;
+   struct btrfs_path path2;
+   struct btrfs_inode_ref *iref;
+   struct btrfs_dir_item *dir_item;
+   struct btrfs_dir_item *dir_index;
+   struct extent_buffer *eb;
+   u64 dir;
+   u64 index;
+   char namebuf[BTRFS_NAME_LEN] = {0};
+   u32 namelen;
+   u8 inode_filetype = BTRFS_FT_UNKNOWN;
+   u8 dir_item_filetype;
+   u8 dir_index_filetype;
+   u8 true_file_type;
+   int slot;
+   int ret;
+
+   key.objectid = ino;
+   key.type = BTRFS_INODE_ITEM_KEY;
+   key.offset = 0;
+
+   btrfs_init_path();
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret < 0)
+   goto out;
+   if (!ret) {
+   struct btrfs_inode_item *ii;
+
+   ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
+   struct btrfs_inode_item);
+   inode_filetype = imode_to_type(btrfs_inode_mode(path.nodes[0],
+   ii));
+   }
+
+   key.objectid = ino;
+   key.type = BTRFS_INODE_REF_KEY;
+   key.offset = (u64)-1;
+
+   btrfs_release_path();
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret < 0)
+   goto out;
+   if (!ret) {
+   ret = -EIO;
+   goto out;
+   }
+
+   btrfs_init_path();
+next:
+   btrfs_release_path();
+   ret = btrfs_previous_item(root, , ino, BTRFS_INODE_REF_KEY);
+   if (ret) {
+   ret = -ENOENT;
+   goto out;
+   }
+
+   eb = path.nodes[0];
+   slot = path.slots[0];
+   btrfs_item_key_to_cpu(eb, , slot);
+   dir = key.offset;
+   iref = btrfs_item_ptr(eb, slot, struct btrfs_inode_ref);
+   index = btrfs_inode_ref_index(eb, iref);
+   namelen = btrfs_inode_ref_name_len(eb, iref);
+   read_extent_buffer(eb, namebuf, (unsigned long)(iref + 1), namelen);
+
+   dir_index = btrfs_lookup_dir_index(NULL, root, , dir, namebuf,
+  namelen, index, 0);
+   if (!dir_index)
+   goto next;
+   dir_index_filetype = btrfs_dir_type(path2.nodes[0], dir_index);
+   btrfs_release_path();
+   if (dir_index_filetype == inode_filetype) {
+   true_file_type = inode_filetype;
+   goto found;
+   }
+
+   dir_item = btrfs_lookup_dir_item(NULL, root, , dir, namebuf,
+namelen, 0);
+   if (!dir_item)
+   goto next;
+   dir_item_filetype = btrfs_dir_type(path2.nodes[0], dir_item);
+   btrfs_release_path();
+   if (dir_item_filetype == inode_filetype) {
+   true_file_type = inode_filetype;
+   goto found;
+   }
+
+   if (dir_index_filetype == dir_item_filetype) {
+   true_file_type = dir_index_filetype;
+   goto found;
+   }
+   goto next;
+found:
+   /* rare case, two of three items are both corrupted */
+   if (true_file_type == BTRFS_FT_UNKNOWN ||
+   true_file_type >= BTRFS_FT_MAX)
+   goto next;
+   *type = true_file_type;
+   ret = 0;
+out:
+   btrfs_release_path();
+   return ret;
+}
+
+static int find_normal_file_extent(struct btrfs_root *root, u64 ino);
+/*
+ * Try to determine inode type if type not found.
+ *
+ * For found regular file extent, it must be FILE.
+ * For found dir_item/index, it must be DIR.
+ *
+ * Return 0 if file type is confirmed and BTRFS_FT_* is stored into type.
+ * Return <0 if file type is unknown.
+ */
+static int guess_file_type_lowmem(struct btrfs_root *root, u64 ino, u8 *type)
+{
+   struct btrfs_key key;
+   struct btrfs_path *path = NULL;
+   bool is_dir = false;
+   bool is_file = false;
+   int ret;
+
+   if 

[PATCH 15/15] btrfs-progs: fsck-tests: add image for original and lowmem check

[Su Yue]

This image have two cases mixed:
1) Both filetypes of dir_item and dir_index about inode 258
   are corrupted.
2) inode item 258 is missing.

Signed-off-by: Su Yue 
---
 .../029-mismatched-filetype-no-inode/default_case.img| Bin 0 -> 3072 bytes
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 
tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img

diff --git a/tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img 
b/tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img
new file mode 100644
index 
..57fc55b13c18bb9e5098a2fc4cb78f44fcf4abd9
GIT binary patch
literal 3072
zcmeHIdr;F?7EXCTRG?Bxc@>Nzl**%3o;ExN5K61)k|sa|c?E0?0tS<45}pcFF|3HG
zP(T(C5ekF^)P$A90ufn+LPC@zJVi+;1Oh=If!~6@ywljzjJ23d(ZjK
zojF%g#i^6~k^cn!$27LiPu{I9Fuxu2_TKVAAg%4)oBa+Gw*w%~J8ZGd_-G#$_^81D
zqXPJmONab%?&_8cU%hU@e3P9%Ckpzm)Ff$v)Wc@$>`BOZ#`Pp<)M#>*a!#6gR-n6^
z)1=c4bzkSqAeVbqNE#^6(OCUuf%ng!<@)45b#aP846yZF4X-R_(Y-V^`b{r7%mwmG
zSWwtole>bWF5})+N6j;AXb0Gxy74YnHE}t5`;DOxH;HrQz7u)mWsTfIuMAkZ<(_L<
zC!F*GIJRKTD$r+#ynp(^1;1avE`FkKNIg+L5PnOuanrCwR`tt$)P*2%T!%ObT%-uhb?}*?n%}JRS;pn;z#C$w4a`Mn{yX|Ga9U}x~oOl
zPdy^u@GIhKy*>SRG86A0MnN+-sb03Tf_vZ5-E+liKXMd_+MktgGVy
zvel#Bjd}T{tkT9P=zgwDEa(bJDo>c|p2U#nuEy7Oq}9Yu^C{YMx%Hc`+EvW8NX3^E
zk5b)CX?>IaMfBjmD-m+Yq}J{ZNNVX0?AQ<#|9zRx-J>V|%LlTp#^N=X)8>mh-)=f9
zEYlDVvLL+$VG(Ui8HDAoY(D;gCVKkJ-@HePR*lull}K3i@%ZJsPNw}D=_Eh9%JD@h
z?Erk8B2lV5KN*FmZmG~q(|r;G^8W)cQe$Qm!0sCQG}HAsPz|4CwFrA_^+e2sR_0gZ
zT6?nGG7Cq$kNKf{KyPloT`ZqIZM2Cu7Dyak38$c#V`;-zUb>Sns-7=GO_RFo*VdgR

[PATCH 10/15] btrfs-progs: check: clear I_ERR_FILE_EXTENT_DISCOUNT after repair

[Su Yue]

Original check forgets to clear bit I_ERR_FILE_EXTENT_DISCOUNT
in rec->errors after repair.

So just do it.

Signed-off-by: Su Yue 
---
 cmds-check.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/cmds-check.c b/cmds-check.c
index ae0a9e146399..d8d9a3227c06 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3978,6 +3978,7 @@ static int repair_inode_discount_extent(struct 
btrfs_trans_handle *trans,
if (ret < 0)
goto out;
}
+   rec->errors &= ~I_ERR_FILE_EXTENT_DISCOUNT;
printf("Fixed discount file extents for inode: %llu in root: %llu\n",
   rec->ino, root->objectid);
 out:
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 08/15] btrfs-progs: lowmem check: call get_dir_isize() after repair

[Su Yue]

Since lowmem repair may change size of inode item, introduce
get_dir_isize() to fetch isize after traversing items of inode.

Signed-off-by: Su Yue 
---
 cmds-check.c | 30 ++
 1 file changed, 30 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index caac71a67472..e57eea4e61c9 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -5868,6 +5868,35 @@ out:
return ret;
 }
 
+static int get_dir_isize(struct btrfs_root *root, u64 ino, u64 *size_ret)
+{
+   struct btrfs_inode_item *ii;
+   struct btrfs_key key;
+   struct btrfs_path path;
+   int ret;
+
+   key.objectid = ino;
+   key.type = BTRFS_INODE_ITEM_KEY;
+   key.offset = 0;
+
+   btrfs_init_path();
+   ret = btrfs_search_slot(NULL, root, , , 0, 0);
+   if (ret > 0)
+   ret = -ENOENT;
+   if (ret)
+   goto out;
+
+   ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
+   struct btrfs_inode_item);
+   *size_ret = btrfs_inode_size(path.nodes[0], ii);
+   ret = 0;
+out:
+   if (ret)
+   error("failed to get isize of inode %llu root %llu",
+ ino, root->root_key.objectid);
+   return ret;
+}
+
 /*
  * Traverse the given DIR_ITEM/DIR_INDEX and check related INODE_ITEM and
  * call find_inode_ref() to check related INODE_REF/INODE_EXTREF.
@@ -6591,6 +6620,7 @@ out:
if (repair && (err & DIR_COUNT_AGAIN)) {
err &= ~DIR_COUNT_AGAIN;
count_dir_isize(root, inode_id, );
+   get_dir_isize(root, inode_id, );
}
 
if ((nlink != 1 || refs != 1) && repair) {
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 12/15] btrfs-progs: check: increase counter error in check_inode_recs()

[Su Yue]

Counter @error decides return values of check_inode_recs().

Previously, @error won't be incremented even repair_inode_recs() failed.
It causes 'btrfs check --repair' prints some error information but
returns 0.

So, if root dir is missing and repair is disabled, @error should be
incremented.
And after repair_inode_recs(), increase @error if any errors in inodes
and backrefs.

Signed-off-by: Su Yue 
---
 cmds-check.c | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index b23a4493b12b..a83f0a92f48b 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -4137,6 +4137,7 @@ static int check_inode_recs(struct btrfs_root *root,
return -EAGAIN;
}
 
+   error++;
fprintf(stderr, "root %llu root dir %llu not found\n",
(unsigned long long)root->root_key.objectid,
(unsigned long long)root_dirid);
@@ -4176,10 +4177,9 @@ static int check_inode_recs(struct btrfs_root *root,
free_inode_rec(rec);
continue;
}
-   ret = 0;
}
 
-   if (!(repair && ret == 0))
+   if (rec->errors)
error++;
print_inode_error(root, rec);
list_for_each_entry(backref, >backrefs, list) {
@@ -4189,6 +4189,8 @@ static int check_inode_recs(struct btrfs_root *root,
backref->errors |= REF_ERR_NO_DIR_INDEX;
if (!backref->found_inode_ref)
backref->errors |= REF_ERR_NO_INODE_REF;
+   if (backref->errors)
+   error++;
fprintf(stderr, "\tunresolved ref dir %llu index %llu"
" namelen %u name %s filetype %d errors %x",
(unsigned long long)backref->dir,
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 11/15] btrfs-progs: check: modify indoe_rec and backref after repair

[Su Yue]

repair_inode_backrefs() forgets to modify backrefs and inode records
after repair them.

Signed-off-by: Su Yue 
---
 cmds-check.c | 4 
 1 file changed, 4 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index d8d9a3227c06..b23a4493b12b 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3192,6 +3192,7 @@ static int repair_inode_backrefs(struct btrfs_root *root,
ret = create_inode_item(root, rec, 1);
if (ret)
break;
+   rec->found_inode_item = 1;
repaired++;
}
}
@@ -3268,6 +3269,8 @@ static int repair_inode_backrefs(struct btrfs_root *root,
backref->index);
BUG_ON(ret);
btrfs_commit_transaction(trans, root);
+   backref->found_dir_index = 1;
+   backref->found_dir_item = 1;
repaired++;
}
 
@@ -3279,6 +3282,7 @@ static int repair_inode_backrefs(struct btrfs_root *root,
ret = create_inode_item(root, rec, 0);
if (ret)
break;
+   rec->found_inode_item = 1;
repaired++;
}
 
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 09/15] btrfs-progs: lowmem check: change logic of leaf process if repair

[Su Yue]

In lowmem check without repair, process_one_leaf_v2() will process one
entire leaf and inner check_inode_item() leads path point next
leaf.
In the beginning, process_one_leaf_v2() will let path point fist inode
item or first position where inode id changed.

However, in lowmem repair, process_one_leaf_v2() will be interrupted
to process one leaf because repair will CoW the leaf. Then some items
unprocessed is skipped.
Since repair may also delete some items, we can't use tricks like
record last checked key.

So, only for lowmem repair:
1. check_inode_item is responsible for handle case missing inode item.
2. process_one_leaf_v2() do not modify path manually, and check_inode()
   promise that @path points last checked item.
   Only when something are fixed, process_one_leaf_v2() will continue
   to check in next round.

Signed-off-by: Su Yue 
---
 cmds-check.c | 68 
 1 file changed, 59 insertions(+), 9 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index e57eea4e61c9..ae0a9e146399 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -2007,6 +2007,8 @@ static int process_one_leaf_v2(struct btrfs_root *root, 
struct btrfs_path *path,
 
cur_bytenr = cur->start;
 
+   if (repair)
+   goto again;
/* skip to first inode item or the first inode number change */
nritems = btrfs_header_nritems(cur);
for (i = 0; i < nritems; i++) {
@@ -2033,9 +2035,12 @@ again:
goto out;
 
/* still have inode items in thie leaf */
-   if (cur->start == cur_bytenr)
+   if (cur->start == cur_bytenr) {
+   ret = btrfs_next_item(root, path);
+   if (ret > 0)
+   goto out;
goto again;
-
+   }
/*
 * we have switched to another leaf, above nodes may
 * have changed, here walk down the path, if a node
@@ -5721,6 +5726,8 @@ static int repair_dir_item(struct btrfs_root *root, 
struct btrfs_key *key,
  ino);
}
}
+   } else {
+   true_filetype = filetype;
}
 
/*
@@ -6489,6 +6496,45 @@ out:
return ret;
 }
 
+/*
+ * Try insert new inode item frist.
+ * If failed, jump to next inode item.
+ */
+static int handle_inode_item_missing(struct btrfs_root *root,
+struct btrfs_path *path)
+{
+   struct btrfs_key key;
+   int ret;
+
+   btrfs_item_key_to_cpu(path->nodes[0], , path->slots[0]);
+
+   ret = repair_inode_item_missing(root, key.objectid, 0);
+   if (!ret) {
+   btrfs_release_path(path);
+   ret = btrfs_search_slot(NULL, root, , path, 0, 0);
+   if (ret)
+   goto next_inode;
+   else
+   goto out;
+   }
+
+next_inode:
+   error("inode item[%llu] is missing, skip to check next inode",
+ key.objectid);
+   while (1) {
+   ret = btrfs_next_item(root, path);
+   if (ret > 0)
+   goto out;
+   btrfs_item_key_to_cpu(path->nodes[0], , path->slots[0]);
+   if (key.type == BTRFS_INODE_ITEM_KEY) {
+   ret = 0;
+   break;
+   }
+   }
+out:
+   return ret;
+}
+
 /*
  * Check INODE_ITEM and related ITEMs (the same inode number)
  * 1. check link count
@@ -6536,6 +6582,13 @@ static int check_inode_item(struct btrfs_root *root, 
struct btrfs_path *path,
err |= LAST_ITEM;
return err;
}
+   if (key.type != BTRFS_INODE_ITEM_KEY && repair) {
+   ret = handle_inode_item_missing(root, path);
+   if (ret > 0)
+   err |= LAST_ITEM;
+   if (ret)
+   return err;
+   }
 
ii = btrfs_item_ptr(node, slot, struct btrfs_inode_item);
isize = btrfs_inode_size(node, ii);
@@ -6561,7 +6614,6 @@ static int check_inode_item(struct btrfs_root *root, 
struct btrfs_path *path,
btrfs_item_key_to_cpu(node, , slot);
if (key.objectid != inode_id)
goto out;
-
switch (key.type) {
case BTRFS_INODE_REF_KEY:
ret = check_inode_ref(root, , path, namebuf,
@@ -6608,12 +6660,10 @@ static int check_inode_item(struct btrfs_root *root, 
struct btrfs_path *path,
}
 
 out:
-   if (err & LAST_ITEM) {
-   btrfs_release_path(path);
-   ret = btrfs_search_slot(NULL, root, _key, path, 0, 0);
-   if (ret)
-   return err;
-   }
+   btrfs_release_path(path);
+   ret = btrfs_search_slot(NULL, root, _key, path, 0, 0);
+   if (ret)
+   return err;
 
/* verify INODE_ITEM nlink/isize/nbytes */
if 

[PATCH 01/15] btrfs-progs: lowmem check: introduce repair_inode_item_mismatch()

[Su Yue]

New function repair_inode_item_mismatch() only changes inode item'
filetype.

Signed-off-by: Su Yue 
---
 cmds-check.c | 52 
 1 file changed, 52 insertions(+)

diff --git a/cmds-check.c b/cmds-check.c
index f302724dd840..e3505a7f9d6b 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -4913,6 +4913,58 @@ static void print_inode_ref_err(struct btrfs_root *root, 
struct btrfs_key *key,
  namebuf, filetype);
 }
 
+/*
+ * Change inode mode.
+ *
+ * Returns 0 means success.
+ * Returns <0 means error.
+ */
+static int repair_inode_item_mismatch(struct btrfs_root *root, u64 ino,
+ u8 filetype)
+{
+   struct btrfs_key key;
+   struct btrfs_trans_handle *trans;
+   struct btrfs_path path;
+   struct btrfs_inode_item *ii;
+   u32 mode;
+   int ret;
+
+   key.objectid = ino;
+   key.type = BTRFS_INODE_ITEM_KEY;
+   key.offset = 0;
+
+   btrfs_init_path();
+   trans = btrfs_start_transaction(root, 1);
+   if (IS_ERR(trans)) {
+   ret = -EIO;
+   goto out;
+   }
+
+   ret = btrfs_search_slot(trans, root, , , 0, 1);
+   if (ret > 0)
+   ret = -ENOENT;
+   if (ret)
+   goto fail;
+
+   ii = btrfs_item_ptr(path.nodes[0], path.slots[0],
+   struct btrfs_inode_item);
+   mode = btrfs_inode_mode(path.nodes[0], ii);
+   mode &= ~S_IFMT;
+   mode |= btrfs_type_to_imode(filetype);
+
+   btrfs_set_inode_mode(path.nodes[0], ii, mode);
+   btrfs_mark_buffer_dirty(path.nodes[0]);
+   ret = 0;
+fail:
+   btrfs_commit_transaction(trans, root);
+out:
+   btrfs_release_path();
+   if (ret)
+   error("failed to repair root %llu INODE ITEM[%llu] MISMATCH",
+ root->objectid, ino);
+   return ret;
+}
+
 /*
  * Insert the missing inode item.
  *
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 14/15] btrfs-progs: check: handle mismatched filetype in repair_inode_backref

[Su Yue]

Original can't handle mismatched well.
Since it only records inode mode and one filetype of dir_item or
dir_index. If backref has mismatched filetype, the one filetype
is untrusted. The only trusted thing is inode mode.

So, if we found mismatched filetype in a backref, just delete
dir_index and dir_item. Then next round of repair will add new
dir_index and dir_item whose filetype bases on the inode mode.

Transform delete_dir_index() to __delete_dir_item() for code reuse.

Signed-off-by: Su Yue 
---
 cmds-check.c | 66 +++-
 1 file changed, 57 insertions(+), 9 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index 6091c7ef3442..156b4063bf40 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3059,27 +3059,41 @@ static int add_missing_dir_index(struct btrfs_root 
*root,
return 0;
 }
 
-static int delete_dir_index(struct btrfs_root *root,
-   struct inode_backref *backref)
+static int __delete_dir_item(struct btrfs_root *root,
+struct inode_backref *backref, bool is_index)
 {
struct btrfs_trans_handle *trans;
struct btrfs_dir_item *di;
struct btrfs_path path;
int ret = 0;
+   u64 offset;
+   u8 type;
+
+   if (is_index) {
+   type = BTRFS_DIR_INDEX_KEY;
+   offset = backref->index;
+   } else {
+   type = BTRFS_DIR_ITEM_KEY;
+   offset = btrfs_name_hash(backref->name, backref->namelen);
+   }
 
trans = btrfs_start_transaction(root, 1);
if (IS_ERR(trans))
return PTR_ERR(trans);
 
-   fprintf(stderr, "Deleting bad dir index [%llu,%u,%llu] root %llu\n",
-   (unsigned long long)backref->dir,
-   BTRFS_DIR_INDEX_KEY, (unsigned long long)backref->index,
-   (unsigned long long)root->objectid);
+   fprintf(stderr, "Deleting bad dir %s [%llu,%u,%llu] root %llu\n",
+   is_index ? "index" : "item", (unsigned long long)backref->dir,
+   type, offset, (unsigned long long)root->objectid);
 
btrfs_init_path();
-   di = btrfs_lookup_dir_index(trans, root, , backref->dir,
-   backref->name, backref->namelen,
-   backref->index, -1);
+   if (is_index)
+   di = btrfs_lookup_dir_index(trans, root, , backref->dir,
+   backref->name, backref->namelen,
+   backref->index, -1);
+   else
+   di = btrfs_lookup_dir_item(trans, root, , backref->dir,
+  backref->name, backref->namelen,
+  -1);
if (IS_ERR(di)) {
ret = PTR_ERR(di);
btrfs_release_path();
@@ -3099,6 +3113,18 @@ static int delete_dir_index(struct btrfs_root *root,
return ret;
 }
 
+static int delete_dir_index(struct btrfs_root *root,
+   struct inode_backref *backref)
+{
+   return __delete_dir_item(root, backref, true);
+}
+
+static int delete_dir_item(struct btrfs_root *root,
+  struct inode_backref *backref)
+{
+   return __delete_dir_item(root, backref, false);
+}
+
 static int __create_inode_item(struct btrfs_trans_handle *trans,
   struct btrfs_root *root, u64 ino, u64 size,
   u64 nbytes, u64 nlink, u32 mode)
@@ -3228,6 +3254,22 @@ static int repair_inode_backrefs(struct btrfs_root *root,
continue;
}
 
+   if (delete &&
+   (backref->errors & REF_ERR_FILETYPE_UNMATCH)) {
+   if (backref->found_dir_index)
+   ret = delete_dir_index(root, backref);
+   if (ret)
+   break;
+   if (backref->found_dir_item)
+   ret = delete_dir_item(root, backref);
+   if (ret)
+   break;
+   backref->found_dir_index = 0;
+   backref->found_dir_item = 0;
+   repaired++;
+   continue;
+   }
+
if (!delete && !backref->found_dir_index &&
backref->found_dir_item && backref->found_inode_ref) {
ret = add_missing_dir_index(root, inode_cache, rec,
@@ -3285,6 +3327,12 @@ static int repair_inode_backrefs(struct btrfs_root *root,
btrfs_commit_transaction(trans, root);
backref->found_dir_index = 1;
backref->found_dir_item = 1;
+   /*
+* We can't judge backre->filetype is right or not.
+* Just rely on 

[PATCH 05/15] btrfs-progs: lowmem check: let check_dir_item() continue if find wrong inode_item

[Su Yue]

check_dir_item can handle missed/mismatched inode item well.
Let it continue to check corresponding dir_item/index and inode_ref.

Signed-off-by: Su Yue 
---
 cmds-check.c | 15 ++-
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index e33dd7db0048..eb65a18fe64b 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -5984,15 +5984,12 @@ begin:
ret = btrfs_search_slot(NULL, root, , path, 0, 0);
if (ret) {
tmp_err |= INODE_ITEM_MISSING;
-   goto next;
-   }
-
-   ii = btrfs_item_ptr(path->nodes[0], path->slots[0],
-   struct btrfs_inode_item);
-   mode = btrfs_inode_mode(path->nodes[0], ii);
-   if (imode_to_type(mode) != filetype) {
-   tmp_err |= INODE_ITEM_MISMATCH;
-   goto next;
+   } else {
+   ii = btrfs_item_ptr(path->nodes[0], path->slots[0],
+   struct btrfs_inode_item);
+   mode = btrfs_inode_mode(path->nodes[0], ii);
+   if (imode_to_type(mode) != filetype)
+   tmp_err |= INODE_ITEM_MISMATCH;
}
 
/* Check relative INODE_REF/INODE_EXTREF */
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 00/15] btrfs-progs: fix filetype mismatch in check

[Su Yue]

This patchset is based on Qu Wenruo's
"[PATCH v2 0/3] Lowmem fsck repair to fix filetype mismatch".
It can be fetched from my github:
https://github.com/Damenly/btrfs-progs/tree/mismatch_filetype

Above Qu's patchset fixes mismatched filetype already. But both
original and lowmem can't handle more complex cases like image in the
last of this patchset:
Both filetypes of dir_item/index are corrupted and inode item is
missing/mismatched.

This patch contains fixes of lowmem check and original check.
For lowmem:
Patch[1-5] fix the complex cases by this way:
  Check filetypes of couple (dir_item,dir_index) and inode mode.
  If two of three have same filetype, choose it as right filetype.
  Handle it in repair_dir_item().
  
Patch[6-9] fix minor bugs of lowmem repair.

Patch[10-12] fix minor bugs about error bit and return value of
 original repair.
 
Patch[13-14] fix the complex cases by another way:
  Since original mode store one filetype from dir_item/dir_index and
  inode mode, if backref has mismatched filetype, we think inode mode
  is trusted. If inode item is missing, get filetype from a normal
  couple (dir_item and dir_index);
  
Patch[15] provides a test image.

Su Yue (15):
  btrfs-progs: lowmem check: introduce repair_inode_item_mismatch()
  btrfs-progs: lowmem check: find and guess inode filetype
  btrfs-progs: lowmem check: find filetype in repair_inode_missing()
  btrfs-progs: lowmem check: repair complex cases in repair_dir_item()
  btrfs-progs: lowmem check: let check_dir_item() continue if find wrong
inode_item
  btrfs-progs: lowmem check: let check_dir_item() return if repaired
  btrfs-progs: lowmem check: find_dir_item by di_key in check_dir_item()
  btrfs-progs: lowmem check: call get_dir_isize() after repair
  btrfs-progs: lowmem check: change logic of leaf process if repair
  btrfs-progs: check: clear I_ERR_FILE_EXTENT_DISCOUNT after repair
  btrfs-progs: check: modify indoe_rec and backref after repair
  btrfs-progs: check: increase counter error in check_inode_recs()
  btrfs-progs: check: find inode filetype in create_inode_item()
  btrfs-progs: check: handle mismatched filetype in repair_inode_backref
  btrfs-progs: fsck-tests: add image for original and lowmem check

 cmds-check.c   | 585 +++--
 .../default_case.img   | Bin 0 -> 3072 bytes
 2 files changed, 528 insertions(+), 57 deletions(-)
 create mode 100644 
tests/fsck-tests/029-mismatched-filetype-no-inode/default_case.img

-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH 13/15] btrfs-progs: check: find inode filetype in create_inode_item()

[Su Yue]

Now, find_file_type() doesn't return 0 if mismatched filetype is in a
backref.

Let create_inode_item() first call find_file_type() to get filetype.
If it failed, then guess filetype.

Signed-off-by: Su Yue 
---
 cmds-check.c | 29 ++---
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/cmds-check.c b/cmds-check.c
index a83f0a92f48b..6091c7ef3442 100644
--- a/cmds-check.c
+++ b/cmds-check.c
@@ -3140,6 +3140,8 @@ static int create_inode_item_lowmem(struct 
btrfs_trans_handle *trans,
return __create_inode_item(trans, root, ino, 0, 0, 0, mode);
 }
 
+static u32 btrfs_type_to_imode(u8 type);
+static int find_file_type(struct inode_record *rec, u8 *type);
 static int create_inode_item(struct btrfs_root *root,
 struct inode_record *rec, int root_dir)
 {
@@ -3148,14 +3150,19 @@ static int create_inode_item(struct btrfs_root *root,
u32 mode = 0;
u64 size = 0;
int ret;
+   u8 type = 0;
 
-   trans = btrfs_start_transaction(root, 1);
-   if (IS_ERR(trans)) {
-   ret = PTR_ERR(trans);
-   return ret;
+   nlink = root_dir ? 1 : rec->found_link;
+   ret = find_file_type(rec, );
+   if (!ret) {
+   mode = btrfs_type_to_imode(type) | 0755;
+   if (type == BTRFS_FT_REG_FILE)
+   size = rec->found_size;
+   else
+   size = rec->extent_end;
+   goto create_inode;
}
 
-   nlink = root_dir ? 1 : rec->found_link;
if (rec->found_dir_item) {
if (rec->found_file_extent)
fprintf(stderr, "root %llu inode %llu has both a dir "
@@ -3167,7 +3174,14 @@ static int create_inode_item(struct btrfs_root *root,
size = rec->found_size;
} else if (!rec->found_dir_item) {
size = rec->extent_end;
-   mode =  S_IFREG | 0755;
+   mode = S_IFREG | 0755;
+   }
+
+create_inode:
+   trans = btrfs_start_transaction(root, 1);
+   if (IS_ERR(trans)) {
+   ret = PTR_ERR(trans);
+   return ret;
}
 
ret = __create_inode_item(trans, root, rec->ino, size, rec->nbytes,
@@ -3307,7 +3321,8 @@ static int find_file_type(struct inode_record *rec, u8 
*type)
}
 
list_for_each_entry(backref, >backrefs, list) {
-   if (backref->found_dir_index || backref->found_dir_item) {
+   if ((backref->found_dir_index || backref->found_dir_item) &&
+   (!(backref->errors & REF_ERR_FILETYPE_UNMATCH))) {
*type = backref->filetype;
return 0;
}
-- 
2.16.1



--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/7] btrfs-progs: Do some clean up to be consistent

[Qu Wenruo]

On 2018年01月26日 15:25, Gu Jinxiang wrote:
> Patch 1~4 and 7: clean up use of btrfs_root.
> Patch 5: remove redundancy value assignment.
> Patch 6: remove no longer be used function define.

Overall it looks good, just one small nitpick, which should not need
another update.

> Changelog:
> v2->v1:
> Patch 2,4: To be consistent with kernel, change macro to inline function.
> Patch 3:
>   Change macro to inline function to be consistent with kernel.
>   And change the function body to match kernel.
> 
> Gu Jinxiang (7):
>   btrfs-progs: Use fs_info instead of root for BTRFS_LEAF_DATA_SIZE
>   btrfs-progs: Use fs_info instead of root for BTRFS_NODEPTRS_PER_BLOCK
>   btrfs-progs: Use fs_info instead of root for
> BTRFS_MAX_INLINE_DATA_SIZE

Not only this patch modified the parameter but also sync the code with
kernel.
It would be better to mention it in commit message, but anyway, just a
small nitpick, nothing to worry about.

Reviewed-by: Qu Wenruo 

Thanks,
Qu

>   btrfs-progs: Use fs_info instead of root for BTRFS_MAX_XATTR_SIZE
>   btrfs-progs: do clean up for redundancy value assignment
>   btrfs-progs: remove no longer be used btrfs_alloc_extent
>   btrfs-progs: Cleanup use of root in leaf_data_end
> 
>  cmds-check.c  | 10 ++---
>  convert/source-ext2.c |  4 +-
>  convert/source-reiserfs.c |  6 +--
>  ctree.c   | 96 
> +--
>  ctree.h   | 45 +-
>  dir-item.c|  3 +-
>  extent-tree.c |  1 -
>  file-item.c   |  2 +-
>  mkfs/main.c   |  4 +-
>  print-tree.c  |  2 +-
>  quick-test.c  |  2 +-
>  volumes.c |  2 +-
>  12 files changed, 97 insertions(+), 80 deletions(-)
> 



signature.asc
Description: OpenPGP digital signature