Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Wed, Nov 23, 2016 at 11:00:49AM +0800, Anand Jain wrote:
> > Preparatory bits are in patches:
> >
> >   btrfs-progs: mkfs: add names of matching sysfs feature names
> >   btrfs-progs: mkfs: enhance feature table
> >   btrfs-progs: mkfs: extend mkfs features with compat, safe and default 
> > versions
> 
>   I don't seen these patches in the ML. Looks like it was integrated
>   in 4.8.3.

Yeah the patches and many other from me haven't been sent to the
mailinglist, unless I need input regarding usecase. Sending everything
would increase work for me and most of the patches are boring cleanups
anyway.

> > the core work is not yet done.
> >
> > The original usecase has been extended to allow for selecting features
> > that are compatible, ie. can be mounted, safe, ie. are ok wrt The
> > Status page, and default, ie. automatically selected by mkfs.
> 
> > Additionally to
> > that, a system-wide preset from a config file in /etc.
> 
> 
>   Certainly these patches could have been on (cleanup/enhancement)
>   top of my patches which was sent a year back.

Well, back then I started on top of your patches and found out that I'd
have to rework too much code so it was easier to start from scratch.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

On 11/22/16 21:16, David Sterba wrote:

On Tue, Nov 22, 2016 at 04:54:58PM +0800, Anand Jain wrote:

On 11/14/16 20:13, David Sterba wrote:

On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:

  This patch isn't integrated, any idea why ?


Because it does not cover all usecases.


You have stated one of the use case here
 https://www.spinics.net/lists/linux-btrfs/msg49391.html

which I have it implemented in these patches
  [PATCH 5/7] btrfs-progs: introduce framework version to features
  [PATCH 6/7] btrfs-progs: add -O comp= option for mkfs.btrfs
  [PATCH 7/7] btrfs-progs: add -O comp= option for btrfs-convert

was their more ?


Yeah, it turned out there's more.


I've committed the preparatory
bits from branch where I've been working on that feature so we can
actually get to the hard parts.


  these patch set is a year old as well. are you planning to drop
  these patches ? the clarity is missing. IMO.


Preparatory bits are in patches:

  btrfs-progs: mkfs: add names of matching sysfs feature names
  btrfs-progs: mkfs: enhance feature table
  btrfs-progs: mkfs: extend mkfs features with compat, safe and default versions


 I don't seen these patches in the ML. Looks like it was integrated
 in 4.8.3.


the core work is not yet done.

The original usecase has been extended to allow for selecting features
that are compatible, ie. can be mounted, safe, ie. are ok wrt The
Status page, and default, ie. automatically selected by mkfs.



Additionally to
that, a system-wide preset from a config file in /etc.



 Certainly these patches could have been on (cleanup/enhancement)
 top of my patches which was sent a year back.



Thanks. Anand


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Tue, Nov 22, 2016 at 04:54:58PM +0800, Anand Jain wrote:
> On 11/14/16 20:13, David Sterba wrote:
> > On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:
> >>   This patch isn't integrated, any idea why ?
> >
> > Because it does not cover all usecases.
> 
> You have stated one of the use case here
>  https://www.spinics.net/lists/linux-btrfs/msg49391.html
> 
> which I have it implemented in these patches
>   [PATCH 5/7] btrfs-progs: introduce framework version to features
>   [PATCH 6/7] btrfs-progs: add -O comp= option for mkfs.btrfs
>   [PATCH 7/7] btrfs-progs: add -O comp= option for btrfs-convert
> 
> was their more ?

Yeah, it turned out there's more.

> > I've committed the preparatory
> > bits from branch where I've been working on that feature so we can
> > actually get to the hard parts.
> 
>   these patch set is a year old as well. are you planning to drop
>   these patches ? the clarity is missing. IMO.

Preparatory bits are in patches:

  btrfs-progs: mkfs: add names of matching sysfs feature names
  btrfs-progs: mkfs: enhance feature table
  btrfs-progs: mkfs: extend mkfs features with compat, safe and default versions

the core work is not yet done.

The original usecase has been extended to allow for selecting features
that are compatible, ie. can be mounted, safe, ie. are ok wrt The
Status page, and default, ie. automatically selected by mkfs. Additionally to
that, a system-wide preset from a config file in /etc.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

(sorry for the delay due to my vacation).

On 11/14/16 20:13, David Sterba wrote:

On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:

  This patch isn't integrated, any idea why ?


Because it does not cover all usecases.


 David,

   You have stated one of the use case here
https://www.spinics.net/lists/linux-btrfs/msg49391.html

   which I have it implemented in these patches
 [PATCH 5/7] btrfs-progs: introduce framework version to features
 [PATCH 6/7] btrfs-progs: add -O comp= option for mkfs.btrfs
 [PATCH 7/7] btrfs-progs: add -O comp= option for btrfs-convert

   was their more ?


I've committed the preparatory
bits from branch where I've been working on that feature so we can
actually get to the hard parts.


 these patch set is a year old as well. are you planning to drop
 these patches ? the clarity is missing. IMO.

Thanks, Anand
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Tue, Nov 08, 2016 at 09:14:39PM +0800, Anand Jain wrote:
>   This patch isn't integrated, any idea why ?

Because it does not cover all usecases. I've committed the preparatory
bits from branch where I've been working on that feature so we can
actually get to the hard parts.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Hi David,

 This patch isn't integrated, any idea why ? Just a note, if it matters,
 this set has already been integrated into Oracle Linux.

Thanks, Anand


On 11/23/15 20:56, Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.

Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.

With this I hope all the concerns from the review comments are
addressed.


Anand Jain (5):
  btrfs-progs: introduce framework to check kernel supported features
  btrfs-progs: add framework to check features supported by sysfs
  btrfs-progs: kernel based default features for mkfs
  btrfs-progs: kernel based default features for btrfs-convert
  btrfs-progs: add warning when we fail to read sysfs or kernel version

 btrfs-convert.c |  18 ++-
 mkfs.c  |  22 -
 utils.c | 146 +++-
 utils.h |   2 +
 4 files changed, 173 insertions(+), 15 deletions(-)


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
> Liu Bo wrote on 2015/12/03 17:44 -0800:
> > On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> >> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> >>> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> >>> be compatible w any set of older/newer kernels.
> >>>
> >>> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> >>> skinny-metadata even if the running kernel does not supports it, and
> >>> so the mount fails on the running.
> >>
> >> So the default behaviour of mkfs will try to best guess the feature set
> >> of currently running kernel. I think this is is the most common scenario
> >> and justifies the change in default behaviours.
> >>
> >> For the other cases I'd like to introduce some human-readable shortcuts
> >> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> >> options supported by the unpatched mainline kernel of version 3.2. This
> >> would be present for all version, regardless if there was a change in the
> >> options or not.
> >>
> >> Similarly for convenience, add 'running' that would pick the options
> >> from running kernel but will be explicit.
> >>
> >> A remaining option should override the 'running' behaviour and pick the
> >> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> >> name is yet to be determined.
> >>
> >>> Here in this set of patches will make sure the progs understands the
> >>> kernel supported features.
> >>>
> >>> So in this patch, checks if sysfs tells whether the feature is
> >>> supported if not, then it will relay on static kernel version which
> >>> provided that feature (skinny-metadata here in this example), next
> >>> if for some reason the running kernel does not provide the kernel
> >>> version, then it will fall back to the original method to enable
> >>> the feature with a hope that kernel will support it.
> >>>
> >>> Also the last patch adds a warning when we fail to read either
> >>> sysfs features or the running kernel version.
> >>
> >> Your patchset is a good start, the additional options I've described can
> >> be added on top of that. We might need to switch the version
> >> representation from string to KERNEL_VERSION but that's an
> >> implementation detail.
> >
> > Depending on sysfs is stable but depending on kernel version may be not,
> > we may have a distro kernel which backports some incompat features from
> > upstream, then we have to decide based on sysfs interface.
> 
> +1.
> 
> Although sysfs does not always show up even for supported kernel, e.g 
> btrfs modules is not loaded after boot.
> So we need to consider twice before choosing a fallback method.

There are several factors that we have to take into account for the
default behaviour and fallback. I'm close to a final proposal yet missed
the possibility of unloaded module that would remove the access to
sysfs, as you point out.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

David,


the possibility of unloaded module that would remove the access to
sysfs, as you point out.


 Kindly note, the patch below made /dev/btrfs-control a static node,

-
commit 578454ff7eab61d13a26b568f99a89a2c9edc881
Author: Kay Sievers 
Date: Thu May 20 18:07:20 2010 +0200

driver core: add devname module aliases to allow module on-demand 
auto-loading

--

 And here the function, check_or_load_btrfs_ko(), in the PATCH v2 2/5,
 will take care of this problem.


+
+int check_or_load_btrfs_ko()
+{
+   int fd;
+
+   /*
+* open will load btrfs kernel module if its not loaded,
+* and if the kernel has CONFIG auto load set?
+*/
+   fd = open("/dev/btrfs-control", O_RDONLY);
+   if (fd < 0)
+   return -errno;
+
+   close(fd);
+   return 0;
+}
+


 Since now static minor number for /dev/btrfs-control is mapped to
 the btrfs kernel module, it will ensure btrfs is loaded when
 /dev/btrfs-control is accessed.

 Further, /dev/btrfs-control node is created by udevd, by reading
 the modules.devname which is either supplied/updated by the distro
 or compilation.

 For systems without udev, IMO should run mknod ..btrfs-control
 in their install script which I guess is a must.


# ls -li /dev/btrfs-control
7338 crw-rw 1 root disk 10, 234 Dec  5 10:45 /dev/btrfs-control

# cat modules.devname | egrep btrfs
btrfs btrfs-control c10:234

# cat ./include/linux/miscdevice.h | egrep BTRFS
#define BTRFS_MINOR 234


 So IMO this is not a real problem.

Thanks, Anand

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Liu Bo]

On Fri, Dec 04, 2015 at 11:57:55AM +0800, Qu Wenruo wrote:
> 
> 
> Liu Bo wrote on 2015/12/03 18:53 -0800:
> >On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
> >>
> >>
> >>Liu Bo wrote on 2015/12/03 17:44 -0800:
> >>>On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> >Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> >be compatible w any set of older/newer kernels.
> >
> >So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> >skinny-metadata even if the running kernel does not supports it, and
> >so the mount fails on the running.
> 
> So the default behaviour of mkfs will try to best guess the feature set
> of currently running kernel. I think this is is the most common scenario
> and justifies the change in default behaviours.
> 
> For the other cases I'd like to introduce some human-readable shortcuts
> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> options supported by the unpatched mainline kernel of version 3.2. This
> would be present for all version, regardless if there was a change in the
> options or not.
> 
> Similarly for convenience, add 'running' that would pick the options
> from running kernel but will be explicit.
> 
> A remaining option should override the 'running' behaviour and pick the
> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> name is yet to be determined.
> 
> >Here in this set of patches will make sure the progs understands the
> >kernel supported features.
> >
> >So in this patch, checks if sysfs tells whether the feature is
> >supported if not, then it will relay on static kernel version which
> >provided that feature (skinny-metadata here in this example), next
> >if for some reason the running kernel does not provide the kernel
> >version, then it will fall back to the original method to enable
> >the feature with a hope that kernel will support it.
> >
> >Also the last patch adds a warning when we fail to read either
> >sysfs features or the running kernel version.
> 
> Your patchset is a good start, the additional options I've described can
> be added on top of that. We might need to switch the version
> representation from string to KERNEL_VERSION but that's an
> implementation detail.
> >>>
> >>>Depending on sysfs is stable but depending on kernel version may be not,
> >>>we may have a distro kernel which backports some incompat features from
> >>>upstream, then we have to decide based on sysfs interface.
> >>
> >>+1.
> >>
> >>Although sysfs does not always show up even for supported kernel, e.g btrfs
> >>modules is not loaded after boot.
> >>So we need to consider twice before choosing a fallback method.
> >>
> >>>
> >>>However, this brings another problems, for very old kernels, they don't
> >>>have sysfs, do you have any suggestions for that?
> >>
> >>Other fs, like xfs/ext* doesn't even have sysfs feature interface, only
> >>release announcement mentioning default behavior change.
> >>And I don't see many users complaining about it.
> >>
> >>Here is the example of xfsprogs changed its default feature recently:
> >>In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default feature
> >>of enabling CRC for fs.
> >>The first supported kernel is 3.15, which is release in 8th Jun, 2014.
> >>Almost one year ago.
> >
> >It's the same thing, if you use a earlier version(before v5) xfs and a
> >v5 xfsprogs, you are not going to mount it.
> >
> >>
> >>On the other hand, the sysfs feature is introduced at the end of year 2013.
> >>It's already over 2 years.
> >>
> >>So just forgot the extra minor case of super old kernel would be good
> >>enough.
> >
> >Sorry we're not able to do that since most users won't keep up upgrading 
> >their
> >kernels to the latest one, instead they use the stable one they think.
> >
> >The fact is that btrfs has way more incompatible features than either ext4 
> >or xfs,
> >and no complain on ext4/xfs from them won't solve our btrfs issue anyway.
> >
> >The problem is much more serious for enterprise users which are sort of
> >conservative, they would backport what they need, if they use
> >btrfs they will experience the painful things.
> 
> Only if enterprise really think btrfs is stable enough.
> For this point, xfs is considered more stable than btrfs, but v5 xfs recent
> change doesn't introduce such facility to do that compatibility check in
> xfsprogs.

Xfs on kernel side obviously refuses to mount if you create an incompatible
feature with a recent xfsprogs but try to mount it with older kernel.

STATIC int
xfs_mount_validate_sb()
{
...
if (xfs_sb_has_incompat_feature(sbp, XFS_SB_FEAT_INCOMPAT_UNKNOWN)) {   
  
xfs_warn(mp,  

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Qu Wenruo]

Liu Bo wrote on 2015/12/03 18:53 -0800:

On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:



Liu Bo wrote on 2015/12/03 17:44 -0800:

On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.


So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.

For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options

>from running kernel but will be explicit.


A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.


Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.


Your patchset is a good start, the additional options I've described can
be added on top of that. We might need to switch the version
representation from string to KERNEL_VERSION but that's an
implementation detail.


Depending on sysfs is stable but depending on kernel version may be not,
we may have a distro kernel which backports some incompat features from
upstream, then we have to decide based on sysfs interface.


+1.

Although sysfs does not always show up even for supported kernel, e.g btrfs
modules is not loaded after boot.
So we need to consider twice before choosing a fallback method.



However, this brings another problems, for very old kernels, they don't
have sysfs, do you have any suggestions for that?


Other fs, like xfs/ext* doesn't even have sysfs feature interface, only
release announcement mentioning default behavior change.
And I don't see many users complaining about it.

Here is the example of xfsprogs changed its default feature recently:
In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default feature
of enabling CRC for fs.
The first supported kernel is 3.15, which is release in 8th Jun, 2014.
Almost one year ago.


It's the same thing, if you use a earlier version(before v5) xfs and a
v5 xfsprogs, you are not going to mount it.



On the other hand, the sysfs feature is introduced at the end of year 2013.
It's already over 2 years.

So just forgot the extra minor case of super old kernel would be good
enough.


Sorry we're not able to do that since most users won't keep up upgrading their
kernels to the latest one, instead they use the stable one they think.

The fact is that btrfs has way more incompatible features than either ext4 or 
xfs,
and no complain on ext4/xfs from them won't solve our btrfs issue anyway.

The problem is much more serious for enterprise users which are sort of
conservative, they would backport what they need, if they use
btrfs they will experience the painful things.


Only if enterprise really think btrfs is stable enough.
For this point, xfs is considered more stable than btrfs, but v5 xfs 
recent change doesn't introduce such facility to do that compatibility 
check in xfsprogs.




There're plenty of fixes for progs code, people needs stabler recovery
tools rather than new features they may not use.

So we'd like to have a univeral progs code for old kernels.


Overall, btrfs is considered as a fast-moving and not that stable fs (at 
least not as stable as ext4/xfs).

And users are always encourages to use latest kernel for these reason.

Shouldn't we do such thing when btrfs is stable enough?

Thanks,
Qu



Thanks,

-liubo





--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Liu Bo]

On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> > Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> > be compatible w any set of older/newer kernels.
> > 
> > So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> > skinny-metadata even if the running kernel does not supports it, and
> > so the mount fails on the running.
> 
> So the default behaviour of mkfs will try to best guess the feature set
> of currently running kernel. I think this is is the most common scenario
> and justifies the change in default behaviours.
> 
> For the other cases I'd like to introduce some human-readable shortcuts
> to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> options supported by the unpatched mainline kernel of version 3.2. This
> would be present for all version, regardless if there was a change in the
> options or not.
> 
> Similarly for convenience, add 'running' that would pick the options
> from running kernel but will be explicit.
> 
> A remaining option should override the 'running' behaviour and pick the
> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> name is yet to be determined.
> 
> > Here in this set of patches will make sure the progs understands the
> > kernel supported features.
> > 
> > So in this patch, checks if sysfs tells whether the feature is
> > supported if not, then it will relay on static kernel version which
> > provided that feature (skinny-metadata here in this example), next
> > if for some reason the running kernel does not provide the kernel
> > version, then it will fall back to the original method to enable
> > the feature with a hope that kernel will support it.
> > 
> > Also the last patch adds a warning when we fail to read either
> > sysfs features or the running kernel version.
> 
> Your patchset is a good start, the additional options I've described can
> be added on top of that. We might need to switch the version
> representation from string to KERNEL_VERSION but that's an
> implementation detail.

Depending on sysfs is stable but depending on kernel version may be not,
we may have a distro kernel which backports some incompat features from
upstream, then we have to decide based on sysfs interface.

However, this brings another problems, for very old kernels, they don't
have sysfs, do you have any suggestions for that?

Thanks,

-liubo

> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Qu Wenruo]

Liu Bo wrote on 2015/12/03 17:44 -0800:

On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.


So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.

For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.

A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.


Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.


Your patchset is a good start, the additional options I've described can
be added on top of that. We might need to switch the version
representation from string to KERNEL_VERSION but that's an
implementation detail.


Depending on sysfs is stable but depending on kernel version may be not,
we may have a distro kernel which backports some incompat features from
upstream, then we have to decide based on sysfs interface.


+1.

Although sysfs does not always show up even for supported kernel, e.g 
btrfs modules is not loaded after boot.

So we need to consider twice before choosing a fallback method.



However, this brings another problems, for very old kernels, they don't
have sysfs, do you have any suggestions for that?


Other fs, like xfs/ext* doesn't even have sysfs feature interface, only 
release announcement mentioning default behavior change.

And I don't see many users complaining about it.

Here is the example of xfsprogs changed its default feature recently:
In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default 
feature of enabling CRC for fs.

The first supported kernel is 3.15, which is release in 8th Jun, 2014.
Almost one year ago.

On the other hand, the sysfs feature is introduced at the end of year 2013.
It's already over 2 years.

So just forgot the extra minor case of super old kernel would be good 
enough.


Thanks,
Qu



Thanks,

-liubo


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html





--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Liu Bo]

On Fri, Dec 04, 2015 at 10:08:35AM +0800, Qu Wenruo wrote:
> 
> 
> Liu Bo wrote on 2015/12/03 17:44 -0800:
> >On Mon, Nov 23, 2015 at 06:56:09PM +0100, David Sterba wrote:
> >>On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> >>>Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> >>>be compatible w any set of older/newer kernels.
> >>>
> >>>So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> >>>skinny-metadata even if the running kernel does not supports it, and
> >>>so the mount fails on the running.
> >>
> >>So the default behaviour of mkfs will try to best guess the feature set
> >>of currently running kernel. I think this is is the most common scenario
> >>and justifies the change in default behaviours.
> >>
> >>For the other cases I'd like to introduce some human-readable shortcuts
> >>to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
> >>options supported by the unpatched mainline kernel of version 3.2. This
> >>would be present for all version, regardless if there was a change in the
> >>options or not.
> >>
> >>Similarly for convenience, add 'running' that would pick the options
> >>from running kernel but will be explicit.
> >>
> >>A remaining option should override the 'running' behaviour and pick the
> >>latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
> >>name is yet to be determined.
> >>
> >>>Here in this set of patches will make sure the progs understands the
> >>>kernel supported features.
> >>>
> >>>So in this patch, checks if sysfs tells whether the feature is
> >>>supported if not, then it will relay on static kernel version which
> >>>provided that feature (skinny-metadata here in this example), next
> >>>if for some reason the running kernel does not provide the kernel
> >>>version, then it will fall back to the original method to enable
> >>>the feature with a hope that kernel will support it.
> >>>
> >>>Also the last patch adds a warning when we fail to read either
> >>>sysfs features or the running kernel version.
> >>
> >>Your patchset is a good start, the additional options I've described can
> >>be added on top of that. We might need to switch the version
> >>representation from string to KERNEL_VERSION but that's an
> >>implementation detail.
> >
> >Depending on sysfs is stable but depending on kernel version may be not,
> >we may have a distro kernel which backports some incompat features from
> >upstream, then we have to decide based on sysfs interface.
> 
> +1.
> 
> Although sysfs does not always show up even for supported kernel, e.g btrfs
> modules is not loaded after boot.
> So we need to consider twice before choosing a fallback method.
> 
> >
> >However, this brings another problems, for very old kernels, they don't
> >have sysfs, do you have any suggestions for that?
> 
> Other fs, like xfs/ext* doesn't even have sysfs feature interface, only
> release announcement mentioning default behavior change.
> And I don't see many users complaining about it.
> 
> Here is the example of xfsprogs changed its default feature recently:
> In 10th, June, 2015, xfsprogs v3.2.3 is released, with new default feature
> of enabling CRC for fs.
> The first supported kernel is 3.15, which is release in 8th Jun, 2014.
> Almost one year ago.

It's the same thing, if you use a earlier version(before v5) xfs and a
v5 xfsprogs, you are not going to mount it.

> 
> On the other hand, the sysfs feature is introduced at the end of year 2013.
> It's already over 2 years.
> 
> So just forgot the extra minor case of super old kernel would be good
> enough.

Sorry we're not able to do that since most users won't keep up upgrading their
kernels to the latest one, instead they use the stable one they think.

The fact is that btrfs has way more incompatible features than either ext4 or 
xfs,
and no complain on ext4/xfs from them won't solve our btrfs issue anyway.

The problem is much more serious for enterprise users which are sort of
conservative, they would backport what they need, if they use
btrfs they will experience the painful things.

There're plenty of fixes for progs code, people needs stabler recovery
tools rather than new features they may not use.

So we'd like to have a univeral progs code for old kernels.

Thanks,

-liubo
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Thanks for comments.


Distro would also want to use the latest btrfs-progs on older kernel
since it will have latest fsck/send/receive fixes, better UI
and updated man pages.

btrfs-progs which claim backward kernel compatible and it shouldn't
fail on the below cmd when the btrfs-progs is upgraded.
   mkfs.btrfs /dev/sda && mount /dev/sda /btrfs
but it does in some test cases.


A warning is unnecessary IMO. Imagine user who upgrade progs for
better doc/UI and have no intention to upgrade the kernel, gets a 
Warning!. If user does not upgrade kernel its a fair assumption

that they don't need/not-looking for latest kernel features. (What
did I miss ?).


Next.
For users looking to have a disk-layout which is compatible with
older kernels (and may not be a running kernel), then with the
current patch set its quite possible to do something like below,

mkfs.btrfs -O as-per-kernel=3.2
mkfs.btrfs -O as-per-kernel=4.0
mkfs.btrfs -O as-per-kernel=x.x (anything)

And only those features that are supported until version x.x
(mainline) will be enabled by default unless user want to over
default totally by using -O .

Thanks, Anand



Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.

Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.

With this I hope all the concerns from the review comments are
addressed.


Anand Jain (5):
   btrfs-progs: introduce framework to check kernel supported features
   btrfs-progs: add framework to check features supported by sysfs
   btrfs-progs: kernel based default features for mkfs
   btrfs-progs: kernel based default features for btrfs-convert
   btrfs-progs: add warning when we fail to read sysfs or kernel version

  btrfs-convert.c |  18 ++-
  mkfs.c  |  22 -
  utils.c | 146 +++-
  utils.h |   2 +
  4 files changed, 173 insertions(+), 15 deletions(-)


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

David Sterba wrote:

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.


So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.

For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2.


 This is a nice idea. I am planning. How about 'as-per-kernel=x.x'
 instead of compat-3.2.

 Also looks like it better to list the feature and version mapping
 as btrfs-progs already knows it at this patchset.



This
would be present for all version, regardless if there was a change in the
options or not.


  Hmm.. I didn't quite get that.


Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.

A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.


Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.


Your patchset is a good start, the additional options I've described can
be added on top of that.


 Yes.

Thanks, Anand




We might need to switch the version
representation from string to KERNEL_VERSION but that's an
implementation detail.





--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Austin S Hemmelgarn]

On 2015-11-23 12:56, David Sterba wrote:

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.


So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.
I feel that Christoph's suggestion in the other sub-thread to have it 
spit out a notice that it disabled something because of the kernel it's 
running on is worth adding also.  We should probably also spit out a 
warning if the user asks for a feature that isn't supported on the 
current kernel (but still let them create the filesystem regardless).


For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.
Is the intent to enable stuff that the devs consider stable that's 
supported by the running kernel, or all the features supported by the 
running kernel?  It's probably best to use the first as the defaults, 
and then have an option to pull in everything the running kernel 
supports (possibly name that option something like 'running-all').


A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.

Maybe something like 'recommended' or 'suggested'?

It might also be nice to have an option to tell it to turn on everything 
the tools support (possibly call that one something like 
'max-features'), though this is probably less useful due to the fact 
that most mkfs features in BTRFS are incompat features.




smime.p7s
Description: S/MIME Cryptographic Signature

[PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Anand Jain]

Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
be compatible w any set of older/newer kernels.

So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
skinny-metadata even if the running kernel does not supports it, and
so the mount fails on the running.

Here in this set of patches will make sure the progs understands the
kernel supported features.

So in this patch, checks if sysfs tells whether the feature is
supported if not, then it will relay on static kernel version which
provided that feature (skinny-metadata here in this example), next
if for some reason the running kernel does not provide the kernel
version, then it will fall back to the original method to enable
the feature with a hope that kernel will support it.

Also the last patch adds a warning when we fail to read either
sysfs features or the running kernel version.

With this I hope all the concerns from the review comments are
addressed.


Anand Jain (5):
  btrfs-progs: introduce framework to check kernel supported features
  btrfs-progs: add framework to check features supported by sysfs
  btrfs-progs: kernel based default features for mkfs
  btrfs-progs: kernel based default features for btrfs-convert
  btrfs-progs: add warning when we fail to read sysfs or kernel version

 btrfs-convert.c |  18 ++-
 mkfs.c  |  22 -
 utils.c | 146 +++-
 utils.h |   2 +
 4 files changed, 173 insertions(+), 15 deletions(-)

-- 
2.6.2

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[Duncan]

Austin S Hemmelgarn posted on Mon, 23 Nov 2015 15:14:44 -0500 as
excerpted:

>> A remaining option should override the 'running' behaviour and pick the
>> latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
>> name is yet to be determined.

> Maybe something like 'recommended' or 'suggested'?

Just keep it simple.  If it enables the latest defaults, just call it 
"latest".  =:^)

Better yet, match compat-3.2, etc, with compat-running and compat-latest, 
making it crystal clear they're parallel options, as well as labeling 
them all compat-*, making it crystal clear what the intended 
functionality is.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 0/5] Make btrfs-progs really compatible with any kernel version

[David Sterba]

On Mon, Nov 23, 2015 at 08:56:13PM +0800, Anand Jain wrote:
> Btrfs-progs is a tool for the btrfs kernel and we hope latest btrfs-progs
> be compatible w any set of older/newer kernels.
> 
> So far mkfs.btrfs and btrfs-convert sets the default features, for eg,
> skinny-metadata even if the running kernel does not supports it, and
> so the mount fails on the running.

So the default behaviour of mkfs will try to best guess the feature set
of currently running kernel. I think this is is the most common scenario
and justifies the change in default behaviours.

For the other cases I'd like to introduce some human-readable shortcuts
to the --features option. Eg. 'mkfs.btrfs -O compat-3.2' will pick all
options supported by the unpatched mainline kernel of version 3.2. This
would be present for all version, regardless if there was a change in the
options or not.

Similarly for convenience, add 'running' that would pick the options
from running kernel but will be explicit.

A remaining option should override the 'running' behaviour and pick the
latest mkfs options. Naming it 'defaults' sounds a bit ambiguous so the
name is yet to be determined.

> Here in this set of patches will make sure the progs understands the
> kernel supported features.
> 
> So in this patch, checks if sysfs tells whether the feature is
> supported if not, then it will relay on static kernel version which
> provided that feature (skinny-metadata here in this example), next
> if for some reason the running kernel does not provide the kernel
> version, then it will fall back to the original method to enable
> the feature with a hope that kernel will support it.
> 
> Also the last patch adds a warning when we fail to read either
> sysfs features or the running kernel version.

Your patchset is a good start, the additional options I've described can
be added on top of that. We might need to switch the version
representation from string to KERNEL_VERSION but that's an
implementation detail.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html