Re: ext4 metadata_csum and backwards compatibility

[TJ]

There are some Enterprise and Data-centre scenarios where this may cause
some pain.

1. iSCSI SAN where target and initiator are using incompatible versions
of e2fsprogs
2. LVM SAN (sanlock or DLM) where host and clients are using
incompatible versions

In both cases the problem arises where SAN host and client have
incompatible versions of e2fsprogs and at some point both need to do
file system management.

For example, a stable 16.04 LTS SAN host may be set to do regular fsck
operations on block devices exported over the network to prevent
saturating the network, as well as utilising the host's native I/O
advantages.

If the exported block devices have been formatted by a 16.04 client then
the operations on the 16.04 SAN host may fail.

In reverse, there's an issue where a 18.04 SAN host formats the block
devices but 16.04 clients do routine boot/mount-time fsck.

I'd favour an SRU to 16.04.

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ext4 metadata_csum and backwards compatibility

[David Britton]

Hi All --

@Robie: Thanks for highlighting this problem.

+1 from me on just documenting in the release notes.  It also seems a
reasonable enough *backport* if we want, but I don't think it's
necessary.

-- 
David Britton 

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ext4 metadata_csum and backwards compatibility

[Robie Basak]

On Mon, Mar 19, 2018 at 02:36:05PM +0100, Julian Andres Klode wrote:
> I think it might be useful to snap e2fsprogs, so we have a solution
> for all older releases. Not to replace the package, but as a backport
> that does not conflict with the system binaries.

A snap wouldn't be a bad thing, but I think there would be more to it
than "just" publishing a snap in order to maintain parity against an
SRU:

0. A user hitting the issue probably wouldn't have the snap installed,
so would see e2fsck fail instead of it Just Working via an SRU.

1. While the snap would make a workaround available, I'm concerned that
there would be no discoverability. How would a user, upon hitting a
problem with e2fsck on Xenial fscking a Bionic system, find out about
the snap solution?

2. What about maintenance? Right now, the Ubuntu security team is
committed to maintaining e2fsprogs in the "deb archive" for security in
Xenial, and Ubuntu as a whole for bugfixes, and an SRU wouldn't change
that. A prudent sysadmin would want to avoid losing this support. So
what kind of support commitment would come with that snap, and is anyone
proposing to commit to the additional maintenance of this extra thing?


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ext4 metadata_csum and backwards compatibility

[Julian Andres Klode]

On Mon, Mar 19, 2018 at 01:33:12PM +, Robie Basak wrote:
> On Wed, Mar 14, 2018 at 09:49:17PM -0700, Steve Langasek wrote:
> > It's not ideal for an interface to go from unsupported to mandatory in a
> > single LTS cycle; but I don't believe that the use case of creating a
> > filesystem with one LTS release, then interacting with it using the
> > userspace tools from a previous LTS release, is significant enough to
> > justify holding back features that upstream has recommended as the default.
> > 
> > I think it suffices to document this in the release notes.
> 
> Thanks. What's your opinion on an SRU to Xenial and/or to Trusty that
> allows e2fsprogs to understand the future filesystem feature? Assuming
> that no default behaviour would be changed for stable release users,
> would this be acceptable to you in principle?
> 
> To the rest of the SRU team: any objections to somebody driving this?
> I'm not necessarily committing to this wearing my Canonical Server Team
> hat, but "enyc" in #ubuntu-devel seems quite interested in driving an
> SRU, so it would be useful to get opinions now to avoid any wasted work.

I think it might be useful to snap e2fsprogs, so we have a solution
for all older releases. Not to replace the package, but as a backport
that does not conflict with the system binaries.

Or introduce e2fsprogs-hwe packages to match the hwe kernel which
support the new features.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ext4 metadata_csum and backwards compatibility

[Simon Deziel]

On 2018-03-14 07:17 AM, Robie Basak wrote:
> 3) We backport metadata_csum support to Xenial in an SRU[1] without
> changing the default there. Xenial users will be able to fsck
> Bionic-created ext4 filesystems. There will be forward compatibilty
> problems when skipping across multiple LTSs (eg. Trusty accessing a
> Bionic-created ext4 filesystem), but not across any single LTS.

I'd vote for this ^, SRU to Xenial. When a new LTS arrives, I typically
test it extensively in VMs running off of an hypervisor running the
previous LTS. Being able to fsck the VM's filesystem is sometimes
convenient. Also, since the metadata checksum was enabled in 16.10+ I'd
rather not go back in terms of default enabled features, especially if
this one is now production ready.

Regards,
Simon



signature.asc
Description: OpenPGP digital signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

Re: ext4 metadata_csum and backwards compatibility

[Steve Langasek]

On Wed, Mar 14, 2018 at 11:17:13AM +, Robie Basak wrote:
> I'd like to draw attention to:
> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1601997

> IRC discussion:
> https://irclogs.ubuntu.com/2018/03/13/%23ubuntu-devel.html#t12:05

> tl;dr: by doing nothing we're implicitly breaking some filesystem
> forward compatibility. I'd like us to make an explicit decision about
> whether we want this.

> I think this probably doesn't matter in the desktop use case. Wearing my
> server hat, I think it'll impact more server users, so I'm also CC-ing
> the ubuntu-server list.

> Theodore Ts'o enabled metadata_csum in mke2fs by default in Debian "so
> that in the testing and unstable branches, metadadata_csum checking
> would get some additional exposure, and hence testing" (from the bug).
> At that time he said he hadn't decided if he wanted it there in time for
> the following Debian release. AFAICT, it was enabled by stretch. AIUI,
> Xenial's e2fsprogs currently has no support for metadata_csum at all.

> A consequence of this (I'm told) is that Xenial's e2fsck doesn't work
> against filesystems created by Bionic's installer. IOW, this change
> breaks forwards compatibility between LTSs for Ubuntu.

> I'm not aware that this kind of compatibility is a promise that we've
> ever made, but IME it is a reasonably common thing for sysadmins to want
> to do. IMHO, breaking compatibility like this is sometimes necessary but
> is certainly inconvenient.

> I would prefer us to make an explicit decision on this, rather than have
> it happen to us implicitly as a consequence of how our ecosystem works.

> Here are some options:

> 1) [the default if we do nothing] Bionic will have metadata_csum enabled
> in new filesystems by default. Xenial's e2fsck will not be able to act
> on Bionic-default-created ext4 filesystems.



> I'm not keen on the default option 1 only because I have experiences of
> this kind of forward incompatibility being inconvenient for me in the
> past (when migrating systems in datacentres, etc). If the cost of
> maintaining at least some level of forward compatibility isn't too
> great, I would prefer that we keep it. If we do go with option 1 by
> doing nothing, I would at least like to release note this
> incompatibility.

It's not ideal for an interface to go from unsupported to mandatory in a
single LTS cycle; but I don't believe that the use case of creating a
filesystem with one LTS release, then interacting with it using the
userspace tools from a previous LTS release, is significant enough to
justify holding back features that upstream has recommended as the default.

I think it suffices to document this in the release notes.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel

ext4 metadata_csum and backwards compatibility

[Robie Basak]

I'd like to draw attention to:
https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1601997

IRC discussion:
https://irclogs.ubuntu.com/2018/03/13/%23ubuntu-devel.html#t12:05

tl;dr: by doing nothing we're implicitly breaking some filesystem
forward compatibility. I'd like us to make an explicit decision about
whether we want this.

I think this probably doesn't matter in the desktop use case. Wearing my
server hat, I think it'll impact more server users, so I'm also CC-ing
the ubuntu-server list.

Theodore Ts'o enabled metadata_csum in mke2fs by default in Debian "so
that in the testing and unstable branches, metadadata_csum checking
would get some additional exposure, and hence testing" (from the bug).
At that time he said he hadn't decided if he wanted it there in time for
the following Debian release. AFAICT, it was enabled by stretch. AIUI,
Xenial's e2fsprogs currently has no support for metadata_csum at all.

A consequence of this (I'm told) is that Xenial's e2fsck doesn't work
against filesystems created by Bionic's installer. IOW, this change
breaks forwards compatibility between LTSs for Ubuntu.

I'm not aware that this kind of compatibility is a promise that we've
ever made, but IME it is a reasonably common thing for sysadmins to want
to do. IMHO, breaking compatibility like this is sometimes necessary but
is certainly inconvenient.

I would prefer us to make an explicit decision on this, rather than have
it happen to us implicitly as a consequence of how our ecosystem works.

Here are some options:

1) [the default if we do nothing] Bionic will have metadata_csum enabled
in new filesystems by default. Xenial's e2fsck will not be able to act
on Bionic-default-created ext4 filesystems.

2) We disable metadata_csum in e2fsprogs in Bionic by default. The
feature will only be available on an opt-in basis, which in practice
will mean that only a minority of users will be able to benefit from the
feature itself. There will be no compatibility problems.

2b) 2, but we also enable metadata_csum again in Bionic+1. There will be
subsequent forward compatibilty problems when skipping across multiple
LTSs, but not across any single LTS.

3) We backport metadata_csum support to Xenial in an SRU[1] without
changing the default there. Xenial users will be able to fsck
Bionic-created ext4 filesystems. There will be forward compatibilty
problems when skipping across multiple LTSs (eg. Trusty accessing a
Bionic-created ext4 filesystem), but not across any single LTS.

3b) 3, but we also SRU to Trusty. There will be no compatibility
problems across still-supported releases.

4) [I don't like this but state it for completeness] Leave e2fsprogs as
it is in Bionic, but adjust the installer to create filesystems without
metadata_csum. There will be no compatibility problems except in the
case that users run mkfs.ext4 directly. I don't like this because I
think it will make compatibility behaviour too surprising for users.


I'm not keen on the default option 1 only because I have experiences of
this kind of forward incompatibility being inconvenient for me in the
past (when migrating systems in datacentres, etc). If the cost of
maintaining at least some level of forward compatibility isn't too
great, I would prefer that we keep it. If we do go with option 1 by
doing nothing, I would at least like to release note this
incompatibility.

What do you think?

Robie


[1] I'm not sure that this fits existing SRU policy exactly. It's a
little subjective. I assume that if consensus is to do this, then the
required approvals will be forthcoming.


signature.asc
Description: PGP signature
-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel