Processed: Re: Bug#924591: this requires linking in libsparse, which is from Android sources
Processing commands for cont...@bugs.debian.org: > clone 924591 -1 Bug #924591 [e2fsprogs] fastboot format:ext4 misses /usr/lib/android-sdk/platform-tools/mke2fs Bug 924591 cloned as bug 928551 > reassign 924591 fastboot 1:8.1.0+r23-4 Bug #924591 [e2fsprogs] fastboot format:ext4 misses /usr/lib/android-sdk/platform-tools/mke2fs Bug reassigned from package 'e2fsprogs' to 'fastboot'. No longer marked as found in versions e2fsprogs/1.44.5-1. Ignoring request to alter fixed versions of bug #924591 to the same values previously set Bug #924591 [fastboot] fastboot format:ext4 misses /usr/lib/android-sdk/platform-tools/mke2fs Marked as found in versions android-platform-system-core/1:8.1.0+r23-4. > retitle -1 e2fsprogs: add support for dynamically loading libsparse Bug #928551 [e2fsprogs] fastboot format:ext4 misses /usr/lib/android-sdk/platform-tools/mke2fs Changed Bug title to 'e2fsprogs: add support for dynamically loading libsparse' from 'fastboot format:ext4 misses /usr/lib/android-sdk/platform-tools/mke2fs'. > severity -1 wishlist Bug #928551 [e2fsprogs] e2fsprogs: add support for dynamically loading libsparse Severity set to 'wishlist' from 'serious' > thanks Stopping processing here. Please contact me if you need assistance. -- 924591: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924591 928551: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=928551 Debian Bug Tracking System Contact ow...@bugs.debian.org with problems
Bug#924591: this requires linking in libsparse, which is from Android sources
clone 924591 -1 reassign 924591 fastboot 1:8.1.0+r23-4 retitle -1 e2fsprogs: add support for dynamically loading libsparse severity -1 wishlist thanks I'm reassigning the original bug back to fastboot. I've cloned the bug and made it a feature request of having e2fsprogs dynamically load libsparse if it is available. That's not happening until after Buster is released, though. - Ted
Bug#924591: this requires linking in libsparse, which is from Android sources
On Mon, Apr 22, 2019 at 10:19:46PM +0200, Hans-Christoph Steiner wrote: > > I don't really know how fastboot in stretch provided the mke2fs support, > but judging by the dependencies, it might have been that fastboot used > to do the formatting itself, based on being linked to > android-libext4-utils and android-libsparse. The buster version of > fastboot is clearly calling mk2efs, which in AOSP is built from an > inline e2fsprogs fork. Yes, that's correct. >From running strings on the fastboot binary from Stretch, it's using the statically linked-in make_ext4fs code. The make_ext4fs was code written years and years ago, back when Android senior management (rumor has it was that it was Andy Rubin himself) didn't want to use any GPL'ed code in userspace. Fortunately, that's no longer the case. The old make_ext4fs code was old, creaky, and didn't exactly work the same way as mke2fs (since it was written as a clean-room reimplementation from scratch). As a result I was very happy when we were finally able to take the make_ext4fs code and KILL IT WITH FIRE[1]. :-) [1] https://www.youtube.com/watch?v=Tnod9vtB4xA Unfortunately, the focus was to make the make_ext4fs replacement work with AOSP only. I wasn't aware of Debian's native Android tools; but even if I did, it's not clear that we could have gotten things working within the scope of the intern project to drop make_ext4fs support and port the necessary support code into e2fsprogs. This change started landing in AOSP in November 2016 (it was a Fall 2016 intern project). I'd have to check to be sure, but looking at the Debian changelog, the AOSP release with the actual KILL IT WITH FIRE commit probably landed in Debian sometime in late 2017. Alas, apparently no one had noticed the problem for well over a year. So I'm guessing Debian's fastboot, or at least its format command, is rarely used by Debian users. :-/ - Ted
Bug#924591: this requires linking in libsparse, which is from Android sources
Theodore Ts'o: > On Mon, Apr 22, 2019 at 06:09:23PM +0200, Jonas Meurer wrote: >> Hans-Christoph Steiner: >>> Theodore Ts'o: So your choice --- we can either reassign this bug back to fastboot or android-sdk-platforms-tools, or I can downgrade the severity of this bug for e2fsprogs down to wishlist[1]. Let me know how you want to handle this. [1] This is because I view this both as a "feature request" and "bugs that are very difficult to fix due to major design considerations" (per https://www.debian.org/Bugs/Developer#severities), not to mention that it's going to affect a miniscule fraction of the e2fsprogs package's users. >>> >>> Makes sense to me. I'm fine with this being done post-Buster or as a >>> custom mke2fs in android-platform-system-core. >> >> So the bottom line here is that the ext4 formatting support in fastboot >> remains broken in Buster, right? That would be very unfortunate and a >> regression compared to Stretch. > > I'm not sure whether or not Stretch was using the old-style > make_ext4fs from AOSP, or was including the mke2fs from AOSP, but yes, > it sounds like it's a regression from stretch. I'm not sure how many > Debian users are using the Debian-native fastboot versus using the > version from the Google SDK or the AOSP binaries, though. > > It does seem that if this is considered high priority, the most > straightforward way to address this bug is going to be to include > building mke2fs from AOSP and placing it in > /usr/lib/android-sdk/platform-tools/mke2fs. I know some folks on the > android tools teams aren't excited with that approach, but that > probably is the best thing to do if you want to address this in > Buster. That approach sounds fine for buster. The only question in my mind is who will do the work... I don't really know how fastboot in stretch provided the mke2fs support, but judging by the dependencies, it might have been that fastboot used to do the formatting itself, based on being linked to android-libext4-utils and android-libsparse. The buster version of fastboot is clearly calling mk2efs, which in AOSP is built from an inline e2fsprogs fork. .hc
Bug#924591: this requires linking in libsparse, which is from Android sources
On Mon, Apr 22, 2019 at 06:09:23PM +0200, Jonas Meurer wrote: > Hans-Christoph Steiner: > > Theodore Ts'o: > >> So your choice --- we can either reassign this bug back to fastboot or > >> android-sdk-platforms-tools, or I can downgrade the severity of this > >> bug for e2fsprogs down to wishlist[1]. Let me know how you want to > >> handle this. > >> > >> [1] This is because I view this both as a "feature request" and "bugs > >> that are very difficult to fix due to major design considerations" > >> (per https://www.debian.org/Bugs/Developer#severities), not to mention > >> that it's going to affect a miniscule fraction of the e2fsprogs > >> package's users. > > > > Makes sense to me. I'm fine with this being done post-Buster or as a > > custom mke2fs in android-platform-system-core. > > So the bottom line here is that the ext4 formatting support in fastboot > remains broken in Buster, right? That would be very unfortunate and a > regression compared to Stretch. I'm not sure whether or not Stretch was using the old-style make_ext4fs from AOSP, or was including the mke2fs from AOSP, but yes, it sounds like it's a regression from stretch. I'm not sure how many Debian users are using the Debian-native fastboot versus using the version from the Google SDK or the AOSP binaries, though. It does seem that if this is considered high priority, the most straightforward way to address this bug is going to be to include building mke2fs from AOSP and placing it in /usr/lib/android-sdk/platform-tools/mke2fs. I know some folks on the android tools teams aren't excited with that approach, but that probably is the best thing to do if you want to address this in Buster. - Ted
Bug#924591: this requires linking in libsparse, which is from Android sources
Hans-Christoph Steiner: > Theodore Ts'o: >> So your choice --- we can either reassign this bug back to fastboot or >> android-sdk-platforms-tools, or I can downgrade the severity of this >> bug for e2fsprogs down to wishlist[1]. Let me know how you want to >> handle this. >> >> [1] This is because I view this both as a "feature request" and "bugs >> that are very difficult to fix due to major design considerations" >> (per https://www.debian.org/Bugs/Developer#severities), not to mention >> that it's going to affect a miniscule fraction of the e2fsprogs >> package's users. > > Makes sense to me. I'm fine with this being done post-Buster or as a > custom mke2fs in android-platform-system-core. So the bottom line here is that the ext4 formatting support in fastboot remains broken in Buster, right? That would be very unfortunate and a regression compared to Stretch. Cheers jonas signature.asc Description: OpenPGP digital signature
Bug#924591: this requires linking in libsparse, which is from Android sources
Theodore Ts'o: > On Thu, Apr 18, 2019 at 09:32:06PM +0200, Hans-Christoph Steiner wrote: >> >> One possibility would be including libsparse as a patch, it doesn't >> change a lot: >> https://android.googlesource.com/platform/system/core/+log/master/libsparse >> >> But it depends on Android's libbase and libz-host. > > This might be "serious" bug from the fastboot package's perspective, > but there's no way in heck the release time is going to consider this > a bug that is "serious" priority for e2fsprogs. > > More to the point, there's now way in the world I (or the release and > installer teams) are going to make e2fsprogs, which is an > "important=yes" package with priority "required" drag in the > android-libsparse, android-libbase, and zlib1g packages. > > So the way you changed android-sdk-platforms-tools to use /sbin/mke2fs > was a really bad choice, especially this while we are in release > freeze for Buster. There's no way in the world we are going to make a > change like this to a package like e2fsprogs which is used by the > installer at this point. > > If we had more time, and if android-libsparse-dev shipped a static > library, we could have considered statically linking in > android-libsparse, android-libbase, and libz --- and see if they would > bloat the mke2fs and debugfs binaries by only a minimal amount. > > This would also require making changes to e2fsprogs configure and > Makefiles, since currently we only have support for linking in > libsparse in the AOSP build files. The reason for this is historical; > at the time when the intern working with Android team was working on > replace Android's make_ext4fs program with mke2fs and e2droid, there > was no distribution that was shipping libsparse, and trying to make > libsparse available to Linux desktop environments was *way* beyond the > scope of the Intern's project and time availability. > > We can work on this trying to find a solution post-Buster --- either > using static linking, or *possibly* figuring out a way to optionally > use dlopen() to pull in libsparse for sparse_io.c, much like the way > libss optionally pulls in the readline library using dlopen at > runtime, back when we cared about making mke2fs fit on a two 1.44 MiB > boot/root install floppies. :-) > > Alternatively, you can build your own version of mke2fs using the > libsparse from AOSP. If you want a solution that might make it in > during the Buster release freeze, that's probably the short-term > solution I would suggest. > > So your choice --- we can either reassign this bug back to fastboot or > android-sdk-platforms-tools, or I can downgrade the severity of this > bug for e2fsprogs down to wishlist[1]. Let me know how you want to > handle this. > > Cheers, > > - Ted > > [1] This is because I view this both as a "feature request" and "bugs > that are very difficult to fix due to major design considerations" > (per https://www.debian.org/Bugs/Developer#severities), not to mention > that it's going to affect a miniscule fraction of the e2fsprogs > package's users. Makes sense to me. I'm fine with this being done post-Buster or as a custom mke2fs in android-platform-system-core. .hc
Bug#924591: this requires linking in libsparse, which is from Android sources
> So your choice --- we can either reassign this bug back to fastboot or > android-sdk-platforms-tools, or I can downgrade the severity of this > bug for e2fsprogs down to wishlist[1]. Let me know how you want to > handle this. I would say downgrade it for the moment. We can deal with it after Buster. Indeed this is quite tricky. I agree that `e2fsprogs` shouldn't depend on `libsparse`. I like the `dlopen` approach but it increases the maintenance burden a little bit. But anyway I am too mentally tired of yet another AOSP fork making into Debian's archive... signature.asc Description: OpenPGP digital signature
Bug#924591: this requires linking in libsparse, which is from Android sources
On Thu, Apr 18, 2019 at 09:32:06PM +0200, Hans-Christoph Steiner wrote: > > One possibility would be including libsparse as a patch, it doesn't > change a lot: > https://android.googlesource.com/platform/system/core/+log/master/libsparse > > But it depends on Android's libbase and libz-host. This might be "serious" bug from the fastboot package's perspective, but there's no way in heck the release time is going to consider this a bug that is "serious" priority for e2fsprogs. More to the point, there's now way in the world I (or the release and installer teams) are going to make e2fsprogs, which is an "important=yes" package with priority "required" drag in the android-libsparse, android-libbase, and zlib1g packages. So the way you changed android-sdk-platforms-tools to use /sbin/mke2fs was a really bad choice, especially this while we are in release freeze for Buster. There's no way in the world we are going to make a change like this to a package like e2fsprogs which is used by the installer at this point. If we had more time, and if android-libsparse-dev shipped a static library, we could have considered statically linking in android-libsparse, android-libbase, and libz --- and see if they would bloat the mke2fs and debugfs binaries by only a minimal amount. This would also require making changes to e2fsprogs configure and Makefiles, since currently we only have support for linking in libsparse in the AOSP build files. The reason for this is historical; at the time when the intern working with Android team was working on replace Android's make_ext4fs program with mke2fs and e2droid, there was no distribution that was shipping libsparse, and trying to make libsparse available to Linux desktop environments was *way* beyond the scope of the Intern's project and time availability. We can work on this trying to find a solution post-Buster --- either using static linking, or *possibly* figuring out a way to optionally use dlopen() to pull in libsparse for sparse_io.c, much like the way libss optionally pulls in the readline library using dlopen at runtime, back when we cared about making mke2fs fit on a two 1.44 MiB boot/root install floppies. :-) Alternatively, you can build your own version of mke2fs using the libsparse from AOSP. If you want a solution that might make it in during the Buster release freeze, that's probably the short-term solution I would suggest. So your choice --- we can either reassign this bug back to fastboot or android-sdk-platforms-tools, or I can downgrade the severity of this bug for e2fsprogs down to wishlist[1]. Let me know how you want to handle this. Cheers, - Ted [1] This is because I view this both as a "feature request" and "bugs that are very difficult to fix due to major design considerations" (per https://www.debian.org/Bugs/Developer#severities), not to mention that it's going to affect a miniscule fraction of the e2fsprogs package's users.
Bug#924591: this requires linking in libsparse, which is from Android sources
One possibility would be including libsparse as a patch, it doesn't change a lot: https://android.googlesource.com/platform/system/core/+log/master/libsparse But it depends on Android's libbase and libz-host.
Bug#924591: this requires linking in libsparse, which is from Android sources
Even though buster's e2fsprogs includes support for android_sparse in the source code, it requires linking against libsparse, which is in android-libsparse. That means making e2fsprogs Build-Depend on android-libsparse-dev.
