Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Fri, Apr 14, 2023 at 08:58:55PM -0700, Steve Langasek wrote: > Sorry for the long radio silence on this; I had identified some issues with > my prior report and wanted to rerun it with some fixes, and that analysis > took rather longer than expected (mainly due to infrastructure slowness). > Refreshed output for Ubuntu is here: > https://people.canonical.com/~vorlon/armhf-time_t/ > So comparing with Wookey's results, I see: > Total -dev packages tested: 4603 > Time_t changes ABI: 349 > LFS changes ABI: 71 > ABI unchanged: 2085 > Compilation failed: 2098 > So certainly this appears to be the same order of magnitude as Wookey's > separate analysis. A little more digging here into the failures; I've prioritized them by number of reverse-build-depends: while read devpkg; do count=$(grep-dctrl -w -FBuild-Depends $devpkg -n -sPackage \ /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources \ | sort -u | wc -l) echo "$count $devpkg"; done < failed-compilation \ | sort -n -r > sorted-revdep-count One nice finding is that 807 of the libraries that have failed analysis have no reverse-build-dependencies at all, so can be more or less ignored for the transition (we should still follow through on the analysis and rename packages, but this can happen any time during the trixie release cycle). One *unfortunate* finding, having fixed up the compilation of the top 5 failures, is that each of libssl-dev, libgtk-3-dev, and libgtk2.0-dev breaks ABI with 64-bit time_t. > The other question is what impact this has on the archive as a whole, with > triggering of rebuild transitions. Looking at the > reverse-build-dependencies of those 349 packages gives: > > $ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg > /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < > abi-breaks | sort -u | wc -l > 3626 > $ But on the bright side again, this only increases the number of revdeps that are part of the transition from 3626 to 4004, due to overlap with other libraries. From my perspective, this is entirely achievable and we should move forward with discussing this more broadly on debian-devel. Note that while armhf is the priority wrt Ubuntu, this would also address 2038 compatibility for armel and mipsel as Debian release archs. (I do not think we should change i386 to 64-bit time_t as the primary value of this arch is for binary compatibility with legacy applications that can't be recompiled.) > NB this is a bit of an undercount as a result of packages that ship headers > but aren't the thing being directly build-depended on. A more accurate > count is probably to look at binary packages, from the same source as the > -dev package, that the -dev package depends on and look at > reverse-dependencies of those. But for a first pass, here's where we are. > There are also roughly 2100 packages for which compilation failed. To know > which packages actually have ABI changes and require transitions, we would > need to work through this list and get the count down to 0. > But as a first-order approximation, this looks doable to me as a transition > we could manage. > What do other folks think? Are you on board with proposing this to > debian-devel? Anyone want to help add the necessary quirks to drive down > the list of compilation failures? > On Tue, Mar 21, 2023 at 03:00:41AM +, Wookey wrote: > > On 2023-02-15 17:08 +, Wookey wrote: > > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > > > > > Yes I think we should proceed with this analysis on debian to get a > > > better handle on just how many libraries we are looking at. > > > > OK. We have over 10,000 *-dev package in debian so this took a while, > > but I've now finished an initial pass. > > The results are: > > Total -dev packages: 10323 > > Time_t changes ABI: 329 > > LFS changes ABI: 580.6% > > ABI unchanged: 18405.6% > > Compilation failed: 1637 > > No headers in pkg: 3852 > > No library in pkg: 5086 > > total analysed: 10249 > > (note that the no-headers and no-libs categories are not exclusive > > - the other classifications are) > > > > Not quite sure where we lost 74 packages down the back of the sofa but at > > 0.7% it's noise. > > > > So we have about 5000 packages which can basically be ignored for this > > purpose: > > golang* (1943) > > librust* (1955) - source-only, no dynamic linking, no .so's in any package > > and libghc* (1065) - changes ABI at drop of hat (every new version) anyway. > > > > Of the remaining 5360, 5237 have .so files to check. Of those 329 > > change ABI and another 58 are not built with LFS currently > > so also change ABI due to that. That's 387 (7%). > > > > 34% are fine. And an annoyingly large 1637 (30%) did not build under > > Abi-compliance-checker to determine one way or the other. Assuming 7% > > of those are a problem that's another 114 packages. > > > > I've yet to determine how many of
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Wed, Feb 15, 2023 at 05:08:40PM +, Wookey wrote:
> On 2023-02-04 21:42 -0800, Steve Langasek wrote:
> > On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
> > > This is good. One thing that I think has been missing from the discussion
> > > about armhf rebootstrap is the fact that we do have experience in Debian
> > > doing cross-cutting ABI transitions without having to change an
> > > architecture name. We've done this at least three times in Debian
> > > history:
> > > the libc5->libc6 transition ('g' suffix - which still remains today in
> > > libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
> > > double' transition ('ldbl' suffix, and an example of doing this for an ABI
> > > transition that didn't affect all architectures).
> I feel like I should already know this, but after a bit of thought,
> and reading the transitions wiki page:
> https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
> understand how the suffixes are used in these transitions.
> That wiki page does not appear to mention adding suffixes to package
> names. It appears to only document the process where the version is
> just bumped, everything involved is rebuilt in experimental and then
> re-uploaded to unstable once we are ready to make the change.
> I presume that the suffix is used for transitions involving a large
> number of packages, in order to keep both the old-ABI and new-ABI
> version of the packages around in parallel (and conflicting with each
> other?) for a while which is necessary for larger transitions because
> doing it all in one go would take too long, or get entangled with
> uploads from the unware that might cause breakage, or something?
When we have an ABI change in the distro that affects shared libraries, but
is not the result of sourceful changes to the upstream source, we don't want
to change the soname of the library; that's upstream's domain.
But we do need at the packaging level to distinguish between two packages of
the library that provide different ABIs, so that packages depending on one
ABI don't incorrectly get the package with the other ABI, resulting in
broken runtime behavior.
When we transition, the un-annotated package will remain present in older
releases, but will be entirely superseded in unstable by the package
providing the new ABI.
BTW, I'll suggest "t64" as the library name suffix here.
> Just rebuilding and not renaming sounds like a much smoother process,
> because as soon as we rename then there is a load of renaming in
> reverse dependencies too, and there is presumably a set of rules about
> when conflicts are added and other details that need to be got right.
No, that breaks partial upgrades and makes a right mess of unstable *and* of
upgrades between stable releases. "Just rebuilding and not renaming" is the
"rebootstrap" approach which I am hoping to avoid.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
slanga...@ubuntu.com vor...@debian.org
signature.asc
Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
Sorry for the long radio silence on this; I had identified some issues with my prior report and wanted to rerun it with some fixes, and that analysis took rather longer than expected (mainly due to infrastructure slowness). Refreshed output for Ubuntu is here: https://people.canonical.com/~vorlon/armhf-time_t/ So comparing with Wookey's results, I see: Total -dev packages tested: 4603 Time_t changes ABI: 349 LFS changes ABI: 71 ABI unchanged: 2085 Compilation failed: 2098 So certainly this appears to be the same order of magnitude as Wookey's separate analysis. The other question is what impact this has on the archive as a whole, with triggering of rebuild transitions. Looking at the reverse-build-dependencies of those 349 packages gives: $ while read pkg; do grep-dctrl -n -sPackage -w -FBuild-Depends $pkg /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_lunar_*Sources ; done < abi-breaks | sort -u | wc -l 3626 $ NB this is a bit of an undercount as a result of packages that ship headers but aren't the thing being directly build-depended on. A more accurate count is probably to look at binary packages, from the same source as the -dev package, that the -dev package depends on and look at reverse-dependencies of those. But for a first pass, here's where we are. There are also roughly 2100 packages for which compilation failed. To know which packages actually have ABI changes and require transitions, we would need to work through this list and get the count down to 0. But as a first-order approximation, this looks doable to me as a transition we could manage. What do other folks think? Are you on board with proposing this to debian-devel? Anyone want to help add the necessary quirks to drive down the list of compilation failures? On Tue, Mar 21, 2023 at 03:00:41AM +, Wookey wrote: > On 2023-02-15 17:08 +, Wookey wrote: > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > > > Yes I think we should proceed with this analysis on debian to get a > > better handle on just how many libraries we are looking at. > > OK. We have over 10,000 *-dev package in debian so this took a while, > but I've now finished an initial pass. > The results are: > Total -dev packages: 10323 > Time_t changes ABI: 329 > LFS changes ABI: 580.6% > ABI unchanged: 18405.6% > Compilation failed: 1637 > No headers in pkg: 3852 > No library in pkg: 5086 > total analysed: 10249 > (note that the no-headers and no-libs categories are not exclusive > - the other classifications are) > > Not quite sure where we lost 74 packages down the back of the sofa but at > 0.7% it's noise. > > So we have about 5000 packages which can basically be ignored for this > purpose: > golang* (1943) > librust* (1955) - source-only, no dynamic linking, no .so's in any package > and libghc* (1065) - changes ABI at drop of hat (every new version) anyway. > > Of the remaining 5360, 5237 have .so files to check. Of those 329 > change ABI and another 58 are not built with LFS currently > so also change ABI due to that. That's 387 (7%). > > 34% are fine. And an annoyingly large 1637 (30%) did not build under > Abi-compliance-checker to determine one way or the other. Assuming 7% > of those are a problem that's another 114 packages. > > I've yet to determine how many of the 'no-libs/no-headers' packages > actually expose an ABI we need to worry about. There will be some > more, from stuff like boost and Qt. > > So the scale of the transition task in Debian is quite a lot bigger than in > Ubuntu: Probably 400-500 packages. > > I've linked my lists on the wiki page. > https://wiki.debian.org/ReleaseGoals/64bit-time > > I believe Steve L (who has done most of the heavy lifting on the > script) is running a check too and will have some results soon. > > Wookey > -- > Principal hats: Debian, Wookware, ARM > http://wookware.org/ -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developer https://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On 2023-02-15 17:08 +, Wookey wrote: > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > Yes I think we should proceed with this analysis on debian to get a > better handle on just how many libraries we are looking at. OK. We have over 10,000 *-dev package in debian so this took a while, but I've now finished an initial pass. The results are: Total -dev packages: 10323 Time_t changes ABI: 329 LFS changes ABI: 580.6% ABI unchanged: 18405.6% Compilation failed: 1637 No headers in pkg: 3852 No library in pkg: 5086 total analysed: 10249 (note that the no-headers and no-libs categories are not exclusive - the other classifications are) Not quite sure where we lost 74 packages down the back of the sofa but at 0.7% it's noise. So we have about 5000 packages which can basically be ignored for this purpose: golang* (1943) librust* (1955) - source-only, no dynamic linking, no .so's in any package and libghc* (1065) - changes ABI at drop of hat (every new version) anyway. Of the remaining 5360, 5237 have .so files to check. Of those 329 change ABI and another 58 are not built with LFS currently so also change ABI due to that. That's 387 (7%). 34% are fine. And an annoyingly large 1637 (30%) did not build under Abi-compliance-checker to determine one way or the other. Assuming 7% of those are a problem that's another 114 packages. I've yet to determine how many of the 'no-libs/no-headers' packages actually expose an ABI we need to worry about. There will be some more, from stuff like boost and Qt. So the scale of the transition task in Debian is quite a lot bigger than in Ubuntu: Probably 400-500 packages. I've linked my lists on the wiki page. https://wiki.debian.org/ReleaseGoals/64bit-time I believe Steve L (who has done most of the heavy lifting on the script) is running a check too and will have some results soon. Wookey -- Principal hats: Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Fri, 2023-03-03 at 04:07 +, Wookey wrote: > Is there a place one can grep all buildlogs? Debian members can access them on the buildd.debian.org server for eg: /srv/buildd.debian.org/db/0/0ad/0.0.11-1/i386_1347540026_log.bz2 Please keep an eye on system load while searching them though. If you only need build-dep versions, probably a better option is the archive of buildinfo files and the database for searching it. https://buildinfos.debian.net/ftp-master.debian.org/ https://salsa.debian.org/bremner/builtin-pho/ -- bye, pabs https://wiki.debian.org/PaulWise signature.asc Description: This is a digitally signed message part
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On 2023-02-20 17:53 +0100, Diederik de Haas wrote: > On Monday, 20 February 2023 16:24:37 CET Arnd Bergmann wrote: > > On Wed, Feb 15, 2023, at 18:08, Wookey wrote: > > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > > >> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote: > > >> > > >> Does doing an ABI transition of ~113 libraries seem tractable to folks? > > > > > > It certainly provides evidence for the idea that this is not > > > completely intractable, which I think many people (including me) > > > worried was the case initially. > > > > When I did a similar analysis a few years ago using just pattern-matching > > on header files, the result was that more than half the total packages > > in Debian depended on at least one other package that needed an ABI > > transition, which in my mind made it unrealistic. > If you do it in the early stage of Trixie's dev cycle, would it still be > unrealistic? It may be a bumpy ride and take a while, but I don't see an > immediate issue with that. Sid may finally become Unstable again ;-P > > But *when* you do it, is quite relevant. If you/we are only a few months away > from the Trixie Freeze, then it's probably not a good idea. > But if we're 1-1.5 years before that, there's plenty of time to fix things. > > Or is that too simplistic on my part? No. I think that's reasonable. One thing we do have to worry about right now is that anything which has enabled LFS anytime in the last 15 years that rebuilds against a current gnulib will automatically get 64-bit time_t unless they explicitly turn it off. We may well have packages in the archive that have had this done to them without noticing, and clearly we don't want to do this in bookworm. I'm not quite sure what the best way to check for this is. Is there a place one can grep all buildlogs? AIUI gnulib is statically included in the build a-la autoconf so it probably only happens when a new upstream is pulled in with updated gnulib. BICBW. gnulib doesn't seem to do releases (last tag 9 years ago 'v0.1') so I'm not quite sure when this changed. Ah July 2021: https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=history;f=m4/year2038.m4;h=2e4427e6fac10550c99748abebf31b61e6afda2b;hb=b9bf95fd0a6ab666b484b1e224321664f051f7fa So the module has existed since 2017, but only defaults on since mid 2021. I'm not sure what one should be looking for in build logs to set off the alarms. Wookey -- Principal hats: Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Monday, 20 February 2023 16:24:37 CET Arnd Bergmann wrote: > On Wed, Feb 15, 2023, at 18:08, Wookey wrote: > > On 2023-02-04 21:42 -0800, Steve Langasek wrote: > >> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote: > >> > >> Does doing an ABI transition of ~113 libraries seem tractable to folks? > > > > It certainly provides evidence for the idea that this is not > > completely intractable, which I think many people (including me) > > worried was the case initially. > > When I did a similar analysis a few years ago using just pattern-matching > on header files, the result was that more than half the total packages > in Debian depended on at least one other package that needed an ABI > transition, which in my mind made it unrealistic. If you do it in the early stage of Trixie's dev cycle, would it still be unrealistic? It may be a bumpy ride and take a while, but I don't see an immediate issue with that. Sid may finally become Unstable again ;-P But *when* you do it, is quite relevant. If you/we are only a few months away from the Trixie Freeze, then it's probably not a good idea. But if we're 1-1.5 years before that, there's plenty of time to fix things. Or is that too simplistic on my part? signature.asc Description: This is a digitally signed message part.
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On Wed, Feb 15, 2023, at 18:08, Wookey wrote: > On 2023-02-04 21:42 -0800, Steve Langasek wrote: >> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote: > >> Does doing an ABI transition of ~113 libraries seem tractable to folks? > > It certainly provides evidence for the idea that this is not > completely intractable, which I think many people (including me) > worried was the case initially. When I did a similar analysis a few years ago using just pattern-matching on header files, the result was that more than half the total packages in Debian depended on at least one other package that needed an ABI transition, which in my mind made it unrealistic. It's possible that there were a lot of false-positive results in my analysis then, but I would still suggest looking at what portion of the archive depends on the libraries that you identified this time. Arnd
Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]
On 2023-02-04 21:42 -0800, Steve Langasek wrote:
> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
> > This is good. One thing that I think has been missing from the discussion
> > about armhf rebootstrap is the fact that we do have experience in Debian
> > doing cross-cutting ABI transitions without having to change an
> > architecture name. We've done this at least three times in Debian history:
> > the libc5->libc6 transition ('g' suffix - which still remains today in
> > libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
> > double' transition ('ldbl' suffix, and an example of doing this for an ABI
> > transition that didn't affect all architectures).
I feel like I should already know this, but after a bit of thought,
and reading the transitions wiki page:
https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
understand how the suffixes are used in these transitions.
That wiki page does not appear to mention adding suffixes to package
names. It appears to only document the process where the version is
just bumped, everything involved is rebuilt in experimental and then
re-uploaded to unstable once we are ready to make the change.
I presume that the suffix is used for transitions involving a large
number of packages, in order to keep both the old-ABI and new-ABI
version of the packages around in parallel (and conflicting with each
other?) for a while which is necessary for larger transitions because
doing it all in one go would take too long, or get entangled with
uploads from the unware that might cause breakage, or something?
Can someone point at an explanation of the process, or add it to the above page?
Just rebuilding and not renaming sounds like a much smoother process,
because as soon as we rename then there is a load of renaming in
reverse dependencies too, and there is presumably a set of rules about
when conflicts are added and other details that need to be got right.
I presume there is a good reason for this distinction between 'minor'
transitions and 'mega' transitions. I'd like to understand it properly.
> > Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
> > the ABIs of C libraries to accurately identify what needs to change for
> > time64_t. I think it would be interesting to know from this how many shared
> > libraries expose time_t size in their ABIs.
>
> Well, I didn't see anybody else expressing interest in this, so I had a go.
>
> I've done a first-pass analysis of Ubuntu lunar/armhf using
> abi-compliance-checker. The results of my investigation, including scripts
> and detailed reports/logs, are here:
Thanks very much for this. Really helpful. I tried to use abi-dumper
and abi-compliance-checker but wasn't having much success in getting
useful reports out of them. I found abigail-tools to be quite a lot
simpler to use. Probably we should run both and investigat if they
disagree about differences.
I found out about your effort just in time to include it in my FOSDEM talk,
which was good.
https://fosdem.org/2023/schedule/event/fixing_2038/ for anyone who is
unaware. (The Q has some dropouts and didn't add all that much
useful info so you can stop at about 17:20 without missing much).
> - abi-compliance-checker was used to analyze headers of 767 library
> packages in the archive. Of those, 82 of 558 were successfully analyzed
> and identified as library packages that would need patched for the
> switch to 64-bit time_t to mark them as ABI-incompatible with the
> previous version on armhf.
>
> - an additional 209 packages could not be analyzed, because a-c-c failed
> to compile the headers using its invocation of gcc. Assuming a roughly
> equal distribution of ABI-changing vs non-ABI-changing libraries among
> these that have failed to compile, we'd be looking at around 113 of 767
> libraries that need changed.
>
> - if we decide the level of effort to patch this many libraries is
> worthwhile, to get an actual actionable list of libraries that need
> patched for this transition would require getting the compile failures
> down to zero. I did dig into the failures alphabetically from "a" to
> "libe"; most of these failures were resolved by one of three methods.
> - letting a-c-c know that this is a C library and to not try to compile
> it with gcc's C++ frontend;
> - excluding various headers that can't be included directly, either
> because they are internal headers that are included via other headers
> in the package, or because they have dependencies on other headers not
> in the archive and can't be compiled (and therefore can't affect the
> library ABI);
> - pulling in additional undeclared package dependencies.
> Even when applying these fixes, I still had 8 library packages whose
> headers I wasn't able to get to compile. So there's still quite a bit
> of work to do here.
>
>
