Re: Planning for libidn shared library version transition

[Andreas Metzler]

On 2021-07-27 Simon Josefsson  wrote:
> Hi!  I'm now resuming work on the libidn shared library transition, and
> I'm ready for the upload to experimental.  I wanted to ping back here to
[...]

Hello Simon,

thank you, looks good to me.

cu Andreas

Re: Planning for libidn shared library version transition

[Simon Josefsson]

Hi!  I'm now resuming work on the libidn shared library transition, and
I'm ready for the upload to experimental.  I wanted to ping back here to
get more review.  I'm following Andreas Metzler's outline, but included
some tweaks suggested by Simon McVittie.  I decided to do some more
changes that are unrelated here (e.g., drop the would-be poorly named
libidn11-java package which nobody appears to use, and drop the no
longer working Emacs libraries).

The old package (1.33-3, in bullseye) looks like the following (yeah,
the 'Replaces: libidn11-dev' on the 'libidn11' package is confusing, and
could potentially cause problems here):

Package: libidn11
Architecture: any
Depends: ${misc:Depends}, ${shlibs:Depends}
Conflicts: libidn9-dev
Replaces: libidn11-dev
Pre-Depends: ${misc:Pre-Depends}
Multi-Arch: same
Description: GNU Libidn library, implementation of IETF IDN specifications

Package: libidn11-dev
Section: libdevel
Architecture: any
Depends: libidn11 (= ${binary:Version}), pkg-config, ${misc:Depends}
Conflicts: libidn9-dev
Multi-Arch: same
Description: Development files for GNU Libidn, an IDN library

The new package (1.38-1) that I'm planning to upload has this:

Package: libidn12
Architecture: any
Depends: ${misc:Depends}, ${shlibs:Depends}
Pre-Depends: ${misc:Pre-Depends}
Multi-Arch: same
Description: GNU Libidn library, implementation of IETF IDN specifications

Package: libidn-dev
Section: libdevel
Architecture: any
Depends: libidn12 (= ${binary:Version}), pkg-config, ${misc:Depends}
Breaks: libidn11-dev (<< 1.38-1~)
Replaces: libidn11-dev (<< 1.38-1~)
Suggests: idn
Multi-Arch: same
Description: Development files for GNU Libidn, an IDN library

Package: libidn11-dev
Section: oldlibs
Architecture: any
Depends: libidn-dev (>= 1.38-1~), ${misc:Depends}
Multi-Arch: same
Description: Transitional development package for GNU Libidn

Libidn11-dev's '>= 1.38-1~' dependency, and the '<< 1.38-1~' in
libidn12, came as a suggestion from Simon McVittie.

It builds fine, and reverse dependencies builds too (the two failures
are not libidn-issues):
https://salsa.debian.org/debian/libidn/-/pipelines/270968

I'm including more complete debdiff below.

What do you think?

I'm not only looking to catch fatal mistakes, but also want ideas on how
to improve this further, so all feedback is welcome.

Thanks,
/Simon

jas@latte:~/dpkg$ debdiff --show-moved libidn_1.33-3_amd64.changes 
libidn_1.38-1_amd64.changes 
Warning: these package names were in the second list but not in the first:
--
libidn-dev
libidn12
libidn12-dbgsym


Warning: these package names were in the first list but not in the second:
--
libidn11
libidn11-dbgsym
libidn11-java

[The following lists of changes regard files as different if they have
different names, permissions or owners.]

Files only in first set of .debs, found in package idn
--
-rw-r--r--  root/root   /usr/share/doc/idn/AUTHORS.gz
-rw-r--r--  root/root   /usr/share/doc/idn/TODO
-rw-r--r--  root/root   /usr/share/emacs/site-lisp/idna.el
-rw-r--r--  root/root   /usr/share/emacs/site-lisp/punycode.el

Files only in first set of .debs, found in package idn-dbgsym
-
-rw-r--r--  root/root   
/usr/lib/debug/.build-id/b8/51518b886fad3ef49ddda44ba667ac043e60c0.debug

Files only in first set of .debs, found in package libidn11
---
-rw-r--r--  root/root   /lib/x86_64-linux-gnu/libidn.so.11.6.16
-rw-r--r--  root/root   /usr/share/doc/libidn11/changelog.Debian.gz
-rw-r--r--  root/root   /usr/share/doc/libidn11/changelog.gz
-rw-r--r--  root/root   /usr/share/doc/libidn11/copyright
lrwxrwxrwx  root/root   /lib/x86_64-linux-gnu/libidn.so.11 -> libidn.so.11.6.16

Files moved from package libidn11 to package libidn12
-
-rw-r--r--  root/root   DEBIAN/shlibs
-rw-r--r--  root/root   DEBIAN/symbols
-rw-r--r--  root/root   DEBIAN/triggers

Files only in first set of .debs, found in package libidn11-dbgsym
--
-rw-r--r--  root/root   
/usr/lib/debug/.build-id/14/d4bc0771a40445c675c5c4682d56465463ccc8.debug
lrwxrwxrwx  root/root   /usr/share/doc/libidn11-dbgsym -> libidn11

Files only in first set of .debs, found in package libidn11-dev
---
-rw-r--r--  root/root   /usr/share/doc-base/libidn11
-rw-r--r--  root/root   /usr/share/doc/libidn11-dev/examples/README
-rw-r--r--  root/root   /usr/share/doc/libidn11-dev/examples/example.c
-rw-r--r--  root/root   /usr/share/doc/libidn11-dev/examples/example2.c
-rw-r--r--  root/root   /usr/share/doc/libidn11-dev/examples/example3.c
-rw-r--r--  root/root   /usr/share/doc/libidn11-dev/examples/exampl

Re: Planning for libidn shared library version transition

[David Kalnischkies]

On Fri, May 28, 2021 at 05:12:01PM +0100, Simon McVittie wrote:
> On Thu, 27 May 2021 at 16:53:45 +0200, David Kalnischkies wrote:
> > dpkg has the notion of "disappearing packages" (packages which have no
> > files left on a system) which could solve this cleanup compulsion, but
> > it is currently not supported (as in forbidden in practice) in Debian.
> 
> Am I correct to think that the reason this is forbidden in practice is
> the requirement that every package contains either its changelog and
> copyright file in /usr/share/doc/${package}/{changelog.Debian.gz,copyright},
> or a symlink /usr/share/doc/${package} -> /usr/share/doc/${other}, either
> of which will prevent the package from fully disappearing because the
> replacement package isn't going to contain those files?

Well, the idea is that the newpkg provides+conflicts+replaces with the
oldpkg. oldpkg contains just a softlink from /usr/share/doc/oldpkg
to newpkg and Depends on newpkg. The newpkg replaces this symlink by
containing it as well which leaves oldpkg with no remaining files which
in turn leads dpkg to make oldpkg disappear.

dpkg can only do this if no package on the system currently depends on
oldpkg though. And that is where the tricky problems begin as dpkg could
have made oldpkg disappear before another package is installed which
happens to depend on oldpkg (I guess versioned provides come to our
rescue here actually, now that I think about it).

It also means that this can only be used for the most straightforward of
transitions: a simple package rename – as as soon as you want the
transition package to actually contain meaningful content (like a node
→ nodejs symlink for compat) its not applicable anymore, so its rather
special interest.


I seem to have forgotten what the big problem preventing "widespread"
use was last time. I thought it would be forbidden by policy, but it
isn't as you point out. At this point it might be just "nobody used it
in production for a decade, there might be bugs". Guillem as the dpkg
maintainer probably knows more about it than I do if there is interest
in pursuing this further.


Best regards

David Kalnischkies


signature.asc
Description: PGP signature

Re: Planning for libidn shared library version transition

[Simon McVittie]

On Thu, 27 May 2021 at 16:53:45 +0200, David Kalnischkies wrote:
> So, here is a thing: I like transitional packages – because it means the
> package is not removed.

Right - it tells both apt and human users "this
replacement/removal/upgrade (depending how you look at it) is intentional,
according to the uploader of the transitional package".

For what it's worth, I think I agree that this is preferable to
Obsoletes.

> dpkg has the notion of "disappearing packages" (packages which have no
> files left on a system) which could solve this cleanup compulsion, but
> it is currently not supported (as in forbidden in practice) in Debian.

Am I correct to think that the reason this is forbidden in practice is
the requirement that every package contains either its changelog and
copyright file in /usr/share/doc/${package}/{changelog.Debian.gz,copyright},
or a symlink /usr/share/doc/${package} -> /usr/share/doc/${other}, either
of which will prevent the package from fully disappearing because the
replacement package isn't going to contain those files?

smcv

Re: Planning for libidn shared library version transition

[Holger Levsen]

On Thu, May 27, 2021 at 11:14:43AM -0700, Russ Allbery wrote:
> This is an incredibly useful thread that contains a lot of concrete
> recommendations for how to do library transitions smoothly.  Could someone
> who has a moment collect this information into some how-to documentation
> and submit it as a bug against developers-reference so that we don't lose
> it?
> 
> (We can then also grab some of it for Policy where appropriate, but I
> think most of it is more of a Developer's Reference thing.)

yes, please. and just commit to developers-reference.git please, it's
easier and less work to polish those commits than have yet another todo
reminder in the bts. (and if thats not possible a bug is still better than
no bug.)


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

There are no jobs on a dead planet.


signature.asc
Description: PGP signature

Re: Planning for libidn shared library version transition

[Russ Allbery]

This is an incredibly useful thread that contains a lot of concrete
recommendations for how to do library transitions smoothly.  Could someone
who has a moment collect this information into some how-to documentation
and submit it as a bug against developers-reference so that we don't lose
it?

(We can then also grab some of it for Policy where appropriate, but I
think most of it is more of a Developer's Reference thing.)

-- 
Russ Allbery (r...@debian.org)  

Re: Planning for libidn shared library version transition

[Marvin Renich]

* Marvin Renich  [210527 13:41]:
> packages use the same wording in the description, and searching for the
> specific word "transitional" has a non-negligible chance of a false
^
or  "dummy"

...Marvin

Re: Planning for libidn shared library version transition

[Marvin Renich]

* Andrey Rahmatullin  [210527 10:38]:
> On Thu, May 27, 2021 at 10:10:15AM -0400, Marvin Renich wrote:
> > If transitional packages either had a debtag or a control file field
> > that identified them, then tools like deborphan could easily be made to
> > find and remove them
> Oh they already do (both the oldlibs section and the "dummy" description
> substring are supported by deborphan), but deborphan needs to be run
> manually.

oldlibs has a different purpose:  to identify libraries that are
outdated, but that the user might want for various reasons (e.g.
non-Debian software that requires an older library version).

Using text in the description seems, uh, kludgey at best.  Something
more specific and canonical would be an immense improvement.  Just
looking through oldlibs makes it obvious that not all transitional
packages use the same wording in the description, and searching for the
specific word "transitional" has a non-negligible chance of a false
positive.

...Marvin

Re: Planning for libidn shared library version transition

[David Kalnischkies]

On Thu, May 27, 2021 at 11:18:28AM +0100, Simon McVittie wrote:
> On Thu, 27 May 2021 at 08:30:11 +0200, Simon Josefsson wrote:
> > I do think it would be a wortwhile release
> > goal, at some point, to fix tooling so that dummy packages are no longer
> > necessary for package transitions.
> 
> Transitional packages have the huge advantage that they already work
> (and the reason they work is a side-effect of how ordinary non-transition
> upgrades work, so it isn't going to go away). Making them unnecessary
> seems likely to be more code (therefore more bugs), so it would have to
> be counterbalanced with a real advantage. If we are replacing a "real"
> package with a transitional package, then by definition that package
> name is no longer in use, so having the transitional package "occupy"
> the name is harmless.

So, here is a thing: I like transitional packages – because it means the
package is not removed. The thread kinda claims that apt has problems
with removes and in some way it has, but it is actually one of the more
relaxed ones as its greedy solver can also decide to remove too much…

Users can be really averse to package removals. And we reinforce this
with things like "upgrade vs. full-upgrade". Many solvers take the
amount of packages to be removed as a prime indicator for how "good"
a solution is. The result is that a clever solver (which apt is not) can
decided to upgrade KDE3 to GNOME instead of KDE4 because the later would
remove more packages (that isn't to pick on either desktop environment
or other solvers, its just a colorful and simple stupid example which
actually happened while people tried to show how bad apt is compared to
a SAT solver years ago).

So, I think this is not so much about teaching software and users alike
that there exists an upgrade path without a transitional package, but
that a given package removal is "okay" as you don't loose features which
is the main fear of users. The knowledge can usually be found somewhere
deep in a bugreport or the changelog of a package and if you are super
lucky in the release notes. Same for "not okay, but required" removals
like python2 stuff obsoleted by two other apps to 70% each…


> The one non-cosmetic reason I can think of why transitional packages
> are potentially bad is that they aren't removed automatically, so systems
> that have been upgraded many times can end up with a lot of transitional
> packages; not needing transitional packages would make
> https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#obsolete
> somewhat less necessary (although still necessary, to deal with packages
> that were removed with no direct replacement). That doesn't seem hugely
> compelling to me?

A transitional package can be semi-automatic removed by autoremove like
any other package if nothing has a positive dependency on it anymore, is
not manually installed and not priority required or "worse" (aka
essential and such). A package which switches its section to 'oldlibs'
can even get rid of its own marking as manual, as if it has this
marking it will move the marking to the package(s) it depends on and
mark itself as automatic installed (in apt and "normal" libapt users at
least, never tried with aptitude which is an "unusual" libapt user).

And as a sidenote, in bullseye apt supports patterns similar to
aptitude, so e.g. ~o works as noted in the release notes in apt now.

dpkg has the notion of "disappearing packages" (packages which have no
files left on a system) which could solve this cleanup compulsion, but
it is currently not supported (as in forbidden in practice) in Debian.


> If you want to pursue this, the right way to do it would be to send a
> bug report to apt. I'm not sure how feasible it is to make superseded
> packages identifiable and make apt assign a more favourable "score"
> to removing them, but apt maintainers would know. One possible route
> would be an equivalent of rpm's Obsoletes field.

Fun fact: apt has some support for rpm's Obsolete, but mostly from
a failed attempt to merge apt-rpm decades ago and probably entirely
broken if we would actually end up using it. ABI wise it exists though.
From the same area came support for versioned provides by the way, but
I had used that years before already for implementing MultiArch in APT
in 2010 before it was made available to the general public (another apt
internals thing is the "libsame breaks (!= 1)" also used by M-A).

The problem with Obsolete and giving it too much value is a bit about
social issues like "vim obsoletes emacs" and that transitions tends to
not be that clear cut like multiple forks/package splits and at times
transitions are even rolled back a few months later… so the clear
upgrade path we tend to imagine here looks in practice more like the
tricky mess which results if you put three cables neatly separated
in a box and you look away for a split second…


> If it's feasible to solve this, then I suspect the only packages tha

automated pkg set installation and upgrade tests (was Re: Planning for libidn shared library version transition)

[Holger Levsen]

hi,

On Wed, May 26, 2021 at 11:28:24PM +0100, Simon McVittie wrote:
> piuparts only routinely tests relatively small installations - there are
> periodic QA tests done on larger systems like the union of all desktop
> tasks, but those are more expensive to run.

indeed, on jenkins.debian.net there are 421 of those package installation
tests running (called chroot-installation there for historic reasons.)

I'd be more than glad to run *more* tests there! *many* more tests!
please read this mail completly and then reply, send MR, join #debian-qa
and ask me there or some such.

those 421 tests are divided into testing these suites:

holger@jenkins:/var/lib/jenkins/jobs$ for i in stretch buster bullseye sid ; do 
echo -n "$i: " ; ls -1d chroot-installation_*|grep -c $i ; done
stretch: 97
buster: 208
bullseye: 162
sid: 162

so quite many of them test installation in one suite and then
upgrading to the next.

trigger_times = {
'stretch':  '30 10 1 */2 *',
'buster': '30 10 4,11,18,25 * *',
'bullseye': '30 10 */3 * *',
'sid': '30 4 * * *',
}

is when the jobs are triggered and that's coming from the job
definitions in 
https://salsa.debian.org/qa/jenkins.debian.net/-/blob/master/job-cfg/chroot-installation.yaml.py

the packages to be installed are defined in 
https://salsa.debian.org/qa/jenkins.debian.net/-/blob/master/bin/chroot-installation.sh
in lines 227 until 267.

currently the package sets being tested are mostly desktop 
environments and Debian Edu package sets:

all_targets = [
   'gnome',
   'kde',
   'kde-full',
   'cinnamon',
   'lxde',
   'lxqt',
   'xfce',
   'full_desktop',
   'qt4',
   'qt5',
   'haskell',
   'developer',
   'debconf-video',
   'education-tasks',
   'education-menus',
   'education-astronomy',
   'education-chemistry',
   'education-common',
   'education-desktop-gnome',
   'education-desktop-kde',
   'education-desktop-lxde',
   'education-desktop-lxqt',
   'education-desktop-mate',
   'education-desktop-other',
   'education-desktop-xfce',
   'education-desktop-cinnamon',
   'education-development',
   'education-electronics',
   'education-geography',
   'education-graphics',
   'education-language',
   'education-laptop',
   'education-logic-games',
   'education-ltsp-server',
   'education-main-server',
   'education-mathematics',
   'education-misc',
   'education-music',
   'education-networked',
   'education-physics',
   'education-primaryschool',
   'education-services',
   'education-standalone',
   'education-thin-client',
   'education-roaming-workstation',
   'education-video',
   'education-workstation',
   'parl-desktop-eu',
   'parl-desktop-strict',
   'parl-desktop-world',
   'design-desktop-animation',
   'design-desktop-graphics',
   'design-desktop-strict',
   'design-desktop-web',
   ]

(again from that yaml.py file linked above.)

A list of all tests is attached and a list of all tests currently flagged
being problematic is available at 
https://jenkins.debian.net/view/chroot-installation/view/Problems/

Let's test all the things!


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

Society: Be Yourself!
Society: No, not like that.
chroot-installation_bullseye_bootstrap
chroot-installation_bullseye_bootstrap_upgrade_to_sid
chroot-installation_bullseye_install_cinnamon
chroot-installation_bullseye_install_cinnamon_upgrade_to_sid
chroot-installation_bullseye_install_debconf-video
chroot-installation_bullseye_install_debconf-video_upgrade_to_sid
chroot-installation_bullseye_install_design-desktop-animation
chroot-installation_bullseye_install_design-desktop-animation_upgrade_to_sid
chroot-installation_bullseye_install_design-desktop-graphics
chroot-installation_bullseye_install_design-desktop-graphics_upgrade_to_sid
chroot-installation_bullseye_install_design-desktop-strict
chroot-installation_bullseye_install_design-desktop-strict_upgrade_to_sid
chroot-installation_bullseye_install_design-desktop-web
chroot-installation_bullseye_install_design-desktop-web_upgrade_to_sid
chroot-installation_bullseye_install_developer
chroot-installation_bullseye_install_developer_upgrade_to_sid
chroot-installation_bullseye_install_education-astronomy
chroot-installation_bullseye_install_education-astronomy_upgrade_to_sid
chroot-installation_bullseye_install_education-chemistry
chroot-installation_bullseye_install_education-chemistry_upgrade_to_sid
chroot-installation_bullseye_install_education-common
chroot-installation_bullseye_install_education-common_upgrade_to_sid
chroot-installation_bullseye_install_education-desktop-cinnamon
chroot-installation_bullseye_install_education-desktop-cinnamon_upgrade_to_sid
chroot-installation_bullseye_install_education-desktop-gnome
chroot-installation_bullseye_install_education-desktop-gnome_upgrade_to_sid
chroot-installation_bullseye_install_education-desktop-kde
chroot-installation_bullseye_install_educa

Re: Planning for libidn shared library version transition

[Andrey Rahmatullin]

On Thu, May 27, 2021 at 10:10:15AM -0400, Marvin Renich wrote:
> > The one non-cosmetic reason I can think of why transitional packages
> > are potentially bad is that they aren't removed automatically, so systems
> > that have been upgraded many times can end up with a lot of transitional
> > packages;
> 
> If transitional packages either had a debtag or a control file field
> that identified them, then tools like deborphan could easily be made to
> find and remove them
Oh they already do (both the oldlibs section and the "dummy" description
substring are supported by deborphan), but deborphan needs to be run
manually.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Planning for libidn shared library version transition

[Marvin Renich]

* Simon McVittie  [210527 06:19]:
> The one non-cosmetic reason I can think of why transitional packages
> are potentially bad is that they aren't removed automatically, so systems
> that have been upgraded many times can end up with a lot of transitional
> packages;

If transitional packages either had a debtag or a control file field
that identified them, then tools like deborphan could easily be made to
find and remove them, perhaps ensuring that the replacement package was
appropriately marked as manual or automatic.

Once that is common, apt and friends could be made smarter as that
particular itch bothered the responsible maintainers.

...Marvin

Re: Planning for libidn shared library version transition

[Simon Josefsson]

Simon McVittie  writes:

> On Wed, 26 May 2021 at 19:18:24 +0200, Simon Josefsson wrote:
>> Andreas Metzler  writes:
>> > Why not use a versioned Provides *instead* of the dummy package?
>> 
>> Yeah, I never understand exactly when these dummy packages are needed.
>
> My understanding is that they are usually still necessary for smooth
> upgrades from one stable release to the next, because apt's dependency
> resolver tries not to remove packages (to minimize risk of removing
> something you were relying on), and removing a "real" libidn11-dev
> package because its lockstep version dependency is no longer satisfiable
> will still count as removing a package, even if a new libidn-dev now
> Provides it.
>
> It's a probabilistic thing, because apt calculates a "score" for various
> options for the whole upgrade transaction and chooses the one with the
> highest score - on some systems it will be able to figure out the right
> upgrade transaction even without a transitional package, but on others it
> will not. Having a transitional package makes it more likely that the
> intended upgrade happens, because apt gives a high "score" to upgrading a
> package that was already installed.
>
> piuparts only routinely tests relatively small installations - there are
> periodic QA tests done on larger systems like the union of all desktop
> tasks, but those are more expensive to run.

Thanks -- you convinced me to prefer libidn is not a guinea pig for
testing something like this.  I do think it would be a wortwhile release
goal, at some point, to fix tooling so that dummy packages are no longer
necessary for package transitions.  The information to make proper
decisions for 'apt' should be there without using dummy packages, as far
as I understand.  On the other hand, there are probably more important
areas (say, 100% reproducible builds) to work on instead.

/Simon


signature.asc
Description: PGP signature

Re: Planning for libidn shared library version transition

[Simon McVittie]

On Thu, 27 May 2021 at 08:30:11 +0200, Simon Josefsson wrote:
> I do think it would be a wortwhile release
> goal, at some point, to fix tooling so that dummy packages are no longer
> necessary for package transitions.

Transitional packages have the huge advantage that they already work
(and the reason they work is a side-effect of how ordinary non-transition
upgrades work, so it isn't going to go away). Making them unnecessary
seems likely to be more code (therefore more bugs), so it would have to
be counterbalanced with a real advantage. If we are replacing a "real"
package with a transitional package, then by definition that package
name is no longer in use, so having the transitional package "occupy"
the name is harmless.

The one non-cosmetic reason I can think of why transitional packages
are potentially bad is that they aren't removed automatically, so systems
that have been upgraded many times can end up with a lot of transitional
packages; not needing transitional packages would make
https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#obsolete
somewhat less necessary (although still necessary, to deal with packages
that were removed with no direct replacement). That doesn't seem hugely
compelling to me?

If you want to pursue this, the right way to do it would be to send a
bug report to apt. I'm not sure how feasible it is to make superseded
packages identifiable and make apt assign a more favourable "score"
to removing them, but apt maintainers would know. One possible route
would be an equivalent of rpm's Obsoletes field.

If it's feasible to solve this, then I suspect the only packages that
would need code changes would be apt and cupt (and maybe aptitude).
Packages undergoing transitions would likely also need metadata to tell
apt which older packages they supersede.

Bear in mind that if apt is changed to not need transitional packages
(somehow) in stable release n, packages cannot take advantage of that
change until the n+1 cycle.

smcv

Re: Planning for libidn shared library version transition

[Simon McVittie]

On Wed, 26 May 2021 at 19:18:24 +0200, Simon Josefsson wrote:
> Andreas Metzler  writes:
> > Why not use a versioned Provides *instead* of the dummy package?
> 
> Yeah, I never understand exactly when these dummy packages are needed.

My understanding is that they are usually still necessary for smooth
upgrades from one stable release to the next, because apt's dependency
resolver tries not to remove packages (to minimize risk of removing
something you were relying on), and removing a "real" libidn11-dev
package because its lockstep version dependency is no longer satisfiable
will still count as removing a package, even if a new libidn-dev now
Provides it.

It's a probabilistic thing, because apt calculates a "score" for various
options for the whole upgrade transaction and chooses the one with the
highest score - on some systems it will be able to figure out the right
upgrade transaction even without a transitional package, but on others it
will not. Having a transitional package makes it more likely that the
intended upgrade happens, because apt gives a high "score" to upgrading a
package that was already installed.

piuparts only routinely tests relatively small installations - there are
periodic QA tests done on larger systems like the union of all desktop
tasks, but those are more expensive to run.

smcv

Re: Planning for libidn shared library version transition

[Simon Josefsson]

Andreas Metzler  writes:

> On 2021-05-26 Guillem Jover  wrote:
>> On Tue, 2021-05-25 at 19:43:21 +0200, Andreas Metzler wrote:
> [...]
>> I'd probably instead make this a versioned Provides, so that the
>> transitional package can be removed right away from systems, it does
>> not interfere with the transition, and people can switch to the new
>> package in parallel w/o disruption.
>
> Hello,
>
> Why not use a versioned Provides *instead* of the dummy package? We have
> had these a long time our packagment management system should have 100%
> support now.

Yeah, I never understand exactly when these dummy packages are needed.
Before I remembered about dummy packages, I tested a libidn update
without it, and it appeared to build reverse dependencies just fine
(piuparts/reprotest).  The only answer I have seen is that 'some of our
old tooling behaved strange' if you didn't have a dummy transitional
package, without specific references to actual tools or versions.

Maybe we can try a transition without a dummy package, and use it as an
experiment to see what breaks?  I suspect that after a release, it is a
good time to do it.

/Simon


signature.asc
Description: PGP signature

Re: Planning for libidn shared library version transition

[Andreas Metzler]

On 2021-05-26 Guillem Jover  wrote:
> On Tue, 2021-05-25 at 19:43:21 +0200, Andreas Metzler wrote:
[...]
> I'd probably instead make this a versioned Provides, so that the
> transitional package can be removed right away from systems, it does
> not interfere with the transition, and people can switch to the new
> package in parallel w/o disruption.

Hello,

Why not use a versioned Provides *instead* of the dummy package? We have
had these a long time our packagment management system should have 100%
support now.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Planning for libidn shared library version transition

[Simon McVittie]

On Wed, 26 May 2021 at 00:11:29 +0200, Simon Josefsson wrote:
> Andreas Metzler  writes:
> > On 2021-05-24 Simon Josefsson  wrote:
> >> I want to upload a new upstream libidn release into Debian, but upstream
> >> has done a shared library transition.
...
> >>  Package: libidn11-dev
> >> +Section: oldlibs
> >> +Architecture: any
> >> +Depends: libidn-dev, ${misc:Depends}
> >
> > I would use (= ${binary:Version}) for the depends to make sure that
> > libidn11-dev 1.38 together with libidn-dev 1.34 do not fullfill a
> > dependency on libidn11-dev (>= 1.35)

It's better to use (>= ${binary:Version}), or even the hard-coded version
where it became transitional like (>= 1.x.y~), for transitional packages.

This means that when the source package drops the transitional binary
package, anyone who needs to keep it installed for some legacy reason
(such as proprietary or unmaintained software) can keep the transitional
package installed alongside a newer version of the actual library.

> >> +Breaks: libidn11-dev (<< 1.33-4)
> >> +Replaces: libidn11-dev (<< 1.33-4)

If you use 1.33-4~ then it's backports-friendly (1.33-4~bpo11+1 > 1.33.4~).
This also lets you do prereleases with something like
 in an automated way.

smcv

Re: Planning for libidn shared library version transition

[Simon Josefsson]

Andreas Metzler  writes:

> On 2021-05-24 Simon Josefsson  wrote:
>> Hi!  This is for post-bullseye, but I appreciate guidance if anyone has
>> time.  Shared library version transitions trigger uncertainty in me.
>
>> I want to upload a new upstream libidn release into Debian, but upstream
>> has done a shared library transition.
>
> Hello Simon,
>
> I will assume that "shared library transition" means that that libidn
> 1.34 will have a soname bump (libidn12.so) but that the new release is
> API compatible. The following assumes this is true ...

Hi.  Thanks for review!

For all practical matters, yes, it is completely API compatible, the
only thing that changed was this struct (that should never have been in
the public .h anyway) that changed from:

  struct Stringprep_table
  {
Stringprep_profile_steps operation;
Stringprep_profile_flags flags;
const Stringprep_table_element *table;
  };
  typedef struct Stringprep_table Stringprep_profile;

into:

  struct Stringprep_table
  {
Stringprep_profile_steps operation;
Stringprep_profile_flags flags;
const Stringprep_table_element *table;
size_t table_size;
  };
  typedef struct Stringprep_table Stringprep_profile;

Arguably, doing a soname bump for this minor change could probably have
been avoided, but this was back in 2018 and other distributions have
upgraded to newer libidn.  If we revert the soname upstream now, it will
cause a lot of problems for others, so I don't think that is a good
idea.

We could patch upstream in Debian to avoid the soname bump, but that is
rather confusing.  It is good timing now to fix this once bullseye is
released, so I think we should just do that.

>> Bullseye will ship with
>> libidn11-dev instead of libidn-dev too.  Is the first step on the
>> transition to provide a libidn-dev package experimental (and after the
>> release, unstable) and make all reverse build-dependencies use it?
>
> That would delay the whole thing unnecessarily, you would not want to
> artificially make changing all rdeps a pre-dependency of your work
>
> First off if the new version of libidn is API compatible with 1.33 there
> will be no hard /requirement/ for renaming libidn11-dev to libidn12-dev.
> You could have libidn11-dev as a dev-package for libidn12. It is not
> aesthetically pleasing and confusing but would continue to work and has
> the least potential for error. OTOH a soname bump is good time for
> renaming the development package since you will have a new binary
> package (libidn12) and will need to go through NEW processing anyway.
> However you should do soname bump and  dev-package rename in one upload
> (to experimental) instead of beforehand. This ways ftp-master will only
> need to check it once. So you would use this timelime:
>
> 1. Prepare packages of the new upstream targeted for experimental
> (libidn12, libidn-dev, libidn11-dev transition package). Test.
> 2. Upload to experimental.
> 3. Wait for NEW processing
> 4. Do some more tests, once you thinks all rdeps could be built against
> libidn12 you request a transition slot. Please note that at this point
> no source changes to rdeps related to packaging changes needed to happen,
> they can (and should) continue to b-d on libidn11-dev which pulls in the
> libidn-dev.
> 5. debian-release gives you the GO for uploading the package to testing.
> 6. You do this and debian-release triggers rebuilds of all rdeps.
> 7. All rebuilt rdeps and libidn12/libidn-dev/libidn11-dev propagate to
> testing together 10 days later.
> 8. Now you can start submitting bug-reports against rdeps asking for a
> change of the b-d.

That (and your suggestions below) seems good.  I like it better than my
approach.  I'll start to implement this and post a debdiff for review
before actually making the upload into experimental.

/Simon

> [...]
>> A 'git diff' between the version in bullseye and what I want to upload
>> to experimental is shown below, together with debdiff-output between the
>> built packages.
>
>> Generally, does things looks okay?  Specifically, what about the
>> Breaks/Replaces/Conflicts?  The d/changelog entry?  Will the confusing
>> 'Replaces: libidn11-dev' for the libidn11 (!) package in bullseye cause
>> any problem?  Am I right to drop it here?
>
>>  Package: libidn11-dev
>> +Section: oldlibs
>> +Architecture: any
>> +Depends: libidn-dev, ${misc:Depends}
>
> I would use (= ${binary:Version}) for the depends to make sure that
> libidn11-dev 1.38 together with libidn-dev 1.34 do not fullfill a
> dependency on libidn11-dev (>= 1.35)
>
>> +Package: libidn-dev
>>  Section: libdevel
>>  Architecture: any
>>  Depends: libidn11 (= ${binary:Version}), pkg-config, ${misc:Depends}
>> -Conflicts: libidn9-dev
>> +Breaks: libidn11-dev (<< 1.33-4)
>> +Replaces: libidn11-dev (<< 1.33-4)
>> +Provides: libidn11-dev
>>  Multi-Arch: same
>>  Description: Development files for GNU Libidn, an IDN library
>
> You should drop the Provides - It is superfluous since you have got 

Re: Planning for libidn shared library version transition

[Guillem Jover]

On Tue, 2021-05-25 at 19:43:21 +0200, Andreas Metzler wrote:
> On 2021-05-24 Simon Josefsson  wrote:
> > Generally, does things looks okay?  Specifically, what about the
> > Breaks/Replaces/Conflicts?  The d/changelog entry?  Will the confusing
> > 'Replaces: libidn11-dev' for the libidn11 (!) package in bullseye cause
> > any problem?  Am I right to drop it here?
> 
> >  Package: libidn11-dev
> > +Section: oldlibs
> > +Architecture: any
> > +Depends: libidn-dev, ${misc:Depends}
> 
> I would use (= ${binary:Version}) for the depends to make sure that
> libidn11-dev 1.38 together with libidn-dev 1.34 do not fullfill a
> dependency on libidn11-dev (>= 1.35)
> 
> > +Package: libidn-dev
> >  Section: libdevel
> >  Architecture: any
> >  Depends: libidn11 (= ${binary:Version}), pkg-config, ${misc:Depends}
> > -Conflicts: libidn9-dev
> > +Breaks: libidn11-dev (<< 1.33-4)
> > +Replaces: libidn11-dev (<< 1.33-4)
> > +Provides: libidn11-dev
> >  Multi-Arch: same
> >  Description: Development files for GNU Libidn, an IDN library
> 
> You should drop the Provides - It is superfluous since you have got a
> libidn11-dev transition package.

I'd probably instead make this a versioned Provides, so that the
transitional package can be removed right away from systems, it does
not interfere with the transition, and people can switch to the new
package in parallel w/o disruption.

Thanks,
Guillem

Re: Planning for libidn shared library version transition

[Andreas Metzler]

On 2021-05-24 Simon Josefsson  wrote:
> Hi!  This is for post-bullseye, but I appreciate guidance if anyone has
> time.  Shared library version transitions trigger uncertainty in me.

> I want to upload a new upstream libidn release into Debian, but upstream
> has done a shared library transition.

Hello Simon,

I will assume that "shared library transition" means that that libidn
1.34 will have a soname bump (libidn12.so) but that the new release is
API compatible. The following assumes this is true ...

> Bullseye will ship with
> libidn11-dev instead of libidn-dev too.  Is the first step on the
> transition to provide a libidn-dev package experimental (and after the
> release, unstable) and make all reverse build-dependencies use it?

That would delay the whole thing unnecessarily, you would not want to
artificially make changing all rdeps a pre-dependency of your work

First off if the new version of libidn is API compatible with 1.33 there
will be no hard /requirement/ for renaming libidn11-dev to libidn12-dev.
You could have libidn11-dev as a dev-package for libidn12. It is not
aesthetically pleasing and confusing but would continue to work and has
the least potential for error. OTOH a soname bump is good time for
renaming the development package since you will have a new binary
package (libidn12) and will need to go through NEW processing anyway.
However you should do soname bump and  dev-package rename in one upload
(to experimental) instead of beforehand. This ways ftp-master will only
need to check it once. So you would use this timelime:

1. Prepare packages of the new upstream targeted for experimental
(libidn12, libidn-dev, libidn11-dev transition package). Test.
2. Upload to experimental.
3. Wait for NEW processing
4. Do some more tests, once you thinks all rdeps could be built against
libidn12 you request a transition slot. Please note that at this point
no source changes to rdeps related to packaging changes needed to happen,
they can (and should) continue to b-d on libidn11-dev which pulls in the
libidn-dev.
5. debian-release gives you the GO for uploading the package to testing.
6. You do this and debian-release triggers rebuilds of all rdeps.
7. All rebuilt rdeps and libidn12/libidn-dev/libidn11-dev propagate to
testing together 10 days later.
8. Now you can start submitting bug-reports against rdeps asking for a
change of the b-d.

[...]
> A 'git diff' between the version in bullseye and what I want to upload
> to experimental is shown below, together with debdiff-output between the
> built packages.

> Generally, does things looks okay?  Specifically, what about the
> Breaks/Replaces/Conflicts?  The d/changelog entry?  Will the confusing
> 'Replaces: libidn11-dev' for the libidn11 (!) package in bullseye cause
> any problem?  Am I right to drop it here?

>  Package: libidn11-dev
> +Section: oldlibs
> +Architecture: any
> +Depends: libidn-dev, ${misc:Depends}

I would use (= ${binary:Version}) for the depends to make sure that
libidn11-dev 1.38 together with libidn-dev 1.34 do not fullfill a
dependency on libidn11-dev (>= 1.35)

> +Package: libidn-dev
>  Section: libdevel
>  Architecture: any
>  Depends: libidn11 (= ${binary:Version}), pkg-config, ${misc:Depends}
> -Conflicts: libidn9-dev
> +Breaks: libidn11-dev (<< 1.33-4)
> +Replaces: libidn11-dev (<< 1.33-4)
> +Provides: libidn11-dev
>  Multi-Arch: same
>  Description: Development files for GNU Libidn, an IDN library

You should drop the Provides - It is superfluous since you have got a
libidn11-dev transition package.

hth, cu Andreas


-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Planning for libidn shared library version transition

[Simon Josefsson]

mån 2021-05-24 klockan 20:45 +0200 skrev Timo Röhling:
> Hi Simon!
> 
> * Simon Josefsson  [2021-05-24 19:34]:
> > I want to upload a new upstream libidn release into Debian, but
> > upstream
> > has done a shared library transition. 
> You should probably read the Release Team documentation [1] on
> library transitions if you haven't done so already.

Thanks -- I will take a look.

> > Is the first step on the transition to provide a libidn-dev package
> > experimental (and after the release, unstable) and make all reverse
> > build-dependencies use it?
> Generally, it is a good idea to have a libidn-dev package instead of
> libidn11-dev, so +1 on that. It will avoid sourceful uploads of
> reverse-dependencies in the future.

Indeed.

> However, I noticed that Debian ships with the successor library
> libidn2 already, and its homepage says it provides a compatibility
> layer for libidn. So wouldn't it be better to sunset libidn in favor
> of the new version?

It would, but there are still some valid reasons for using libidn
(e.g., stringprep), and I doubt all dependencies will be able to
migrate before bullseye+1.  I will include a comment about this in the
bugs I file, maybe some of the reverse dependencies should just stop
using libidn instead.  Still, they need to be modified anyway.

/Simon



signature.asc
Description: This is a digitally signed message part

Re: Planning for libidn shared library version transition

[Timo Röhling]

Hi Simon!

* Simon Josefsson  [2021-05-24 19:34]:

I want to upload a new upstream libidn release into Debian, but upstream
has done a shared library transition. 

You should probably read the Release Team documentation [1] on library
transitions if you haven't done so already.


Is the first step on the transition to provide a libidn-dev package
experimental (and after the release, unstable) and make all reverse
build-dependencies use it?

Generally, it is a good idea to have a libidn-dev package instead of
libidn11-dev, so +1 on that. It will avoid sourceful uploads of
reverse-dependencies in the future.

However, I noticed that Debian ships with the successor library
libidn2 already, and its homepage says it provides a compatibility
layer for libidn. So wouldn't it be better to sunset libidn in favor
of the new version?

Cheers
Timo

--
⢀⣴⠾⠻⢶⣦⠀   ╭╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling   │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄   ╰╯


signature.asc
Description: PGP signature