[PATCH dpkg 0/3] supporting seemless package renames (dpkg --configure --ignore-not-installed)

[Jonathan Nieder]

I was reading some old material about trouble handling renaming binary
packages in a seemless manner [1] [2].  My main thought: this
shouldn’t be hard to fix.

Indeed, I don’t think it is.  There are two steps:

First, dpkg learns a --ignore-not-installed option so that

dpkg --ignore-not-installed --configure disappeared-package new-version

is treated the same as

dpkg --configure new-version

if disappeared-package is not installed.

Next, apt learns to always use this option when calling
dpkg --configure.

This series takes care of the first step.  It is the logical
continuation of a fix from version 1.10.22, when dpkg learned to
ignore not-installed packages during the configuration pass of a
--install run.

Patches 1 and 2 of this series have nothing to do with that topic.
You can easily omit them.  They just make src/packages.c a little
easier to read; I wrote them while reading through that code.

Patch 3 is the important part: it adds the new option and a
test to demonstrate the problem it fixes.

Thoughts?

Jonathan Nieder (3):
  dpkg: Describe the purpose of packages()
  dpkg: factor out package-listing functions from packages()
  dpkg --configure: optionally tolerate not-installed packages in argv

 configure.ac  |1 +
 debian/changelog  |7 ++
 man/dpkg.1|7 ++
 src/Makefile.am   |2 +
 src/main.c|4 +-
 src/main.h|2 +-
 src/packages.c|  127 +-
 src/test/Makefile.am  |5 +
 src/test/t-disappear-configure.sh |  208 +
 9 files changed, 309 insertions(+), 54 deletions(-)
 create mode 100644 src/test/Makefile.am
 create mode 100755 src/test/t-disappear-configure.sh

[1] http://wiki.debian.org/Renaming_a_Package
[2] http://lists.debian.org/deity/2006/06/msg00059.html


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408113220.ga27...@progeny.tock

[PATCH 1/3] dpkg: Document the purpose of packages()

[Jonathan Nieder]

Since this function is only called through a function pointer,
novices unfamiliar with myopt may have trouble tracking down
its purpose.

Signed-off-by: Jonathan Nieder 
---
 src/packages.c |4 
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/src/packages.c b/src/packages.c
index 0c1a69b..2e3c23a 100644
--- a/src/packages.c
+++ b/src/packages.c
@@ -54,6 +54,10 @@ add_to_queue(struct pkginfo *pkg)
   pkg_queue_push(&queue, pkg);
 }
 
+/*
+ * Act on the packages listed in argv.
+ * cipaction->arg contains the requested action (e.g., --configure)
+ */
 void packages(const char *const *argv) {
   struct pkgiterator *it;
   struct pkginfo *pkg;
-- 
1.7.0.4


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408113323.gb27...@progeny.tock

[PATCH 2/3] dpkg: factor out package-listing functions from packages()

[Jonathan Nieder]

Split packages() into bite-sized pieces.  No functional change
intended.

Signed-off-by: Jonathan Nieder 
---
 src/packages.c |  124 ++-
 1 files changed, 68 insertions(+), 56 deletions(-)

diff --git a/src/packages.c b/src/packages.c
index 2e3c23a..f180434 100644
--- a/src/packages.c
+++ b/src/packages.c
@@ -54,16 +54,72 @@ add_to_queue(struct pkginfo *pkg)
   pkg_queue_push(&queue, pkg);
 }
 
-/*
- * Act on the packages listed in argv.
- * cipaction->arg contains the requested action (e.g., --configure)
- */
-void packages(const char *const *argv) {
+/* Fill queue with pending packages. */
+static void queue_pending(void)
+{
+  struct pkgiterator *it;
+  struct pkginfo *pkg;
+
+  it= iterpkgstart();
+  while ((pkg = iterpkgnext(it)) != NULL) {
+switch (cipaction->arg) {
+case act_configure:
+  if (!(pkg->status == stat_unpacked ||
+pkg->status == stat_halfconfigured ||
+pkg->trigpend_head))
+continue;
+  if (pkg->want != want_install)
+continue;
+  break;
+case act_triggers:
+  if (!pkg->trigpend_head)
+continue;
+  if (pkg->want != want_install)
+continue;
+  break;
+case act_remove:
+case act_purge:
+  if (pkg->want != want_purge) {
+if (pkg->want != want_deinstall) continue;
+if (pkg->status == stat_configfiles) continue;
+  }
+  if (pkg->status == stat_notinstalled)
+continue;
+  break;
+default:
+  internerr("unknown action '%d'", cipaction->arg);
+}
+add_to_queue(pkg);
+  }
+  iterpkgend(it);
+}
+
+/* Fill queue with specified packages. */
+static void queue_specified(const char *const *argv)
+{
   struct pkgiterator *it;
   struct pkginfo *pkg;
   const char *thisarg;
   size_t l;
-  
+
+  while ((thisarg = *argv++) != NULL) {
+pkg= findpackage(thisarg);
+if (pkg->status == stat_notinstalled) {
+  l= strlen(pkg->name);
+  if (l >= sizeof(DEBEXT) && !strcmp(pkg->name+l-sizeof(DEBEXT)+1,DEBEXT))
+badusage(_("you must specify packages by their own names,"
+ " not by quoting the names of the files they come in"));
+}
+add_to_queue(pkg);
+  }
+}
+
+/*
+ * Act on the packages listed in argv.
+ * cipaction->arg contains the requested action (e.g., --configure)
+ */
+void packages(const char *const *argv)
+{
   trigproc_install_hooks();
 
   modstatdb_init(admindir,
@@ -74,59 +130,15 @@ void packages(const char *const *argv) {
   log_message("startup packages %s", cipaction->olong);
 
   if (f_pending) {
-
 if (*argv)
-  badusage(_("--%s --pending does not take any non-option 
arguments"),cipaction->olong);
-
-it= iterpkgstart();
-while ((pkg = iterpkgnext(it)) != NULL) {
-  switch (cipaction->arg) {
-  case act_configure:
-if (!(pkg->status == stat_unpacked ||
-  pkg->status == stat_halfconfigured ||
-  pkg->trigpend_head))
-  continue;
-if (pkg->want != want_install)
-  continue;
-break;
-  case act_triggers:
-if (!pkg->trigpend_head)
-  continue;
-if (pkg->want != want_install)
-  continue;
-break;
-  case act_remove:
-  case act_purge:
-if (pkg->want != want_purge) {
-  if (pkg->want != want_deinstall) continue;
-  if (pkg->status == stat_configfiles) continue;
-}
-if (pkg->status == stat_notinstalled)
-  continue;
-break;
-  default:
-internerr("unknown action '%d'", cipaction->arg);
-  }
-  add_to_queue(pkg);
-}
-iterpkgend(it);
-
+  badusage(_("--%s --pending does not take any non-option arguments"),
+   cipaction->olong);
+queue_pending();
   } else {
-
 if (!*argv)
-  badusage(_("--%s needs at least one package name argument"), 
cipaction->olong);
-
-while ((thisarg = *argv++) != NULL) {
-  pkg= findpackage(thisarg);
-  if (pkg->status == stat_notinstalled) {
-l= strlen(pkg->name);
-if (l >= sizeof(DEBEXT) && 
!strcmp(pkg->name+l-sizeof(DEBEXT)+1,DEBEXT))
-  badusage(_("you must specify packages by their own names,"
-   " not by quoting the names of the files they come in"));
-  }
-  add_to_queue(pkg);
-}
-
+  badusage(_("--%s needs at least one package name argument"),
+   cipaction->olong);
+queue_specified(argv);
   }
 
   ensure_diversions();
-- 
1.7.0.4


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408113350.gc27...@progeny.tock

[PATCH 3/3] dpkg --configure: optionally tolerate not-installed packages in argv

[Jonathan Nieder]

When renaming a package "pkg-old" to "pkg-new", it would be nice if
upgrading pkg-old automatically ensures that pkg-new is installed and
pkg-old removed.  In fact, Debian policy has provided a facility that
should allow this since early on: if the new version of pkg-old
depends on a version of pkg-new that replaces all its files, it will
"disappear", providing the desired seemless upgrade.

dpkg learned to support this with version 1.10.22, with the fix to
bug #202997 (previously, dpkg would try to configure the disappeared
package and error out because it was confused).

Unfortunately, this was not enough to fix the problem for upgrades
driven by APT.  APT will unpacks and configures packages in separate
dpkg runs.  But unlike dpkg, APT does not have the information to
quickly tell whether a package it plans to configure is not installed.
So it is doomed to always be confused.

Fix this by providing a new --ignore-not-installed option for APT
to use.  If this option is supplied with --configure, any packages
on the command line that are not installed will be ignored on the
assumption that they recently disappeared.

Based on suggestions by Daniel Kobras and Ian Jackson.

Reference: <17545.29786.400376.667...@davenant.relativity.greenend.org.uk>
Signed-off-by: Jonathan Nieder 
---
That’s the end of the series.  Thanks for reading.

Yours,
Jonathan

 configure.ac  |1 +
 debian/changelog  |7 ++
 man/dpkg.1|7 ++
 src/Makefile.am   |2 +
 src/main.c|4 +-
 src/main.h|2 +-
 src/packages.c|7 ++
 src/test/Makefile.am  |5 +
 src/test/t-disappear-configure.sh |  208 +
 9 files changed, 241 insertions(+), 2 deletions(-)
 create mode 100644 src/test/Makefile.am
 create mode 100755 src/test/t-disappear-configure.sh

diff --git a/configure.ac b/configure.ac
index 8b7e213..1b45d6f 100644
--- a/configure.ac
+++ b/configure.ac
@@ -148,6 +148,7 @@ AC_CONFIG_FILES([ Makefile
  scripts/Makefile
  scripts/po/Makefile.in
  src/Makefile
+ src/test/Makefile
  utils/Makefile ])
 AC_CONFIG_HEADERS([config.h])
 AC_OUTPUT
diff --git a/debian/changelog b/debian/changelog
index adaa95f..c84a224 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -15,6 +15,13 @@ dpkg (1.15.6.2) UNRELEASED; urgency=low
   * Improve explanation of --all option in dpkg-parsechangelog(1). Thanks to
 Jari Aalto. Closes: #575706
 
+  [ Jonathan Nieder ]
+  * dpkg now supports an --ignore-not-installed option to silently tolerate
+requests to configure a package that is not installed.
+Frontends should use this option when configuring packages they just
+unpacked to avoid confusion from disappearing transitional packages.
+This is the logical continuation of #202997.
+
   [ Updated dpkg translations ]
   * German (Sven Joachim).
 
diff --git a/man/dpkg.1 b/man/dpkg.1
index 0f6b337..5530c31 100644
--- a/man/dpkg.1
+++ b/man/dpkg.1
@@ -449,6 +449,13 @@ Install a package even if it fails authenticity check.
 Ignore dependency-checking for specified packages (actually, checking is
 performed, but only warnings about conflicts are given, nothing else).
 .TP
+\fB\-\-ignore\-not\-installed
+Silently tolerate attempts to configure a package that is
+not\-installed.  Front-ends should use this option when unpacking
+and configuring packages in separate steps, since a package could
+\fIdisappear\fP before there is a chance to configure it if another
+package replaces all its files.
+.TP
 \fB\-\-new\fP, \fB\-\-old\fP
 Select new or old binary package format. This is a \fBdpkg\-deb\fP(1)
 option.
diff --git a/src/Makefile.am b/src/Makefile.am
index b3f2fe5..f680b44 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -1,5 +1,7 @@
 ## Process this file with automake to produce Makefile.in
 
+SUBDIRS = test
+
 localedir = $(datadir)/locale
 pkgconfdir = $(sysconfdir)/@PACKAGE@
 
diff --git a/src/main.c b/src/main.c
index 9f3a5b7..ccee956 100644
--- a/src/main.c
+++ b/src/main.c
@@ -135,6 +135,7 @@ usage(const struct cmdinfo *ci, const char *value)
 "  --log=   Log status changes and actions to .\n"
 "  --ignore-depends=,...\n"
 " Ignore dependencies involving .\n"
+"  --ignore-not-installed Tolerate attempt to configure disappeared 
package.\n"
 "  --force-...Override problems (see --force-help).\n"
 "  --no-force-...|--refuse-...\n"
 " Stop when problems encountered.\n"
@@ -169,7 +170,7 @@ const char printforhelp[]= N_(
 
 const struct cmdinfo *cipaction = NULL;
 int f_pending=0, f_recursive=0, f_alsoselect=1, f_skipsame=0, f_noact=0;
-int f_autodeconf=0, f_nodebsig=0;
+int f_autodeconf=0, f_nodebsig=0, f_ignoreuninst=0;
 int f_triggers = 0;
 unsigned long f_debug=0;

[PATCH/RFC] APT: do not error out for packages that disappear during installation

[Jonathan Nieder]

Hi again,

Here’s an approximation to the corresponding APT change.  Untested.
I hope it is clear enough to get the idea across.

Ciao,
Jonathan

=== modified file 'apt-pkg/deb/dpkgpm.cc'
--- apt-pkg/deb/dpkgpm.cc   2010-03-26 15:04:49 +
+++ apt-pkg/deb/dpkgpm.cc   2010-04-08 11:45:33 +
@@ -844,6 +844,8 @@
 break;
 
 case Item::Configure:
+Args[n++] = "--ignore-not-installed"
+Size += strlen(Args[n-1]);
 Args[n++] = "--configure";
 Size += strlen(Args[n-1]);
 break;

=== modified file 'debian/changelog'
--- debian/changelog2010-03-26 15:04:49 +
+++ debian/changelog2010-04-08 11:50:25 +
@@ -8,6 +8,14 @@
   * Add "manpages-pl (<< 20060617-3~)" to avoid file conflicts with
 that package that was providing some manpages for APT utilities.
 
+  [ Jonathan Nieder ]
+  * apt-pkg/deb/dpkgpm.cc:
+- Use the new --ignore-not-installed option of dpkg to ensure
+  requests to configure a package that disappeared are tolerated.
+  This way, APT can upgrade and remove a package renamed using the
+  method described at http://wiki.debian.org/Renaming_a_Package
+  without errors.
+
   [ Julian Andres Klode ]
   * cmdline/apt-cache.cc:
 - Change behavior of showsrc to match the one of show (Closes: #512046).

=== modified file 'debian/control'
--- debian/control  2010-03-19 03:19:15 +
+++ debian/control  2010-04-08 11:51:59 +
@@ -11,7 +11,7 @@
 
 Package: apt
 Architecture: any
-Depends: ${shlibs:Depends}, debian-archive-keyring, ${misc:Depends}
+Depends: ${shlibs:Depends}, debian-archive-keyring, dpkg (>= 1.15.6.2), 
${misc:Depends}
 Replaces: libapt-pkg-doc (<< 0.3.7), libapt-pkg-dev (<< 0.3.7), manpages-pl 
(<< 20060617-3~)
 Provides: ${libapt-pkg:provides}
 Conflicts: python-apt (<< 0.7.93.2~)


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408115820.ga27...@progeny.tock

Re: [PATCH dpkg 0/3] supporting seemless package renames (dpkg --configure --ignore-not-installed)

[Eugene V. Lyubimkin]

On Thursday 08 April 2010 14:32:20 Jonathan Nieder wrote:
> I was reading some old material about trouble handling renaming binary
> packages in a seemless manner [1] [2].  My main thought: this
> shouldn’t be hard to fix.

Hi Jonathan,

Firstly, I'd ask you next time you CC de...@l.d.o, CC c...@packages.debian.org 
as well.

Secondly, AFAIK, Guillem Jover some time ago added a notice of disappeared 
packages to --status-fd dpkg output, so theoretically any high-level package 
manager can use it (practically, I don't know whether apt uses/will use it or 
not, but cupt does not, due to to 'Thirdly' below).

Thirdly, IMO this 'disappear' thing is a design flaw in dpkg/policy:

- Policy says that disappearted packages will not receive 'prerm' signal. 
Then, what's the point of having it when maintainer cannot guarantee it will 
be called?
- When you upgrades your system by some high-level package manager it usually 
says you that 'packages oldpkg and newpkg will be upgraded' (or 'newpkg will 
be installed and oldpkg is upgraded'). Once oldpkg gets suddenly dropped, it's 
inconvenient (at least) to high-level package managers, may confuse users, 
and, in case, just a lie. If the package manager says it will upgraded, it 
should be upgraded and not removed.
- The use-case for 'fast transition' doesn't look good to me, because when 
user will try to install the package he/she will see 'the package does not 
exist' instead of installing newpkg and transitional oldpkg.


Thanks for attention.

--
Eugene V. Lyubimkin


--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/201004081508.57002.jackyf.de...@gmail.com

Re: supporting seemless package renames

[Jonathan Nieder]

Hi,

Eugene V. Lyubimkin wrote:

> Firstly, I'd ask you next time you CC de...@l.d.o, CC 
> c...@packages.debian.org 
> as well.

CC added.

> Secondly, AFAIK, Guillem Jover some time ago added a notice of disappeared 
> packages to --status-fd dpkg output, so theoretically any high-level package 
> manager can use it (practically, I don't know whether apt uses/will use it or 
> not, but cupt does not, due to to 'Thirdly' below).

Is there a reason to prefer this approach over --ignore-not-installed?
I think simplest is best.

> - Policy says that disappearted packages will not receive 'prerm' signal. 
> Then, what's the point of having it when maintainer cannot guarantee it will 
> be called?

Hmm, sounds like a bug in policy.  I think disappeared packages ought
to receive prerm.  Would you like to raise the issue, or should I?

In the short term, maintainers would have to work around this in the
preinst of the replacing package.

> - When you upgrades your system by some high-level package manager it usually 
> says you that 'packages oldpkg and newpkg will be upgraded' (or 'newpkg will 
> be installed and oldpkg is upgraded'). Once oldpkg gets suddenly dropped, 
> it's 
> inconvenient (at least) to high-level package managers, may confuse users, 
> and, in case, just a lie. If the package manager says it will upgraded, it 
> should be upgraded and not removed.

It would be better if the package manager could say ‘the oldpkg
package will renamed by this upgrade; its new name is newpkg’.
Unfortunately, in the current control file format, I don’t know a way
to express that.  Maybe debtags can help (role::dummy).

The main advantage to the disappearing package trick is to avoid the
confusing situation of asking the operator to do something that would
be very easy to do mechanically (removing the transitional packages).

> - The use-case for 'fast transition' doesn't look good to me, because when 
> user will try to install the package he/she will see 'the package does not 
> exist' instead of installing newpkg and transitional oldpkg.

I think you misunderstood what I meant by seemless.  I think the oldpkg
package should still stay available for at least as long as it already does;
it just should not be installed on anyone’s system any more.

So the person will see 'the package exists, but you do not have it
installed.

Thanks for the comments.
Jonathan


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408125219.gb27...@progeny.tock

Re: supporting seemless package renames

[Eugene V. Lyubimkin]

On Thursday 08 April 2010 15:52:19 Jonathan Nieder wrote:
> Eugene V. Lyubimkin wrote:
> > Firstly, I'd ask you next time you CC de...@l.d.o, CC
> > c...@packages.debian.org as well.
> 
> CC added.
Thanks.

> > Secondly, AFAIK, Guillem Jover some time ago added a notice of
> > disappeared packages to --status-fd dpkg output, so theoretically any
> > high-level package manager can use it (practically, I don't know whether
> > apt uses/will use it or not, but cupt does not, due to to 'Thirdly'
> > below).
> 
> Is there a reason to prefer this approach over --ignore-not-installed?
> I think simplest is best.
Your solution is more convenient, indeed. If it gets accepted, I would use it 
rather than --status-fd one. So OK, nevermind that argument. But I'd prefer to 
not have disappeared packages at all, even if this is unlikely to happen.

> > - Policy says that disappearted packages will not receive 'prerm' signal.
> > Then, what's the point of having it when maintainer cannot guarantee it
> > will be called?
> 
> Hmm, sounds like a bug in policy.  I think disappeared packages ought
> to receive prerm.  Would you like to raise the issue, or should I?
Disappeared packages do not receive prerm because of complicated dpkg logic 
behind it (KISS, KISS...), by the time of dpkg decides that package will 
disappear the files of the package may be deleted already. Given that I suppose 
that this behavior is intentional. However if you as dpkg contributor want to 
change it - I will support you.


> In the short term, maintainers would have to work around this in the
> preinst of the replacing package.
I don't think it worth the trouble, but YMMV.

> > - When you upgrades your system by some high-level package manager it
> > usually says you that 'packages oldpkg and newpkg will be upgraded' (or
> > 'newpkg will be installed and oldpkg is upgraded'). Once oldpkg gets
> > suddenly dropped, it's inconvenient (at least) to high-level package
> > managers, may confuse users, and, in case, just a lie. If the package
> > manager says it will upgraded, it should be upgraded and not removed.
> 
> It would be better if the package manager could say ‘the oldpkg
> package will renamed by this upgrade; its new name is newpkg’.
> Unfortunately, in the current control file format, I don’t know a way
> to express that.  Maybe debtags can help (role::dummy).
I would argue that package manager should not consider debtags as a source of 
critical control information. And if that information is going to be added, 
then there is no point of having '--ignore-not-installed', right? :)

> The main advantage to the disappearing package trick is to avoid the
> confusing situation of asking the operator to do something that would
> be very easy to do mechanically (removing the transitional packages).
Removing transitional package is not so important task. Nothing changed if it 
stays in the system, apart of list of installed packages. Seamless upgrades 
(in the meaning of dpkg not erroring out in the middle of) are IMO much more 
important.

> > - The use-case for 'fast transition' doesn't look good to me, because
> > when user will try to install the package he/she will see 'the package
> > does not exist' instead of installing newpkg and transitional oldpkg.
> 
> I think you misunderstood what I meant by seemless.  I think the oldpkg
> package should still stay available for at least as long as it already
>  does; it just should not be installed on anyone’s system any more.
> 
Ah, OK then.


--
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/201004081618.09821.jackyf.de...@gmail.com

Re: supporting seemless package renames

[Jonathan Nieder]

Eugene V. Lyubimkin wrote:
> On Thursday 08 April 2010 15:52:19 Jonathan Nieder wrote:

>> Hmm, sounds like a bug in policy.  I think disappeared packages ought
>> to receive prerm.  Would you like to raise the issue, or should I?
>
> Disappeared packages do not receive prerm because of complicated dpkg logic 
> behind it (KISS, KISS...), by the time of dpkg decides that package will 
> disappear the files of the package may be deleted already. Given that I 
> suppose 
> that this behavior is intentional.

Ah, so the problem is that if prerm errors out there is no way to
cancel the removal?  And if I understand correctly, you are suggesting
to run prerm before unpacking the replacing package to fix this.  For
what it’s worth, what I was thinking of before was to run prerm but
resign to ignoring errors from it.  Your idea is interesting; I will
think about it.

>> It would be better if the package manager could say ‘the oldpkg
>> package will renamed by this upgrade; its new name is newpkg’.
>> Unfortunately, in the current control file format, I don’t know a way
>> to express that.  Maybe debtags can help (role::dummy).
>
> I would argue that package manager should not consider debtags as a source of 
> critical control information.

Of course --- I only meant for the sake of displaying what is about
to happen, not controlling the package manager’s behavior.  Still, the
suggestion was ugly, and I am happy to discard it.

I have to go now, but I can return to this topic tonight.  Thanks for
the food for thought.

Ciao,
Jonathan


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408134357.ga31...@progeny.tock

Re: supporting seemless package renames

[Eugene V. Lyubimkin]

Jonathan Nieder wrote:
> Eugene V. Lyubimkin wrote:
>> On Thursday 08 April 2010 15:52:19 Jonathan Nieder wrote:
> 
>>> Hmm, sounds like a bug in policy.  I think disappeared packages ought
>>> to receive prerm.  Would you like to raise the issue, or should I?
>> Disappeared packages do not receive prerm because of complicated dpkg logic 
>> behind it (KISS, KISS...), by the time of dpkg decides that package will 
>> disappear the files of the package may be deleted already. Given that I 
>> suppose 
>> that this behavior is intentional.
> 
> Ah, so the problem is that if prerm errors out there is no way to
> cancel the removal?  And if I understand correctly, you are suggesting
> to run prerm before unpacking the replacing package to fix this.  For
> what it’s worth, what I was thinking of before was to run prerm but
> resign to ignoring errors from it.  Your idea is interesting; I will
> think about it.

Eeeh, well, I didn't quite understand you thoughts here. The point is, as I
understand, dpkg doesn't know will the package be removed or not until its
last file is replaced, and when it is it's too late to run prerm. Anyway, if
you find something else to think out of my words, it's fine :)

-- 
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
C++/Perl developer, Debian Developer


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4bbde6dd.6080...@gmail.com

Re: Reimplement mksplit.pl in C

[Guillem Jover]

Hi Marcelo!

On Fri, 2010-04-02 at 22:55:38 -0600, Marcelo E. Magallon wrote:
>  somewhere I read that a reimplementation of mksplit.pl in C was
>  needed in order to remove a dependency of dpkg on Perl, but I
>  can't find the reference anymore.

The main reasons for this are to not be subject to the perl upgrade
problems as seen in 489132, to couple those perl programs more tightly
with the rest of the C parts, to be able to reuse parts of the code,
for example there's programs that could benefit from a shared library
for update-alternativesy, and to be able to remove perl-base from
essential some day which would make life for embedded distros easier.

There's other small programs in essential which would be nice to get
rewritten [0], another problematic package is adduser, although right
now I think most of the missing features have been implemented directly
in the shadow package.

  [0] 
  (Err, yeah should have finished them long time ago...)

>  Anyways, attached is a patch to do just that.  It's implemented
>  against the master Git branch.  As indicated in the patch, this
>  is a straight translation and it would like to have some
>  refactoring love.  I did compare the output with the Perl
>  version in several test cases and it's identical.

Ouch :(, first of all thanks for the patches, although I already had
this written in C integrated directly in dpkg-split in my pu/c-rewrite
branch [1], although not yet merged, there were still some rough egdes
I wanted to polish first. And second sorry for the duplicated work. I
thought I had written this down on the wiki at least, but it seems not,
I might have commented it on the list and on IRC. Will amend the wiki
right now, and will take a look at the patches later to see what can
be reused.

  [1] 

thanks,
guillem


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100408165754.ga17...@gaara.hadrons.org

Re: [PATCH dpkg 0/3] supporting seemless package renames (dpkg --configure --ignore-not-installed)

[Guillem Jover]

Hi!

On Thu, 2010-04-08 at 15:08:56 +0300, Eugene V. Lyubimkin wrote:
> On Thursday 08 April 2010 14:32:20 Jonathan Nieder wrote:
> > I was reading some old material about trouble handling renaming binary
> > packages in a seemless manner [1] [2].  My main thought: this
> > shouldn’t be hard to fix.

> Secondly, AFAIK, Guillem Jover some time ago added a notice of
> disappeared packages to --status-fd dpkg output, so theoretically any
> high-level package manager can use it (practically, I don't know
> whether apt uses/will use it or not, but cupt does not, due to to
> 'Thirdly' below).

For reference, this was due to bug 537338. I've not checked but I'd
expect this would imply only few lines of code on the apt side, just
removing the just disappeared package from the to be configured queue.

> Thirdly, IMO this 'disappear' thing is a design flaw in dpkg/policy:

With this I disagree and I think it's a nice and useful feature to have.

> - Policy says that disappearted packages will not receive 'prerm'
> signal. Then, what's the point of having it when maintainer cannot
> guarantee it will be called?

Regardless of it being possible to call prerm by making the code in
dpkg more complex/intelligent, the thing is if it would be the correct
thing to do. Something to consider is that a disappearing package is
just one for which its files have been completely taken over by another
package, so arguably its contents do not get removed, they just change
ownership, and no maintainer scripts do get called either on
non-disappeared replaced files. So as I see it, the fact the the postrm
gets called is more to notify that the package state is changing and it's
going away (for which it might need to do additional cleanup) than that
the files got removed (which didn't).

The usual code run at prerm handles files getting removed which does not
apply here, as no files have possibly gotten removed. But if there was a
demonstrable need to run prerm script in this situation, I'd not see any
problem in an evaluation on adding such call.

> - When you upgrades your system by some high-level package manager it
> usually says you that 'packages oldpkg and newpkg will be upgraded' (or
> 'newpkg will be installed and oldpkg is upgraded'). Once oldpkg gets
> suddenly dropped, it's inconvenient (at least) to high-level package
> managers, may confuse users, and, in case, just a lie. If the package
> manager says it will upgraded, it should be upgraded and not removed.

So even if it might be nice for the high-level front-ends to know before
hand what will happen during a dpkg run, the fact is, it cannot be known
and the front-end should be prepared to cope with unexpected state
changes. There's more to the installation/removal process than just
the metadata, there's the maintainer scripts which can fail at any
point, there's file conflicts, there's triggers, etc. And then there's
disappearing packages (even w/o an explicit Replaces).

The main reasons I chose to notify the front-ends via status-fd is so
that we don't need to add yet another force option which in this case
needs to be used always, and because it makes it explicit so that the
front-ends know exactly *why* the package is not there anymore, and they
can correct their view of the world accordingly.

> - The use-case for 'fast transition' doesn't look good to me, because
> when user will try to install the package he/she will see 'the package
> does not exist' instead of installing newpkg and transitional oldpkg.

I don't see the point of leaving cruft behind if the tools can
automatically take care of completely replacing a package with another
one. I don't see the problem with the package not being installed any
longer, that's exactly what was intended to happen, in addition dpkg
should have notified via stdout and status-fd of what has happened,
and I'd assume the package description would state the package is
dummy/transitional/etc, so the situation should be pretty clear for
the user. Also dpkg will not disappear a package currently being
depended on.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100409025754.ga28...@gaara.hadrons.org

Re: [PATCH dpkg 0/3] supporting seemless package renames (dpkg --configure --ignore-not-installed)

[Jonathan Nieder]

Hi,

[quoting out of order for convenience]

Guillem Jover wrote:

> I've not checked but I'd
> expect this would imply only few lines of code on the apt side, just
> removing the just disappeared package from the to be configured queue.

>From my reading of pkgDPkgPM::ProcessDpkgStatusLine, APT does not
care much about statusfd.  It is treated as just some information
about progress to pass to the end-user.

Strangely enough, helping the end-user in this way involves
remembering the state of each package being operated on.  When all is
going to plan, APT can report that a package is

 half-installed --- Preparing foo
 unpacked --- Unpacking foo
 unpacked --- Preparing to configure foo
 half-configured --- Configuring foo
 installed --- Installed foo

And there is a map from package names to integers indicating where in
such a sequence each particular package is.

Currently, this map does not affect APT’s order of operations.
Instead, the list of operations is planned in advance and followed
rigidly, regardless of the situation in the field.

APT experts: would it make sense to build in additional intelligence
of this kind?  In other words, would it be safe and worthwhile to
teach pkgDPkgPM::Go() to use the PackageOps map to skip a request to
configure a package if it has disappeared?

> There's more to the installation/removal process than just
> the metadata, there's the maintainer scripts which can fail at any
> point, there's file conflicts, there's triggers, etc.

AFAIK both APT’s and cupt’s approach is rather to hope everything goes
according to plan and either do as much as possible or stop at the
first error.

A high-level package manager that collaborates with the user to deal
appropriately with problems as they come might be a lovely thing.
apt-get uses a simpler approach: to fix problems, you run it again.

One final thought.

dpkg itself does not use the kind of intelligence you are talking
about to handle disappearing packages.  It simply assumes that during
an --install run, any package it is trying to configure that is not
installed must have disappeared.

The idea of the proposed patch is to allow front-ends to make the same
assumption.

> For reference, this was due to bug 537338.

Thanks for the pointer --- that was very helpful.

Cheers,
Jonathan

Rough patch teaching APT to report "processing: disappear" lines:

=== modified file 'apt-pkg/deb/dpkgpm.cc'
--- apt-pkg/deb/dpkgpm.cc   2010-03-26 15:04:49 +
+++ apt-pkg/deb/dpkgpm.cc   2010-04-09 05:07:54 +
@@ -50,6 +50,7 @@
 std::make_pair("configure", N_("Configuring %s")),
 std::make_pair("remove",N_("Removing %s")),
 std::make_pair("purge",N_("Completely removing %s")),
+std::make_pair("disappear", N_("Noting disappearance of %s")),
 std::make_pair("trigproc",  N_("Running post-installation trigger %s"))
   };
 
@@ -407,7 +408,8 @@
   'processing: install: pkg'
   'processing: configure: pkg'
   'processing: remove: pkg'
-  'processing: purge: pkg' - but for apt is it a ignored "unknown" action
+  'processing: purge: pkg'
+  'processing: disappear: pkg'
   'processing: trigproc: trigger'

*/


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100409051011.ga9...@progeny.tock

prerm for disappearing packages?

[Jonathan Nieder]

Hi again,

Guillem Jover wrote:

> Regardless of it being possible to call prerm by making the code in
> dpkg more complex/intelligent, the thing is if it would be the correct
> thing to do.

Yes, this is a useful question.  Thank you for bringing it up.

First note that I don’t think this problem would come up often in practice.
Usually the package being replaced is an empty or almost-empty package, and
it is rare for such a package to need a prerm.

Most prerm scripts look something like this [1], but simpler.

#!/bin/sh
set -e
pkg=me

if type update-python-modules >/dev/null 2>&1
then
update-python-modules -c "$pkg.private"
fi
if test -x /usr/lib/emacsen-common/emacs-package-remove
then
/usr/lib/emacsen-common/emacs-package-remove "$pkg"
fi
if test -x "/etc/init.d/$pkg"
then
if test -x "$(which invoke-rc.d 2>/dev/null)"
then
invoke-rc.d "$pkg" stop || :
else
"/etc/init.d/$pkg" stop || :
fi
fi
if test -L "/usr/doc/$pkg"
then
rm -f "/usr/doc/$pkg"
fi
case "$1" in
remove|deconfigure)
update-alternatives --remove "$pkg" /usr/bin/whatever
esac

> Something to consider is that a disappearing package is
> just one for which its files have been completely taken over by another
> package, so arguably its contents do not get removed, they just change
> ownership, and no maintainer scripts do get called either on
> non-disappeared replaced files.

prerm is used for upgrades, too.

> But if there was a
> demonstrable need to run prerm script in this situation, I'd not see any
> problem in an evaluation on adding such call.

Waiting until it comes up does make some sense.

> Also dpkg will not disappear a package currently being
> depended on.

Ah, I didn’t realize this!  That is very comforting to know and should
be helpful for transitions.  It requires some front-end support to
work well, though: an obsolete package and any package that depends on
it have to be installed in a single dpkg run.

I will try it out.

Thanks,
Jonathan

[1] Except bash.prerm --- I don’t know what’s going on with that one.
automake and bsd-mailx do not remove alternatives when deconfiguring, but
I suspect they should.  byacc removes alternatives on failed-upgrade.
I stopped reading there.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100409061453.ga9...@progeny.tock