Suggested improvements for handling Architecture independent packages

[Julian Andres Klode]

Dear readers:

I would like to suggest two major improvements related to packages which are
Architecture: all.

The first thing I want to suggest is the handling of dependencies. When building
a package, you can use Depends: package [arch1 arch2] which means that it
depends on package only on the architectures arch1 and arch2. But this only
works for architecture dependent packages. Therefore, I would like to not
process this command during the build-time, but do it at the installation 
time.

This is especially useful for recommends, since all recommends have to be
available. Without it, the best way is to only suggest the package.

One could also use package | not+ARCHITECTURE, but this results in half-broken
dependencies. And it's more complicated.

I suggested this first in Bug#436733 [1]

The other suggestion [2] is to add a field called Install-Architecture to the
control file, listing architectures for which this package should be available.
Another idea is to use Architecture: all [i386 amd64 ppc], which seems to be
better [3].

It would be great to add the needed functionality before the release of Lenny,
so we can start using it in Lenny+1.

The second suggested improvement could be used before the release of lenny,
because it would only require changes to dak (and maybe dpkg-dev, lintian). dak
would parse the value of Architectures to check for the architectures listed
in [i386 amd64 ppc].

When the package is added to the Packages file, the field gets changed to
Architecture: all. This would be the easiest way.

Regards,

Julian

[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=436733
[2] http://lists.debian.org/debian-devel/2008/02/msg00045.html
[3] http://lists.debian.org/debian-devel/2008/02/msg00355.html
-- 
Julian Andres Klode, Fellow of the Free Software Foundation Europe
 Debian Maintainer | Developer | Ubuntu Member

try Debian: http://www.debian.org/ | my site: http://jak-linux.org/
jabber: [EMAIL PROTECTED] | IRC: juliank (FreeNode, OFTC)
languages: German  | English



signature.asc
Description: OpenPGP digital signature

Re: [buildd-tools-devel] Bug#542843: Does not include orig.tar.gz tarball in changes file.

[Julian Andres Klode]

reassign 542843 dpkg-dev
thanks

Am Dienstag, den 02.02.2010, 17:38 +0100 schrieb Philipp Kern:
 On Tue, Feb 02, 2010 at 05:23:07PM +0100, Julian Andres Klode wrote:
  On Sun, Sep 13, 2009 at 11:05:33AM -0400, Andres Mejia wrote:
   Just tested 'sbuild -As' and couldn't reproduce the problem. The 
   orig.tar.gz was 
   included in the changes file.
  See the attached build log and changes as a proof.
 
 `--force-orig-source'
 
  sbuild (Debian sbuild) 0.59.0 (02 Aug 2009) on hp
  
  ╔══╗
  ║ computer-janitor 1.14.1-1 (amd64)  02 Feb 2010 
  15:59 ║
  ╚══╝
 
 I don't think that's sbuild's fault but rather dpkg's.  I suspect because
 there's most likely a revision  1.14.1-1 and  1.14.1 with an Ubuntu 
 version,
 no?  I, as dpkg, would then assume that the source for 1.14.1 is already
 uploaded.

Reassigning this to dpkg-dev. In my opinion, the .orig.tar.* should
always be included in a -1 release, even if there has already been an
-0ubuntu1 release.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.



-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: [PATCH dpkg 0/3] supporting seemless package renames (dpkg --configure --ignore-not-installed)

[Julian Andres Klode]

On Fri, Apr 09, 2010 at 06:32:34PM +0300, Eugene V. Lyubimkin wrote:
 This is a part I can agree with, though, stop. Hah. Correct me if I am wrong:
 new package got installed as a dependency of transitional package. So, it's
 automatically installed. Now, after upgrade the transitional package is
 removed, new package is alone and will be removed on next front-end run. Nice.
 Am I wrong?
Not if ftp-master creates a 'metapackages' section (#574851) in which
we can put transitional packages. The dependencies of packages within
this section are not marked as automatically installed (or maybe they
are currently, as APT::Never-MarkAuto-Sections:: is empty [#431737]).

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20100409175734.ga3...@debian.org

Re: Bug#598922: apt-cache showsrc prints too many spaces

[Julian Andres Klode]

reassign 598922 dpkg-dev
thanks

On So, 2010-10-03 at 11:10 +0200, Cyril Brulebois wrote:
 Package: apt
 Version: 0.8.5
 Severity: normal
 
 Hi,
 
 one can span fields like Uploaders on multiple lines, using several
 spaces to align everything, which is what evolution-mapi people did:
 | Uploaders: Heikki Henriksen heik...@gmail.com,
 |Lawrence Walton lawre...@the-penguin.otak.com,
 |Loic Minier l...@dooz.org,
 |Oystein Gisnas oyst...@gisnas.net,
 |Josselin Mouette j...@debian.org,
 |Jelmer Vernooij jel...@debian.org,
 |Yves-Alexis Perez cor...@debian.org,
 |David Weinehall t...@debian.org
 
 But it looks like apt-cache showsrc almost display them as-is, that is
 just stripping newlines:
 | Uploaders: Heikki Henriksen heik...@gmail.com,   Lawrence Walton 
 lawre...@the-penguin.otak.com,   Loic Minier l...@dooz.org,   
 Oystein Gisnas oyst...@gisnas.net,   Josselin Mouette 
 j...@debian.org,   Jelmer Vernooij jel...@debian.org,   
 Yves-Alexis Perez cor...@debian.org,   David Weinehall 
 t...@debian.org
 

The newlines are stripped by dak or dpkg, they are not in the Sources
file and APT just displays the sources entry as it. Furthermore, Policy
does not state that whitespace should be ignored, only the newlines; so
this behavior is correct.

The following steps seem to be needed to change this:
 1. Change Policy 5.6.3 to ignore multiple whitespace after newline
 2. Strip the whitespace in dpkg

But as this is not an APT bug, I am reassigning it to dpkg-dev and let
them decide what to do next.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.



-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/128616.2549.10.ca...@jak-thinkpad

dpkg: debconf for conffiles? (was: Re: Bug#606025: packagekit: Does not support conffile)

[Julian Andres Klode]

On Sun, 2010-12-05 at 17:44 +0100, Julian Andres Klode wrote:
 Package: packagekit
 Version: 0.6.8-2
 Severity: normal
 
 In addition to the debconf support, PackageKit must also
 support conffiles.
Dear dpkg maintainers,

for PackageKit support we need a way to handle conffiles. PackageKit
allows questions via debconf.

We were wondering whether dpkg could get support for using debconf to
ask conffile questions, seanius posted an initial patch in 2008[1], but
as far as I can tell, there was no further activity.

[1] http://lists.debian.org/debian-dpkg/2008/03/msg00163.html

Keep me and everyone else in CC/To.
-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.



-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1307036477.8715.4.camel@jak-thinkpad

Re: Multiarch interfaces: print foreign arches, pkgname I/O

[Julian Andres Klode]

On Tue, Dec 13, 2011 at 09:19:55AM +0100, Guillem Jover wrote:
 On Mon, 2011-12-12 at 18:14:15 +0100, David Kalnischkies wrote:
  Beside that i wonder which --force flag this should be, given that it
  removes packages while it was requested to install others.
  In a way, the old architecture is disappearing, but there are no
  Replaces (not even implicit) in place to allow that and the implicit
  Conflicts wouldn't allow a seamless disappearing anyway.
  So implementing this sounds for me a bit like black voodoo…
 
 This will be treated like a normal upgrade, the only distinction will
 be that the architecture of the installed package changes, like it
 currently happens on --force-architecture (but w/o the need of the
 option), and only one installed instance can present for this to
 happen.

And that clashes with the view APT has on the universe; pkg:a and
pkg:b are distinct packages of a group pkg. So for that to work
correctly, APT would need to add some kind of implicit 
Replaces: pkg:other
to the packages. At least IIRC.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111213205107.ga25...@debian.org

Re: Multiarch interfaces: print foreign arches, pkgname I/O

[Julian Andres Klode]

On Thu, Dec 15, 2011 at 02:02:36AM +0100, Guillem Jover wrote:
 Hi!
 
 On Tue, 2011-12-13 at 23:46:35 +0100, David Kalnischkies wrote:
  On Tue, Dec 13, 2011 at 08:29, Guillem Jover guil...@debian.org wrote:
   On Mon, 2011-12-12 at 18:15:12 +0100, David Kalnischkies wrote:
   dpkg --remove libc6 # removing libc6:i386 and libc6:amd64
   ?
  
   Users will love you for this, given that it is completely inconsistent 
   with
   what front-ends will understand if the architecture is omitted…
  
   Why will the front-ends have a different understanding than dpkg then?
  
  Because 'apt-get remove libc6' will not remove libc6:amd64 and libc6:i386
  like dpkg, but only :native - simple for the reason that it should be the
  reverse of 'apt-get install libc6' which installs only libc6:native, too,
  and not all possible architectures - as it does the same for all packages
  and not based on M-A state as this might change over time.
 
 Well, if dpkg and the front-ends or even each distinct front-end has a
 different interface regarding this, then we have a problem. Users will
 certainly be way more confused this way for sure.
 
 Not getting the cross-grade case right interface-wise might imply that
 it becomes extremely difficult or outright impossible to implement it
 later on.

APT can only have on native architecture during installation basically,
the one defined in APT::Architecture anyway. Supporting cross-grades is
not really in scope then.

 
 Even then, as I stated on my initial mail, if we ignore the
 cross-grade case, let's consider the situation of a system with just a
 i386 essential package set, where an amd64 apt (and needed dependencies)
 have just been installed. The notion of native arch is going to be
 different for both dpkg and apt which depending on the interface might
 produce interesting results to say the least.

So, a beginning would be to ask dpkg for the native architecture at
run-time; if none is set in APT::Architecture.

 
 In addition all this is for M-A: same packages, which should in most
 (if not all?) cases be just dependency packages, not something the
 user might need to request by hand, and as such if they'd need to,
 they can use the proper syntax.
 
 Regarding your 'apt-get install libc6', I don't see why it could not
 be made to install libc6 for all configured foreign architectures,
 which would match nicely the remove case, and would be pretty
 consistent. The user would see the proposed list of packages and could
 opt-out in case that was not wanted.

That's the way Fedora did it with 32-bit/64-bit packages IIRC. Not all
users are intelligent enough and will then have lots of duplicate stuff
installed.

We discussed command-line last year as well, see
http://lists.debian.org/deity/2010/08/msg00077.html

 
  The idea of not printing the architecture for M-A:foreign packages is also
  a dpkg-vs-apt thing as APT needs to provide the user with a way to choose
  which architecture should it be and if it changes the architecture it has to
  display this somehow and showing 'removing libc-bin; installing libc-bin'
  isn't exactly useful to understand what actually happens.
 
 Well, I'd argue that's a flaw in the current implementation then, the
 need for removal and reinstallation is arbitrary and not something that
 should be really required, dpkg has supported until now cross-grading a
 package to a different architecture even if it stopped doing so in some
 cases now on the multiarch branch, and as you pointed out already
 something that will definitely cause problems with essential or
 pseudo-essential packages.
 
 So it's something that should either be not supported at all, or
 supported fully, I don't think removal and install is an acceptable
 behaviour.

I don't think it was ever specified anywhere, or am I wrong? 

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


-- 
To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20111215105305.ga3...@debian.org

Bug#436733: dpkg: Improve handling of dependencies on arch: all packages

[Julian Andres Klode]

Package: dpkg
Version: 1.14.5
Severity: wishlist

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

dpkg should export dependencies like pkg [amd64 i386] to
the binary package and ignore dependencies for non-matching
architectures at runtime.
Also, I propose to add a field Install-to: for arch-indep packages, 
which defines to which Packages file it will be installed.
This would allow packages like ndisgtk to be arch-indep, but installed
only into the i386 and amd64 Packages files.

- -- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable')
Architecture: i386 (i686)

Kernel: Linux 2.6.22-1-686 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

Versions of packages dpkg depends on:
ii  coreutils 5.97-5.3   The GNU core utilities
ii  libc6 2.6-5  GNU C Library: Shared libraries

dpkg recommends no packages.

- -- no debconf information

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGugyyrCpf/gCCPsIRAqqcAJ48u/vn9j5YKhfS1IJJvigNp5HqOwCfc9Q5
Uz2zeNcxBteT9+mhW7DaIrw=
=pdBD
-END PGP SIGNATURE-



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Bug#486323: dpkg-dev: Does not handle all comments in quilt series files

[Julian Andres Klode]

Package: dpkg-dev
Version: 1.14.19
Severity: normal
Tags: patch

The current version does not handle comments in the series file, if
they are at the beginning of the line.

I wrote a patch and will send it once I know the bug number.

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.25-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg-dev depends on:
ii  binutils2.18.1~cvs20080103-6 The GNU assembler, linker and bina
ii  bzip2   1.0.5-0.1high-quality block-sorting file co
ii  cpio2.9-13   GNU cpio -- a program to manage ar
ii  dpkg1.14.19  package maintenance system for Deb
ii  libtimedate-perl1.1600-9 Time and date functions for Perl
ii  lzma4.43-12  Compression method of 7z format in
ii  make3.81-5   The GNU version of the make util
ii  patch   2.5.9-5  Apply a diff file to an original
ii  perl [perl5]5.10.0-10Larry Wall's Practical Extraction 
ii  perl-modules5.10.0-10Core Perl modules

Versions of packages dpkg-dev recommends:
ii  build-essential   11.3   informational list of build-essent
ii  gcc [c-compiler]  4:4.3.0-8  The GNU C compiler
ii  gcc-4.1 [c-compiler]  4.1.2-22   The GNU C compiler
ii  gcc-4.3 [c-compiler]  4.3.1-2The GNU C compiler

-- no debconf information


signature.asc
Description: Digital signature

Bug#486323: [PATCH] scripts/Dpkg/Source/Package/V3/quilt.pm: Handle comments at the beginning of a line in quilt series files. Closes: #486323

[Julian Andres Klode]

---
 debian/changelog|6 +-
 scripts/Dpkg/Source/Package/V3/quilt.pm |2 +-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/debian/changelog b/debian/changelog
index aeb3a53..4ab7a47 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -43,7 +43,11 @@ dpkg (1.15.0) UNRELEASED; urgency=low
   [ Updated dpkg translations ]
   * Portuguese (Miguel Figueiredo).
 
- -- Guillem Jover [EMAIL PROTECTED]  Tue, 29 Apr 2008 06:01:40 +0300
+  [ Julian Andres Klode ]
+  * scripts/Dpkg/Source/Package/V3/quilt.pm: Handle comments at the beginning
+of a line in quilt series files. Closes: #486323
+
+ -- Julian Andres Klode [EMAIL PROTECTED]  Sun, 15 Jun 2008 13:21:06 +0200
 
 dpkg (1.14.20) UNRELEASED; urgency=low
 
diff --git a/scripts/Dpkg/Source/Package/V3/quilt.pm 
b/scripts/Dpkg/Source/Package/V3/quilt.pm
index ef27023..c4eb2fc 100644
--- a/scripts/Dpkg/Source/Package/V3/quilt.pm
+++ b/scripts/Dpkg/Source/Package/V3/quilt.pm
@@ -77,7 +77,7 @@ sub get_patches {
 open(SERIES,  , $series) || syserr(_g(cannot read %s), $series);
 while(defined($_ = SERIES)) {
 chomp; s/^\s+//; s/\s+$//; # Strip leading/trailing spaces
-s/\s#.*$//; # Strip trailing comment
+s/(^|\s)#.*//; # Strip comment
 next unless $_;
 if (/^(\S+)\s+(.*)$/) {
 $_ = $1;
-- 
1.5.5.4





-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Bug#486323: dpkg-dev: Does not handle all comments in quilt series files

[Julian Andres Klode]

Raphael Hertzog wrote:
 On Sun, 15 Jun 2008, Julian Andres Klode wrote:
 I wrote a patch and will send it once I know the bug number.
 
 Please before starting to write a patch, check if there's no fix
 in the git tree already... in particular when I pointed out 
 that the upcoming dpkg 1.14.20 contained some changes related to the new
 format.
I did debcheckout dpkg and the patch was not in there. I did not know about the
lenny branch, else I would have checked it.
 
 http://git.debian.org/?p=dpkg/dpkg.git;a=commit;h=7f752eaecf07b776978f11eee8602d908782d8d9
 
 Cheers,


-- 
Julian Andres Klode, Fellow of the Free Software Foundation Europe
 Debian Maintainer | Developer | Ubuntu Member

try Debian: http://www.debian.org/ | my site: http://jak-linux.org/
jabber: [EMAIL PROTECTED] | IRC: juliank (FreeNode, OFTC)
languages: German  | English



signature.asc
Description: OpenPGP digital signature

Bug#490929: dpkg-dev: If gpg hangs, dpkg-source hangs without explanation

[Julian Andres Klode]

Package: dpkg-dev
Version: 1.14.20
Severity: normal

dpkg-source calls gpg --verify to verify the source package. If there
is a lock, gpg displays that it is waiting for the lock. dpkg-source
just appears to hang. The same applies for automatic key retrieval.

1) I suggest to use gpgv to verify the source package, as gpgv seems to not
hang at locks (since it can only use the keyring read-only). It also does
not use key retrieval.

2) I also suggest to display all messages from gpg, so people
can see the problem.

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.25-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg-dev depends on:
ii  binutils2.18.1~cvs20080103-7 The GNU assembler, linker and bina
ii  bzip2   1.0.5-0.1high-quality block-sorting file co
ii  cpio2.9-13   GNU cpio -- a program to manage ar
ii  dpkg1.14.20  Debian package management system
ii  libtimedate-perl1.1600-9 Time and date functions for Perl
ii  lzma4.43-14  Compression method of 7z format in
ii  make3.81-5   The GNU version of the make util
ii  patch   2.5.9-5  Apply a diff file to an original
ii  perl [perl5]5.10.0-11Larry Wall's Practical Extraction 
ii  perl-modules5.10.0-11Core Perl modules

Versions of packages dpkg-dev recommends:
ii  bcc [c-compiler]  0.16.17-2  16-bit x86 C compiler
ii  build-essential   11.4   Informational list of build-essent
ii  gcc [c-compiler]  4:4.3.1-2  The GNU C compiler
ii  gcc-4.1 [c-compiler]  4.1.2-23   The GNU C compiler
ii  gcc-4.2 [c-compiler]  4.2.4-3The GNU C compiler
ii  gcc-4.3 [c-compiler]  4.3.1-6The GNU C compiler

-- no debconf information


pgpb0gLdhO1UY.pgp
Description: PGP signature

Bug#561104: dpkg: Add --readahead-db option to preload the database

[Julian Andres Klode]

Package: dpkg
Version: 1.15.5.4
Severity: wishlist

Currently, running dpkg on my system is rather slow and can take 10 seconds
or something to read the database. I propose an option --readahead-db which
uses readahead(2) on all database files; and would be called by APT during
package retrieval or dependency solving, so that the database is already
cached when dpkg is started to install packages.

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (350, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.31-1-amd64 (SMP w/2 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  coreutils 8.0-2  GNU core utilities
ii  libc6 2.10.2-2   GNU C Library: Shared libraries
ii  lzma  4.43-14Compression method of 7z format in

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt   0.7.24 Advanced front-end for dpkg

-- no debconf information

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


signature.asc
Description: Digital signature

Bug#593155: Allow dpkg to be run when other process holds the lock

[Julian Andres Klode]

Package: dpkg
Version: 1.15.8.3
Severity: wishlist

APT holds the dpkg lock itself. Currently, it must release
the lock before calling dpkg which creates a short time
frame where the database is unlocked. It would therefore
be useful if dpkg could be run with APT holding the
lock (possibly by adding a --run-unlocked option)

-- System Information:
Debian Release: lenny/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (350, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  coreutils 8.5-1  GNU core utilities
ii  libbz2-1.01.0.5-4high-quality block-sorting file co
ii  libc6 2.11.2-2   Embedded GNU C Library: Shared lib
ii  libselinux1   2.0.96-1   SELinux runtime shared libraries
ii  xz-utils  4.999.9beta+20100713-1 XZ-format compression utilities
ii  zlib1g1:1.2.3.4.dfsg-3   compression library - runtime

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt   0.7.25.3   Advanced front-end for dpkg

-- no debconf information

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


pgpX5UIWhMRdU.pgp
Description: PGP signature

Bug#627412: dpkg: dpkg --install should not unpack package if dependencies are missing

[Julian Andres Klode]

Package: dpkg
Version: 1.16.0.3
Severity: wishlist

While it is formally correct for dpkg to unpack the package
on --install when the dependencies are missing, doing so
creates a problem if users try to install a low-level
package with missing dependencies (remember the APT
Pre-Depends thread).

I would welcome it if dpkg --install were changed to
unpack the package only if its dependencies are
satisfied. For unpacking-only purposes ignoring
dependency, there is --unpack which is used by
APT.


-- System Information:
Debian Release: wheezy/sid
  APT prefers unstable
  APT policy: (950, 'unstable'), (250, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.38-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  coreutils   8.5-1GNU core utilities
ii  libbz2-1.0  1.0.5-6  high-quality block-sorting file co
ii  libc6   2.13-4   Embedded GNU C Library: Shared lib
ii  libselinux1 2.0.98-1+b1  SELinux runtime shared libraries
ii  xz-utils5.0.0-2  XZ-format compression utilities
ii  zlib1g  1:1.2.3.4.dfsg-3 compression library - runtime

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt   0.8.14.1   Advanced front-end for dpkg

-- no debconf information

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


pgp2vRBrPyYb6.pgp
Description: PGP signature

Bug#641309: dpkg-architecture: strxfrm() gets absurd (on C.UTF-8)

[Julian Andres Klode]

Package: dpkg-dev
Version: 1.16.0.3
Severity: normal
File: /usr/bin/dpkg-architecture

When running in the C.UTF-8 locale, strxfrm() apparently
gets absurd.

$ LC_ALL=C.UTF-8 dpkg-architecture -qDEB_HOST_GNU_CPU 
strxfrm() gets absurd.


-- System Information:
Debian Release: wheezy/sid
  APT prefers unstable
  APT policy: (550, 'unstable'), (500, 'testing'), (300, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 3.0.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg-dev depends on:
ii  base-files6.5  
ii  binutils  2.21.53.20110910-1   
ii  bzip2 1.0.5-7  
ii  libdpkg-perl  1.16.0.3 
ii  make  3.81-8.1 
ii  patch 2.6.1-2  
ii  xz-utils  5.1.1alpha+20110809-2

Versions of packages dpkg-dev recommends:
pn  build-essential  11.5 
pn  fakeroot 1.18-1   
pn  gcc [c-compiler] 4:4.6.1-2
pn  gcc-4.4 [c-compiler] 4.4.6-10 
pn  gcc-4.5 [c-compiler] 4.5.3-8  
pn  gcc-4.6 [c-compiler] 4.6.1-10 
pn  gnupg1.4.11-3 
pn  gpgv 1.4.11-3 
pn  libalgorithm-merge-perl  none   

Versions of packages dpkg-dev suggests:
ii  debian-keyring  2011.08.07

-- no debconf information

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


pgpZfU2sZ4B76.pgp
Description: PGP signature

Bug#641309: dpkg-architecture: strxfrm() gets absurd (on C.UTF-8)

[Julian Andres Klode]

reassign 641309 perl-base 5.12.4-4
thanks

On Mon, Sep 12, 2011 at 04:47:48PM +0200, Julian Andres Klode wrote:
 Package: dpkg-dev
 Version: 1.16.0.3
 Severity: normal
 File: /usr/bin/dpkg-architecture
 
 When running in the C.UTF-8 locale, strxfrm() apparently
 gets absurd.
 
 $ LC_ALL=C.UTF-8 dpkg-architecture -qDEB_HOST_GNU_CPU 
 strxfrm() gets absurd.

It seems to be a perl issue, or something below perl that's
causing it. It worked correctly about a month ago.


-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.


pgpSNQNuyWJzw.pgp
Description: PGP signature

Bug#731710: update-alternatives: Segmentation fault

[Julian Andres Klode]

Package: dpkg
Version: 1.17.4
Severity: critical

update-alternative crashes when creating the alternatives for
epiphany-browser, causing epiphany-browser to not be configured,
thus breaking it, and probably other software as well.

The part of the postinst causing the seg fault is:

PRIO=85
for alt in x-www-browser gnome-www-browser; do
update-alternatives --install \
/usr/bin/$alt $alt /usr/bin/epiphany-browser $PRIO \
--slave /usr/share/man/man1/$alt.1.gz $alt.1.gz 
/usr/share/man/man1/epiphany-browser.1.gz
done

Maybe you can reproduce this yourself in a debug build, otherwise
I'll do a debug build myself and try to reproduce it with debugging
symbols.

-- System Information:
Debian Release: jessie/sid
  APT prefers unstable
  APT policy: (980, 'unstable'), (500, 'unstable'), (100, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 3.11-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  libbz2-1.0   1.0.6-5
ii  libc62.17-97
ii  liblzma5 5.1.1alpha+20120614-2
ii  libselinux1  2.2.1-1
ii  tar  1.27-3
ii  zlib1g   1:1.2.8.dfsg-1

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt  0.9.14

-- no debconf information

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Please do not top-post if possible.


-- 
To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#731710: update-alternatives: Segmentation fault

[Julian Andres Klode]

On Mon, Dec 09, 2013 at 12:24:52AM +0100, Guillem Jover wrote:
 Control: tags -1 unreproducible moreinfo
 
 Hi!
 
 On Sun, 2013-12-08 at 22:25:35 +0100, Guillem Jover wrote:
  On Sun, 2013-12-08 at 21:39:01 +0100, Julian Andres Klode wrote:
   update-alternative crashes when creating the alternatives for
   epiphany-browser, causing epiphany-browser to not be configured,
   thus breaking it, and probably other software as well.
   
   The part of the postinst causing the seg fault is:
   
   PRIO=85
   for alt in x-www-browser gnome-www-browser; do
   update-alternatives --install \
   /usr/bin/$alt $alt /usr/bin/epiphany-browser $PRIO \
   --slave /usr/share/man/man1/$alt.1.gz $alt.1.gz 
   /usr/share/man/man1/epiphany-browser.1.gz
   done
   
   Maybe you can reproduce this yourself in a debug build, otherwise
   I'll do a debug build myself and try to reproduce it with debugging
   symbols.
 
  Well it does not trigger on the test suite, anyway looking into it now.
 
 I've tried your recipe, and installing the epiphany-browser package,
 neither segfault. And I've re-reviewed the changes in u-a between
 1.17.1 and 1.17.4, and nothing obvious jumps to me. Could you attach
 your /var/lib/dpkg/alternatives/ and /etc/alternatives/ directories and
 to where do the x-www-browser and gnome-www-browser generic names point
 to currently in your file system? (Those would be the /usr/bin/$alt and
 /usr/share/man/man1/$alt.1.gz links.)

The command failing is:
update-alternatives --install /usr/bin/gnome-www-browser gnome-www-browser 
/usr/bin/epiphany-browser 85 --slave /usr/share/man/man1/gnome-www-browser.1.gz 
gnome-www-browser.1.gz /usr/share/man/man1/epiphany-browser.1.gz

$ ls -l /usr/share/man/man1/gnome-www-browser* /usr/bin/gnome-www-browser 
ls: cannot access /usr/share/man/man1/gnome-www-browser*: No such file or 
directory
lrwxrwxrwx 1 root root 35 Sep  7 10:53 /usr/bin/gnome-www-browser - 
/etc/alternatives/gnome-www-browser

I believe the missing man page link is the problem.

Here's a backtrace for the failing command:

Program received signal SIGSEGV, Segmentation fault.
#0  0x00403657 in alternative_copy_slave (a=0x60f0d0, sl=0x60f2e0) at 
../../utils/update-alternatives.c:992
sl_new = 0x0
#1  0x004070ed in alternative_evolve (a=0x60f0d0, b=0x60f140, 
cur_choice=0x63d4f0 /usr/bin/chromium, fs=0x60f1e0) at 
../../utils/update-alternatives.c:2310
sl = 0x60f2e0
is_link = true
#2  0x004090d3 in main (argc=10, argv=0x7fffe0a8) at 
../../utils/update-alternatives.c:2783
a = 0x60f0d0
inst_alt = 0x60f140
fileset = 0x60f1e0
path = 0x0
current_choice = 0x63d4f0 /usr/bin/chromium
new_choice = 0x0
i = 10
rax0x0  0
rbx0x706140 7364928
rcx0x77dd5640   140737351865920
rdx0x1  1
rsi0x51 81
rdi0x0  0
rbp0x7fffdeb0   0x7fffdeb0
rsp0x7fffde80   0x7fffde80
r8 0x0  0
r9 0x3  3
r100x4111041
r110x246582
r120x401510 4199696
r130x7fffe0a0   140737488347296
r140x0  0
r150x0  0
rip0x403657 0x403657 alternative_copy_slave+89
eflags 0x10246  [ PF ZF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0  0
es 0x0  0
fs 0x0  0
gs 0x0  0

Thread 1 (process 30421):
#0  0x00403657 in alternative_copy_slave (a=0x60f0d0, sl=0x60f2e0) at 
../../utils/update-alternatives.c:992
#1  0x004070ed in alternative_evolve (a=0x60f0d0, b=0x60f140, 
cur_choice=0x63d4f0 /usr/bin/chromium, fs=0x60f1e0) at 
../../utils/update-alternatives.c:2310
#2  0x004090d3 in main (argc=10, argv=0x7fffe0a8) at 
../../utils/update-alternatives.c:2783
A debugging session is active.

Inferior 1 [process 30421] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

 
 Thanks,
 Guillem

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Please do not top-post if possible.


-- 
To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#731710: update-alternatives: Segmentation fault

[Julian Andres Klode]

On Mon, Dec 09, 2013 at 05:08:47PM +0100, Guillem Jover wrote:
 On Mon, 2013-12-09 at 16:29:44 +0100, Julian Andres Klode wrote:
  The command failing is:
  update-alternatives --install /usr/bin/gnome-www-browser gnome-www-browser 
  /usr/bin/epiphany-browser 85 --slave 
  /usr/share/man/man1/gnome-www-browser.1.gz gnome-www-browser.1.gz 
  /usr/share/man/man1/epiphany-browser.1.gz
  
  $ ls -l /usr/share/man/man1/gnome-www-browser* /usr/bin/gnome-www-browser 
  ls: cannot access /usr/share/man/man1/gnome-www-browser*: No such file or 
  directory
  lrwxrwxrwx 1 root root 35 Sep  7 10:53 /usr/bin/gnome-www-browser - 
  /etc/alternatives/gnome-www-browser
  
  I believe the missing man page link is the problem.
 
 Didn't crash for me here, but the backtrace was helpful to spot the
 problem. Could you try the attached patch?

Yes, thanks, it works.

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Please do not top-post if possible.


-- 
To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#765952: apt, dpkg: cycle during trigger processing

[Julian Andres Klode]

:amd64 (4.9.1-18) wird eingerichtet ...
python-dateutil (2.2-2) wird eingerichtet ...
gnome-contacts (3.14.1-1) wird eingerichtet ...
libx32gomp1 (4.9.1-18) wird eingerichtet ...
gnome-shell-common (3.14.1-1) wird eingerichtet ...
gnome-system-monitor (3.14.1-1) wird eingerichtet ...
libisc-export95 (1:9.9.5.dfsg-4.3) wird eingerichtet ...
python3-chardet (2.3.0-1) wird eingerichtet ...
libasan1:amd64 (4.9.1-18) wird eingerichtet ...
libx32itm1 (4.9.1-18) wird eingerichtet ...
libdirac-decoder0:amd64 (1.0.2-7.1) wird eingerichtet ...
libsqlite3-0:amd64 (3.8.7-1) wird eingerichtet ...
python2.7-doc (2.7.8-11) wird eingerichtet ...
libvisual-0.4-0:amd64 (0.4.0-6) wird eingerichtet ...
libsbuild-perl (0.65.0-1) wird eingerichtet ...
libktoblzcheck1c2a (1.47-1) wird eingerichtet ...
libwine:amd64 (1.6.2-12) wird eingerichtet ...
libhunspell-1.3-0:amd64 (1.3.3-3) wird eingerichtet ...
bsdmainutils (9.0.6) wird eingerichtet ...
libsasl2-modules:amd64 (2.1.26.dfsg1-12) wird eingerichtet ...
libpython2.7-minimal:amd64 (2.7.8-11) wird eingerichtet ...
lib32quadmath0 (4.9.1-18) wird eingerichtet ...
hicolor-icon-theme (0.13-1) wird eingerichtet ...
libdirac-encoder0:amd64 (1.0.2-7.1) wird eingerichtet ...
lib32itm1 (4.9.1-18) wird eingerichtet ...
libitm1:amd64 (4.9.1-18) wird eingerichtet ...
libpython2.7-stdlib:amd64 (2.7.8-11) wird eingerichtet ...
libx32quadmath0 (4.9.1-18) wird eingerichtet ...
lib32gcc1 (1:4.9.1-18) wird eingerichtet ...
python2.7-minimal (2.7.8-11) wird eingerichtet ...
notification-daemon (0.7.6-2) wird eingerichtet ...
libwxbase3.0-0:amd64 (3.0.2-1+b1) wird eingerichtet ...
cpp (4:4.9.1-5) wird eingerichtet ...
libxen-4.4:amd64 (4.4.1-3) wird eingerichtet ...
libvlc5 (2.2.0~pre4-1+b1) wird eingerichtet ...
libisccfg-export90 (1:9.9.5.dfsg-4.3) wird eingerichtet ...
libsqlite3-dev:amd64 (3.8.7-1) wird eingerichtet ...
vlc-nox (2.2.0~pre4-1+b1) wird eingerichtet ...
python2.7 (2.7.8-11) wird eingerichtet ...
vlc (2.2.0~pre4-1+b1) wird eingerichtet ...
libnss3:amd64 (2:3.17.2-1) wird eingerichtet ...
libpython2.7-dbg:amd64 (2.7.8-11) wird eingerichtet ...
sbuild (0.65.0-1) wird eingerichtet ...
Neue Version der Konfigurationsdatei /etc/sbuild/sbuild.conf wird installiert 
...
libnss3-1d:amd64 (2:3.17.2-1) wird eingerichtet ...
libnss3-dev:amd64 (2:3.17.2-1) wird eingerichtet ...
libdns-export100 (1:9.9.5.dfsg-4.3) wird eingerichtet ...
libwxgtk3.0-0:amd64 (3.0.2-1+b1) wird eingerichtet ...
python2.7-dbg (2.7.8-11) wird eingerichtet ...
libpython2.7:amd64 (2.7.8-11) wird eingerichtet ...
libgcc-4.9-dev:amd64 (4.9.1-18) wird eingerichtet ...
wine64 (1.6.2-12) wird eingerichtet ...
Trigger für man-db (2.7.0.2-2) werden verarbeitet ...
lib32stdc++6 (4.9.1-18) wird eingerichtet ...
libpython2.7-dev:amd64 (2.7.8-11) wird eingerichtet ...
lib32ubsan0 (4.9.1-18) wird eingerichtet ...
lib32cilkrts5 (4.9.1-18) wird eingerichtet ...
libobjc-4.9-dev:amd64 (4.9.1-18) wird eingerichtet ...
python2.7-dev (2.7.8-11) wird eingerichtet ...
libstdc++-4.9-dev:amd64 (4.9.1-18) wird eingerichtet ...
gnome-settings-daemon (3.14.1-1) wird eingerichtet ...
libx32cilkrts5 (4.9.1-18) wird eingerichtet ...
gcc-4.9 (4.9.1-18) wird eingerichtet ...
libx32ubsan0 (4.9.1-18) wird eingerichtet ...
wine (1.6.2-12) wird eingerichtet ...
libirs-export91 (1:9.9.5.dfsg-4.3) wird eingerichtet ...
gnome-shell (3.14.1-1) wird eingerichtet ...
gcc (4:4.9.1-5) wird eingerichtet ...
gnome-control-center (1:3.14.1-1) wird eingerichtet ...
libx32asan1 (4.9.1-18) wird eingerichtet ...
g++-4.9 (4.9.1-18) wird eingerichtet ...
g++ (4:4.9.1-5) wird eingerichtet ...
lib32asan1 (4.9.1-18) wird eingerichtet ...
libx32gcc-4.9-dev (4.9.1-18) wird eingerichtet ...
isc-dhcp-common (4.3.1-5) wird eingerichtet ...
isc-dhcp-client (4.3.1-5) wird eingerichtet ...
lib32gcc-4.9-dev (4.9.1-18) wird eingerichtet ...
gcc-4.9-multilib (4.9.1-18) wird eingerichtet ...
gcc-multilib (4:4.9.1-5) wird eingerichtet ...
Trigger für libc-bin (2.19-11) werden verarbeitet ...


-- System Information:
Debian Release: jessie/sid
  APT prefers unstable
  APT policy: (990, 'unstable'), (500, 'unstable'), (100, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages apt depends on:
ii  adduser 3.113+nmu3
ii  debian-archive-keyring  2014.1
ii  gnupg   1.4.18-4
ii  gnupg2  2.0.26-3
ii  libapt-pkg4.14  1.1~exp7
ii  libc6   2.19-11
ii  libgcc1 1:4.9.1-18
ii  libstdc++6  4.9.1-18

apt recommends no packages.

Versions of packages apt suggests:
pn  apt-doc none
ii  aptitude0.6.11-1
ii  dpkg-dev1.17.18
ii  python-apt  0.9.3.10

-- no debconf information

-- 
Julian Andres Klode  - Debian Developer, Ubuntu Member

See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.

Be friendly, do not top-post, and follow

[RFC/dpkg PATCH] Introducing an relaxed-Essential-like "Important" field

[Julian Andres Klode]

Since a few years, APT supports an "Important" field that is similar
to Essential, but without the requirement for those packages to be
installed (they just need to remain installed) and the ordering
constraints. Previously, it was already an alias for Essential in
APT.

I relaxed the meaning a few years ago to make it suitable for use
on site-specific or system-specific configuration meta packages.

I propose to make this field official and add support to dpkg
for it, as there are new use cases for it, like init systems,
e2fsprogs, and mount - packages that are not needed on all 
systems (like chroots), but once installed should probably
remain installed.

I attached a patch to add support for dpkg, it's also discussed
in a spec in the wiki.

I'd like some feedback.

References:

[1] https://wiki.debian.org/Teams/Dpkg/Spec/ImportantField

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev

When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to (`inline'). Thank you.
>From 141931815cdf663aab5ecccf6b4fdd517f99cbd4 Mon Sep 17 00:00:00 2001
From: Julian Andres Klode <j...@debian.org>
Date: Sun, 6 Mar 2016 00:55:31 +0100
Subject: [PATCH] Add support for the "Important" field

The important field is like essential, but imposes less
requirements.

See: https://wiki.debian.org/Teams/Dpkg/Spec/ImportantField
---
 debian/changelog   |  3 +++
 lib/dpkg/dpkg-db.h |  1 +
 lib/dpkg/parse.c   |  1 +
 lib/dpkg/pkg.c |  1 +
 man/deb-control.5  |  6 ++
 man/dpkg.1 |  5 +
 scripts/Dpkg/Control/FieldsCore.pm |  3 +++
 src/archives.c | 16 +++-
 src/main.c |  3 +++
 src/main.h |  1 +
 src/remove.c   |  4 
 src/select.c   |  2 +-
 src/unpack.c   |  1 +
 13 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/debian/changelog b/debian/changelog
index 546cff5..92cca7b 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -95,6 +95,9 @@ dpkg (1.18.5) UNRELEASED; urgency=medium
   * Simplified Chinese (Zhou Mo). Closes: #809517
   * Vietnamese (Trần Ngọc Quân).
 
+  [ Julian Andres Klode ]
+  * Add support for the "Important" field
+
  -- Guillem Jover <guil...@debian.org>  Fri, 25 Dec 2015 14:20:16 +0100
 
 dpkg (1.18.4) unstable; urgency=medium
diff --git a/lib/dpkg/dpkg-db.h b/lib/dpkg/dpkg-db.h
index 389aaad..4748854 100644
--- a/lib/dpkg/dpkg-db.h
+++ b/lib/dpkg/dpkg-db.h
@@ -108,6 +108,7 @@ struct pkgbin {
   struct dependency *depends;
   /** The ‘essential’ flag, true = yes, false = no (absent). */
   bool essential;
+  bool important;
   enum pkgmultiarch multiarch;
   const struct dpkg_arch *arch;
   /** The following is the "pkgname:archqual" cached string, if this was a
diff --git a/lib/dpkg/parse.c b/lib/dpkg/parse.c
index 2d7f0a3..150b4a7 100644
--- a/lib/dpkg/parse.c
+++ b/lib/dpkg/parse.c
@@ -54,6 +54,7 @@ const struct fieldinfo fieldinfos[]= {
   /* Note: Capitalization of field name strings is important. */
   { FIELD("Package"),  f_name,w_name },
   { FIELD("Essential"),f_boolean, w_booleandefno,   PKGIFPOFF(essential) },
+  { FIELD("Important"),f_boolean, w_booleandefno,   PKGIFPOFF(important) },
   { FIELD("Status"),   f_status,  w_status   },
   { FIELD("Priority"), f_priority,w_priority },
   { FIELD("Section"),  f_section, w_section  },
diff --git a/lib/dpkg/pkg.c b/lib/dpkg/pkg.c
index 1fc3d0c..7fe8ad8 100644
--- a/lib/dpkg/pkg.c
+++ b/lib/dpkg/pkg.c
@@ -96,6 +96,7 @@ void
 pkgbin_blank(struct pkgbin *pkgbin)
 {
 	pkgbin->essential = false;
+	pkgbin->important = false;
 	pkgbin->depends = NULL;
 	pkgbin->pkgname_archqual = NULL;
 	pkgbin->description = NULL;
diff --git a/man/deb-control.5 b/man/deb-control.5
index fea726c..8cbaebe 100644
--- a/man/deb-control.5
+++ b/man/deb-control.5
@@ -98,6 +98,12 @@ a package that is required for proper operation of the system. Dpkg
 or any other installation tool will not allow an
 .B Essential
 package to be removed (at least not without using one of the force options).
+.BR Important: " \fByes\fP|\fBno\fP"
+This field is usually only needed when the answer is \fByes\fP. It denotes
+a package that should not be removed from a system. Dpkg
+or any other installation tool will not allow an
+.B Important
+package to be removed (at least not without using one of the force options).
 .TP
 .BR Build\-Essential: " \fByes\fP|\fBno\fP"
 This field is usual

Bug#830267: dpkg: Segmentation fault when purging package in APT test case

[Julian Andres Klode]

Package: dpkg
Version: 1.18.9
Severity: serious

dpkg fails to purge a package in our test suite, crashing with a segmentation 
fault. You can reproduce
it by building apt and running 
test/integration/test-bug-712116-dpkg-pre-install-pkgs-hook-multiarch -
I have also included the backtrace here.

#0  namenodetouse (namenode=0x0, pkg=pkg@entry=0x560c3548cfb0, 
pkgbin=pkgbin@entry=0x560c3548d000) at ../../src/help.c:58
r = 
#1  0x560c33884750 in removal_bulk_remove_configfiles (pkg=0x560c3548cfb0) 
at ../../src/remove.c:533
usenode = 
removevb_state = {used = 0}
fnvb = {used = 65, size = 74, buf = 0x560c35491730 
"/tmp/user/1000/tmp.g6iz3eF7hX/rootdir/etc/compiz.conf/compiz.conf"}
removevb = {used = 0, size = 0, buf = 0x0}
namenode = 
conffbasenamelen = 
conffbasename = 
lconffp = 
de = 
p = 
dsd = 
rc = 
conffnameused = 
conff = 0x560c3548d1b0
searchfile = 
ext = 
removeconffexts = {0x560c338a8def "~", 0x560c338a4d45 ".bak", 
0x560c338a4d4a "%", 0x560c3389dc6b ".dpkg-tmp", 0x560c3389dc75 ".dpkg-new", 
0x560c338a4d4c ".dpkg-old", 0x560c3389f0b1 ".dpkg-dist", 0x0}
#2  removal_bulk (pkg=pkg@entry=0x560c3548cfb0) at ../../src/remove.c:637
foundpostrm = 
#3  0x560c33885553 in deferred_remove (pkg=0x560c3548cfb0) at 
../../src/remove.c:192
raemsgs = {used = 0, size = 0, buf = 0x0}
dep = 
rok = DEP_CHECK_OK
#4  0x560c33883142 in process_queue () at ../../src/packages.c:288
rundown = 
pkg = 0x560c3548cfb0
action_todo = act_purge
ejbuf = {{__jmpbuf = {0, 1008670450863515041, 1, 140722198930704, 0, 0, 
-1006724711060265567, -6780200583402150495}, __mask_was_saved = 0, __saved_mask 
= {__val = {140240791502169, 140722198930256, 94610404235612, 140722198930256, 
94610404235761, 0, 16, 94610433560496, 94610404104128, 140722198930704, 0, 0, 
140240791507784, 0, 94610404214745, 140722198930704
istobe = 
__func__ = "process_queue"
__PRETTY_FUNCTION__ = "process_queue"
#5  0x560c33883488 in packages (argv=) at 
../../src/packages.c:162
No locals.
#6  0x560c338756a9 in main (argc=, argv=0x7ffc70ade568) at 
../../src/main.c:901
ret = 

-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (900, 'unstable'), (500, 'unstable-debug'), (500, 'testing'), 
(500, 'stable'), (100, 'experimental'), (1, 'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.6.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dpkg depends on:
ii  libbz2-1.0   1.0.6-8
ii  libc62.22-13
ii  liblzma5 5.1.1alpha+20120614-2.1
ii  libselinux1  2.5-3
ii  tar  1.29-1
ii  zlib1g   1:1.2.8.dfsg-2+b1

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt  1.3~exp3

-- no debconf information

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev

When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to (`inline'). Thank you.

[RFC/PATCH] dpkg frontend locking

[Julian Andres Klode]

Hi everyone,

I talked with guillem about this idea on IRC, but I decided to
write this down in an email for further discussion.

Currently, APT and other dpkg frontends have to acquire the dpkg
database lock at the start of their process and then have to release
the dlock before invoking dpkg and re-acquire it afterwards. 

This leads to a race condition where another process could acquire
the database lock while another frontend is about to start a new
dpkg process.

The proposed fix for that is simple: Introduce a new lock called
the frontend lock.

Frontends will acquire both the frontend lock and the database
lock (in that order) and for invoking dpkg only release the
database lock while keeping the frontend lock.

Dpkg will acquire both locks, unless DPKG_FRONTEND_LOCKED is
set, in which case only the database lock is acquired and the
frontend lock is ignored.

Compatibility matrix:

Assuming two frontends (f1 running, f2 not) and a dpkg with the old
and new behavior each, it looks like this.

   f1   f2  dpkgsituation
1. old  old old Same as now (easy race)
2. old  old new Same as now (easy race)
3. old  new old still a race as f1 has no frontend lock
4. old  new new still a race
5. new  old old still a race possible.
6. new  old new no race. dpkg will refuse to run by f2
because it will try to acquire the fe lock
hold by f1.
7. new  new old the frontends can't run at the same time,
but dpkg could be run manually
8. new  new new like 7, but without the manual dpkg backdoor


Branches:

apt: git fetch https://github.com/julian-klode/apt bugfix/big-lock
-> diff: 
https://github.com/Debian/apt/compare/master...julian-klode:bugfix/big-lock?expand=1

dpkg: git fetch https://github.com/julian-klode/dpkg/compare/master 
pu/frontend-lock 
-> diff: 
https://github.com/julian-klode/dpkg/compare/master...julian-klode:pu/frontend-lock

It's also trivial to add to python-apt, meaning that apt and the
whole python-apt ecosystem (including unattended-upgrades) will
be safe against interference from other package manager frontends.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: Bug#748936: apt doesnt understand architecture wildcards

[Julian Andres Klode]

On Sun, Jan 08, 2017 at 01:33:08AM +, James Clarke wrote:
> On Mon, Feb 01, 2016 at 12:05:18AM +0100, David Kalnischkies wrote:
> > On Wed, Jan 20, 2016 at 01:39:45PM +, Colin Watson wrote:
> > > On Wed, Jan 20, 2016 at 12:31:52PM +0100, Balint Reczey wrote:
> > > > On 06/04/2014 03:41 AM, Guillem Jover wrote:
> > > > >  * Other programs could “easily” use dpkg-architecture to check for
> > > > >identity or a match. (This poses problems for programs that do not
> > > >
> > > > I think making apt call dpkg-architecture for matching would be a good
> > > > way of ensuring consistency with dpkg. Caching the results in a hash
> > > > table would make matching even faster than it is currently.
> > >
> > > dpkg-architecture is in dpkg-dev, so not reliably usable at run-time.
> >
> > Also, apt is trying to remain largely independent of the low-level
> > package manager, so bluntly depending on it wouldn't be good, but …
> >
> > … apt could survive this (for now) as these architecture specifications
> > can at the moment only be encountered in
> > a) build-dependencies of source packages
> >(effecting via python-apt also presumably stuff like dak)
> > b) the commandline like 'apt install libfoo:linux-* foo:linux-any'
> >(and aptitude uses it, too, for this)
> >
> > So, we could do that without too much pain, while keeping a fallback
> > around for cases in which we don't have dpkg-architecture around for
> > whatever reason as it doesn't effect apts primary operation (but might
> > effect the primary operation of other tools which would need to be
> > tested first).
> >
> >
> > I wonder through if we aren't creating the debian version of a tar bomb
> > (xkcd#1168) and to illustrate that a little quiz:
> >
> > dpkg-architecture -ai386 -ii386 true
> > dpkg-architecture -ai386 -ilinux-i386   true
> > dpkg-architecture -ai386 -iany-i386 true
> > dpkg-architecture -aarmhf -iarmhf   true
> > dpkg-architecture -aarmhf -ilinux-armhf true
> > dpkg-architecture -aarmhf -iany-armhf   false
> > dpkg-architecture -aarmhf -iarm false
> > dpkg-architecture -aarmhf -ilinux-arm   false
> > dpkg-architecture -aarmhf -iany-arm true
> >
> > Now, given we want to go to -- lets see:
> > dpkg-architecture -aarmhf -iany-linux-arm   true
> > dpkg-architecture -aarmhf -iany-any-arm true
> > dpkg-architecture -aarmhf -ignu-any-arm false
> > dpkg-architecture -aarmhf -ignueabihf-any-arm   true
> >
> > And to cool off, think about what matches any-arm, linux-any, and if
> > gnu-any is even supported and if what that matches…
> >
> >
> > Truth be told, I would have died on 'any-armhf' already even through
> > that is "obvious" as 'linux-armhf' is interpreted as a literal
> > architecture, while 'any-armhf' is a pattern (just that neither dpkg nor
> > the policy highlight that such a difference exists explicitly).
> >
> > Anyway, I somehow doubt it will be a good idea to trouble mere mortals
> > with the difference between gnu and gnueabihf for matching proposes, so
> > I wonder why we have to trouble them with the difference of armhf and
> > arm depending on if that specification is a literal architecture or an
> > architecture pattern – especially if the two are different only for some
> > architectures… I would personally be more happy with any-armhf working
> > (and a special case letting arm match all of the arms).
> >
> >
> > So, I actually like how apt behaves currently as it just makes more
> > sense in my head to expand armhf to gnu-linux-armhf and match it against
> > gnu-any-armhf instead of gnueabihf-linux-arm and and gnueabihf-any-arm,
> > but so be it – it tends to be more important to have a consistent answer
> > in Debian than to keep me sane… (yeah, I know, that triplet makes
> > perfect sense if you know history, Debian and arm – I just don't).
> >
> >
> > I am therefore going to happily accept any patch flying my way
> > implementing architecture wildcards differently if need be, but I am not
> > going to do it myself mainly because I expect that to have fallout – not
> > in apt, but in things using apt – and I don't have the energy (or the
> > rights) to deal with such things efficiently.
> 
> Hi David,
> I actually ran into this bug in the real world: elfutils has a
> Build-Depends: gcc-multilib [any-amd64], which includes x32
> (x32-gnu-linux-amd64), but apt build-dep didn't install it, so
> dpkg-checkbuilddeps errored out when building. I agree some of this
> seems crazy, but it's even more crazy to have apt and dpkg disagree IMO.
> 
> I have attached an initial proof-of-concept[0] patch for apt to embed the
> list of architecture tuples at build-time. It's not especially pretty,

Re: A radically different proposal for differential updates

[Julian Andres Klode]

On Tue, Aug 15, 2017 at 09:26:24AM +0200, Christian Seiler wrote:
> Hi there,
> 
> I've come to believe that binary diff packages are not the best way of
> solving this issue. Intead I'd like to propse a radically different
> solution to this issue.
> 
> The gist of it: instead of adding a format for how deltas work, I
> propose to introduce a new format for storing Debian packages that will
> be used for both the initial installation _and_ incremental updates.
> 
> This idea was inspired by the following talk given by Lennart
> Poettering about a new tool he's written (which is already packaged for
> Debian by the way):
> https://www.youtube.com/watch?v=JnNkBJ6pr9s
> 
> 
> 
> Now to my proposal:
> 
> A Debian package currently consists of two files: control.tar.gz and
> data.tar.xz (or .gz). What I want to propose is a new format that does
> not contain a data.tar.xz at all. Instead I'd like to split the
> data.tar.xz into chunks and have the new format only contain an index
> that references these chunks. Let me call this new format "cdeb" for
> "chunked deb".
> 
[...]
> Anyway: thoughts? Regards, Christian

It's of course an awesome idea. But:

I generally agree with the idea of chunk stores. They'd improve
things a lot. Now, instead of chunking the tarfiles, chunk both
the individual files, and the tarfiles. Then, with an index for
the individual files in control.tar listing the chunks, you can
easily reconstruct just the files that changed on your system
and avoid any rebuilding of debs for upgrades :D

That said, I believe that this change won't sell. Replacing the
basic format the repository is made of won't fare well. Too many
tools (most of which probably are not known) rely on the presence
of deb files in archives.

So as sad as it might be, I think we probably have to settle for
delta files.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Evaluation (Re: Proposal: A new approach to differential debs)

[Julian Andres Klode]

On Sat, Aug 12, 2017 at 02:16:21PM -0400, Julian Andres Klode wrote:
> Hi everyone,
> 
> (I CCed -devel and deity, but we probably should just discuss
>  that on -dpkg)
> 
> while breakfast here at DebConf, the topic of delta upgrades
> came up. I think delta debs are generally a thing we should
> aim to have, but debdelta is not the implementation we want:
> 
> * It is not integrated into APT or dpkg
> * It relies on a script shipped in the debdelta to generate
>   a new deb from and old deb

> We guessed that generating the new deb from the old deb is
> actually the slowest part of the whole debdelta process. I
> propose a new solution, which does not really have a name
> yet.

# Evaluation

I build a few sample delta debs, using bsdiff (.pdeb) and
xdelta3 (.xdeb). I attached the scripts to generate them
and apply them to an unpacked directory tree.

Feel free to check with other packages, here are the current
evaluations, including a comparison against debdelta.

libreoffice-core (size only):

  -rw-r--r-- 1 jak jak 29M Jul 22 20:02 libreoffice-core_5.3.5~rc1-3_amd64.deb
  -rw-r--r-- 1 jak jak 31M Jul 16 00:10 libreoffice-core_5.4.0~rc2-1_amd64.deb
  -rw-r--r-- 1 jak jak 31M Jul 28 18:29 libreoffice-core_5.4.0-1_amd64.deb
  -rw-r--r-- 1 jak jak  18M Aug 15 23:44 
libreoffice-core_5.3.5~rc1-3_5.4.0-1_amd64.pdeb
  -rw-r--r-- 1 jak jak 4.5M Aug 15 23:42 
libreoffice-core_5.4.0~rc2-1_5.4.0-1_amd64.pdeb

For 5.4~rc2 to 5.4 it made a huge difference, for 5.3.5~rc1 to 5.4 not so much,
so it probably is a good fit for stable updates, but not for unstable and 
testing.

firefox (size & performance):

 -rw-r--r-- 1 jak jak 2.3M Aug 15 20:59 firefox_55.0-1_55.0-2_amd64.debdelta
 -rw-r--r-- 1 jak jak 2.4M Aug 15 22:13 firefox_55.0-1_55.0-2_amd64.pdeb
 -rw-r--r-- 1 jak jak 7.4M Aug 15 22:36 firefox_55.0-1_55.0-2_amd64.xdeb
 -rw-r--r-- 1 jak jak  38M Aug 10 06:49 firefox_55.0-1_amd64.deb
 -rw-r--r-- 1 jak jak  38M Aug 10 12:44 firefox_55.0-2_amd64.deb

Generating the -2 deb from the -1 deb using debdelta took about 47 seconds. In
contrast, applying the .pdeb and .xdeb files to an installed directory tree
took about 1.5 seconds.

The .pdeb uses bsdiff compression, the .xdeb uses xdelta 3. It took
96 seconds to generate the pdeb, and 13 seconds to generate the xdeb
on my ThinkPad X230 with 16 GB of RAM and a Core i5-3320M.

# Conclusions

1. xdelta3 is substantially faster than bsdiff, but bsdiff produces 
substantially
   smaller update sizes.
2. deltas for major updates are too big to be massively useful, so focusing on
   stable and security updates seems like a good idea (though we do need to have
   a set of pdebs for testing in unstable and/or testing)

# Further extensions

If you put a pristine-tar delta into the delta file, you can fully
reconstruct debs.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.


build-patch.sh
Description: Bourne shell script


apply-patch.sh
Description: Bourne shell script

Re: Proposal: A new approach to differential debs

[Julian Andres Klode]

On Sun, Aug 13, 2017 at 12:38:56PM +0300, Adrian Bunk wrote:
> On Sat, Aug 12, 2017 at 02:16:21PM -0400, Julian Andres Klode wrote:
> >...
> > I think delta debs are generally a thing we should aim to have,
> >...
> 
> It sounds like something that would have been a cool feature 20 years
> ago when I was downloading Debian updates over an analog modem.
> 
> Today the required effort, infrastructure and added complexity would
> IMHO not be worth it for a potential few percent of bandwidth decrease.
> 
> > The .diff.tar member contains patches to apply to individual
> > files of the old package. No idea about specific algorithm
> > to choose here, yet.
> >...
> 
> Do you really want to ship *patches*, and not just copies of all
> changed files?
> 
> Patching binaries to a new upstream version doesn't sound to me like 
> something that would make sense.

bsdiff was specifically invented for patching binaries. See the
evaluation I posted a (few) hour(s) ago. It's used succesfully by
FreeBSD, Chrome, Android, Apple App Store, and other places.

Chrome switched to courgette meanwhile, which basically just 
disassembles the code and unifies offsets before passing it
to bsdiff.

Especially for security and stable updates, as opposed to new
(major) upstream versions it makes a substantial difference
(93% savings for the firefox 55.0-1 to 55.0-2 release, from 38
 MB down to 2.4).

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: Proposal: A new approach to differential debs

[Julian Andres Klode]

On Sun, Aug 13, 2017 at 10:53:16AM -0400, Peter Silva wrote:
> You are assuming the savings are substantial.  That's not clear.  When
> files are compressed, if you then start doing binary diffs, well it
> isn't clear that they will consistently be much smaller than plain new
> files.  it also isn't clear what the impact on repo disk usage would
> be.
> 
> The most straigforward option:
> The least intrusive way to do this is to add differential files in
> addition to the existing binaries, and any time the differential file,
> compared to a new version exceeds some threhold size (for example:
> 50%) of the original file, then you end up adding the sum total of the
> diff files in addition to the regular files to the repos.  I haven't
> done the math, but it's clear to me that it ends up being about double
> the disk space with this approach. it's also costly in that all those
> files have to be built and managed, which is likely a substantial
> ongoing load (cpu/io/people)  I think this is what people are
> objecting to.
> 
> A more intrusive and less obvious way to do this is to use zsync (
> http://zsync.moria.org.uk/. ) With zsync, you build tables of content
> for each file, using the same 4k blocking that rsync does. To handle
> compression efficiently, there needs to be an understanding of
> blocking, so a customized gzip needs to be used.  With such a format,
> you produce the same .deb's as today, with the .zsyncs (already in
> use?) and the author already provides some debian Packages files as
> examples.  The space penalty here is probably only a few percent.

Today's research has shown that rolling hashes do not perform well
on executables because of changing offsets and so on destroying the
hashes. There were no measurable space savings when adding fairly
similar firefox releases to either a casync or borg repository -
and that's on uncompressed tarballs of the latest 2 firefox uploads.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: Evaluation (Re: Proposal: A new approach to differential debs)

[Julian Andres Klode]

On Wed, Aug 16, 2017 at 12:21:09AM +0200, Julian Andres Klode wrote:
> firefox (size & performance):
> 
>  -rw-r--r-- 1 jak jak 2.3M Aug 15 20:59 firefox_55.0-1_55.0-2_amd64.debdelta
>  -rw-r--r-- 1 jak jak 2.4M Aug 15 22:13 firefox_55.0-1_55.0-2_amd64.pdeb
>  -rw-r--r-- 1 jak jak 7.4M Aug 15 22:36 firefox_55.0-1_55.0-2_amd64.xdeb
>  -rw-r--r-- 1 jak jak  38M Aug 10 06:49 firefox_55.0-1_amd64.deb
>  -rw-r--r-- 1 jak jak  38M Aug 10 12:44 firefox_55.0-2_amd64.deb
> 
> Generating the -2 deb from the -1 deb using debdelta took about 47 seconds. In
> contrast, applying the .pdeb and .xdeb files to an installed directory tree
> took about 1.5 seconds.
> 
> The .pdeb uses bsdiff compression, the .xdeb uses xdelta 3. It took
> 96 seconds to generate the pdeb, and 13 seconds to generate the xdeb
> on my ThinkPad X230 with 16 GB of RAM and a Core i5-3320M.

We're down to about 40 seconds now if we replace the embedded qsufsort in
bsdiff with linking to libdivsufsort. It's still slow, so if we want to do
that, it seems the only plausible way is for buildds to fetch the old
packages after a build, generate the deltas, and append them to .changes
(or build a delta changes file).

I linked the libdivsufsort change and other relevant performance improvements
from chromium in:
  
  https://bugs.debian.org/409664


-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: Proposal: A new approach to differential debs

[Julian Andres Klode]

On Sun, Aug 13, 2017 at 10:53:16AM -0400, Peter Silva wrote:
> You are assuming the savings are substantial.  That's not clear.  When
> files are compressed, if you then start doing binary diffs, well it
> isn't clear that they will consistently be much smaller than plain new
> files.  it also isn't clear what the impact on repo disk usage would
> be.

The suggestion is for it to be opt-in, so packages can optimize their file
layout when opting in - for example, the gzipped changelogs and stuff can be
the rsyncable ones (this might need some changes in debhelper to generate
diffable files if the opt-in option is set).

> 
> The most straigforward option:
> The least intrusive way to do this is to add differential files in
> addition to the existing binaries, and any time the differential file,
> compared to a new version exceeds some threhold size (for example:
> 50%) of the original file, then you end up adding the sum total of the
> diff files in addition to the regular files to the repos.  I haven't
> done the math, but it's clear to me that it ends up being about double
> the disk space with this approach. it's also costly in that all those
> files have to be built and managed, which is likely a substantial
> ongoing load (cpu/io/people)  I think this is what people are
> objecting to.

I would throw away patches that are too large, obviously. I think
that twice the amount of space seems about reasonable, but then we'll
likely restrict that to

(a) packages where it's actually useful
(b) to updates and their base distributions

So we don't keep oldstable->stable diffs, or unstable->unstable
diffs around. It's questionable if we want them for unstable at
all, it might just be too much there, and we should just use them
for stable-(proposed-)updates (and point releases) and stable/updates;
and experimental so we don't accidentally break it during the
dev cycle.

> 
> A more intrusive and less obvious way to do this is to use zsync (
> http://zsync.moria.org.uk/. ) With zsync, you build tables of content
> for each file, using the same 4k blocking that rsync does. To handle
> compression efficiently, there needs to be an understanding of
> blocking, so a customized gzip needs to be used.  With such a format,
> you produce the same .deb's as today, with the .zsyncs (already in
> use?) and the author already provides some debian Packages files as
> examples.  The space penalty here is probably only a few percent.
> 
> the resource penalty is one read through each file to build the
> indices, and you can save that by combining the index building with
> the compression phase.  To get differential patches, one just fetches
> byte-ranges in the existing main files, so no separate diff files
> needed.  And since the same mechanisms can (should?) be used for repo
> replication, the added cost is likely a net savings in bandwidth
> usage, and relatively little complexity.
> 
> The steps would be:
>-- add gzip-ware flavour to package creation logic
>-- add zsync index creation on repos. (potentially combined with first 
> step.)
>-- add zsync client to apt-* friends.
> 
> To me, this makes a lot of sense to do just for repo replication,
> completely ignoring the benefits for people on low bandwidth lines,
> but it does work for both.

It does not work for client use. apt by default automatically deletes
packages files after a successful install, and upgrades are fairly infrequent
(so an expiration policy would not help much either); so we can't rely on old
packages to be available for upgradexs.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Proposal: A new approach to differential debs

[Julian Andres Klode]

Hi everyone,

(I CCed -devel and deity, but we probably should just discuss
 that on -dpkg)

while breakfast here at DebConf, the topic of delta upgrades
came up. I think delta debs are generally a thing we should
aim to have, but debdelta is not the implementation we want:

* It is not integrated into APT or dpkg
* It relies on a script shipped in the debdelta to generate
  a new deb from and old deb

We guessed that generating the new deb from the old deb is
actually the slowest part of the whole debdelta process. I
propose a new solution, which does not really have a name
yet.

# Scope
Patching installed packages to match new state

# Not scope

Generating new package from old package

# Design

Introduce a new format 'ddeb' (delta/differential deb)[1], the
format shall contain a control.tar member and a version member
as in a .deb file. Instead of a data.tar member, it contains
a diff.tar member, however.

The .diff.tar member contains patches to apply to individual
files of the old package. No idea about specific algorithm
to choose here, yet.

The control.tar's control file is extended with an Old-Version
field, so we get some sanity checking. Alternatively, it may
be extended with an mtree of the source we are patching.

The delta files are stored alongside their debs, and referenced
in the packages files in a Patches- section:

Patches-SHA256:
   

APT can then grab the delta, if it fails with a 404, it would
fall back to the full delta. 

The deltas are not incremental, my suggestion is to do the following ones:

unstable: (1) against testing (2) against previous unstable
experimental: (1) against unstable (2) against previous experimental
stable:   (1) against last point release (2) against previous security update
  (or rather current one)

All these files will always be around when dak runs anyway, so we do not
need to keep additional historical packages around.

We probably want to make this opt-in, so packages set a field like
Generate-Patches: yes (there might be problems with applying patches to
live systems and bad maintainer scripts).

[1] name should probably change

# Requisites / Extensions to .deb and dpkg database

We need to keep data about the file tree in packages and
in the database, from what I gathered, there already is
a plan to use mtree for that, so that would fit us well.

# Applying a delta deb

Maintainer scripts are run as normally from the
control tarball. The unpack phase is different: Instead
of unpacking a file  from the tarball into .dpkg-new,
we apply archive's  as a patch to the installed 
and store the result in .dpkg-new.

Files not listed in the new mtree but listed in the old
one will be deleted.

# Issues

We need a way to check if we can apply the diff.tar member; and
if we can't, have to download the full deb in APT. This might need
some kind of new patch-prepare state where we basically generate the
.dpkg-new files, but don't apply them (or run any maintscripts).

# Further work

I guess I should do a proof of concept and then we can look if that
is worthwhile, and how it performs.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: Proposal: A new approach to differential debs

[Julian Andres Klode]

On Sun, Aug 13, 2017 at 08:24:27PM -0400, Peter Silva wrote:
> o in spite of being the *default*, it isn't that universal, and in
> any event, we can just decide to change the default, no? One can say
> to people with bandwidth limitations, that their apt settings should
> not delete packages after receipt, so that they can be used as the
> basis for updates.  And these types of settings would appear to be
> rather common already, so it isn't a huge change.

We also have to consider embedded devices. I was told there are small
embedded devices with really slow connections around that run debian
and automatically update using apt - we don't want those to have to
keep around old packages either.

And quite practically speaking, I do not want to keep around packages
on my system, but I do want deltas, because at 10 Mbit/s, it's a pain
to upgrade. I also want to keep my /var small, so I have actually usable
space rather than duplicate of files.

But since deltas require implementation in dpkg, it's not my
decision anyway, it's guillem's. I'm just proposing ideas that
seem reasonable.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: Evaluation (Re: Proposal: A new approach to differential debs)

[Julian Andres Klode]

On Tue, Aug 22, 2017 at 10:53:47PM +0200, Sebastian Andrzej Siewior wrote:
> On 2017-08-16 00:21:09 [+0200], Julian Andres Klode wrote:
> > libreoffice-core (size only):
> > 
> >   -rw-r--r-- 1 jak jak 29M Jul 22 20:02 
> > libreoffice-core_5.3.5~rc1-3_amd64.deb
> >   -rw-r--r-- 1 jak jak 31M Jul 16 00:10 
> > libreoffice-core_5.4.0~rc2-1_amd64.deb
> >   -rw-r--r-- 1 jak jak 31M Jul 28 18:29 libreoffice-core_5.4.0-1_amd64.deb
> >   -rw-r--r-- 1 jak jak  18M Aug 15 23:44 
> > libreoffice-core_5.3.5~rc1-3_5.4.0-1_amd64.pdeb
> >   -rw-r--r-- 1 jak jak 4.5M Aug 15 23:42 
> > libreoffice-core_5.4.0~rc2-1_5.4.0-1_amd64.pdeb
> > 
> > For 5.4~rc2 to 5.4 it made a huge difference, for 5.3.5~rc1 to 5.4 not so 
> > much,
> > so it probably is a good fit for stable updates, but not for unstable and 
> > testing.
> 
> I would say it depends on the software. Some differs more than other.

True.

> 
> > firefox (size & performance):
> > 
> >  -rw-r--r-- 1 jak jak 2.3M Aug 15 20:59 firefox_55.0-1_55.0-2_amd64.debdelta
> >  -rw-r--r-- 1 jak jak 2.4M Aug 15 22:13 firefox_55.0-1_55.0-2_amd64.pdeb
> 
> and bsdfiff of old data.tar vs new data.dat 2.3MiB which is probably
> what debdelta does. So I am not sure if it is worth the extra mile to
> diff file by file. It might be faster and less memory hungry however.

debdelta also does some file thing, but not sure. We definitely want
to go per file, as we will not generate a new tar but directly patch
the installed files (basically, changing the code that unpacks a file
to .dpkg-new to apply the delta to the old file and write the result
to .dpkg-new) - you might not even have to read (all) the old files.

> 
> > # Further extensions
> > 
> > If you put a pristine-tar delta into the delta file, you can fully
> > reconstruct debs.
> 
> There is deltarpm [0] which does the same thing for rpm/fedora. I tried
> to find a design document but it seems that there is none. It seems
> however they grab all the binaries and make a delta via in-tree copy of
> bsdiff (delta.c). They also have intree-zlib but I didn't look why…
> I was mostly interrested if they have some filters to replace the
> offsets in binaries with something else but didn't find anything yet.

Right. We do have our own fork of bsdiff as well now, it's temporarily
known as jkdiff (https://github.com/julian-klode/jkdiff). The goal is
of course to turn this into a re-usable library and frontend programs,
and not limit it to "just" dpkg.

I did two changes compared to bsdiff:

* Instead of storing three different control (tuples of delta length,
  extra length, and an amount to seek), delta (sumwise difference
  of bytes), and extra (extra bytes to write); it now stores the tuple
  directly followed by the diff data and the extra data.

  This makes the patches streamable - we don't have to read from
  three different positions at the same time. Which is important for
  us because we can now apply the deltas without even extracting
  the individual diffs (which is important due to the next point).

* jkdiff files are uncompressed, bsdiff uses bzip2 on the control,
  delta, and extra blocks. We will store the files in an xz compressed
  tarball, so not compressing them first saves time and space. Also,
  xz is faster than bzip2 (or even gzip) when decompressing AFAICT.

The diff tool is still based on bsdiff, but jkpatch was completely
written from scratch and should be really easy to read. The jkpatch
tool applies diffs using SIMD instructions (using GCCs vector_size
attribute), which improved user time by about 50%.

Also, I applied the following changes for the generator:

* Debian: Do not support large files (> 2GB) - For large files, we need
  (9 * old + 1 * new) memory (essentially because we build a suffix
  offset array, so sizeof(off_t) * input size), so at 2GB that would
  be about 20GB of  RAM needed already. By only supporting small files,
  we reduce the memory overhead to 5 * old + new, so for the maximum of
  2 GB we only need 12 GB of RAM.
 
  This only affects generation though. The patches still use 64 bit
  values, and the patch tool supports large files.

* Chromium changes:
  - Replace embedded qsufsort with linking to libdivsufsort - much
faster diff generation (for libreoffice it went from 93 seconds
to 5 seconds), especially in some pathological
cases
  - Some more fixes for pathological cases
  (See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=409664#44)


I'll be writing a blog post and/or email with more details
shortly.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Bug#802241: please store the hash of the installed .deb and allow to query it

[Julian Andres Klode]

On Sun, Oct 18, 2015 at 06:20:01PM +, Mattia Rizzolo wrote:
> Package: dpkg
> Version: 1.18.3
> Severity: wishlist
> X-Debbugs-CC: reproducible-bui...@lists.alioth.debian.org
> 
> Hi dpkg people,
> 
> in the context of allowing to recreate the same build-environment of a
> past build we would need to know which packages where installed.
> Currently we rely on (pkgname, arch, version) tuples to uniquely
> identify a binary package, but as you can easily imagine this is not
> unique at all, definitly not in the multi distro universe, possibly not
> even across suites.

I also want this for delta debs, to identify local rebuilds being
installed, and prevent delta installation failure in such cases.

> 
> To me it seems that:
> * we are mostly interested in the hash of the whole container: all the
>   use cases highlighted above would require this;
> * If ↑ then the hash can't be pre-computed and stored inside the
>   container.

Practically speaking, for your use case you only need a hash of the
file tree. My proposal for a package id is to use the md5sum of
DEBIAN/md5sums. This can be stored in DEBIAN/control in an id
field and generated at build time. 

We can also use cat DEBIAN/md5sums DEBIAN/control | md5sum (without an
Id field in control) as the ID, and then append that to control. This
means that dependency relations and stuff is included as well. That's
useful for the solver use case; but it's not really relevant for
the reproducible build use case - dependencies on the installed
system, description, etc should not matter for you.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Bug#802241: please store the hash of the installed .deb and allow to query it

[Julian Andres Klode]

On Sat, Aug 26, 2017 at 03:16:52PM +0200, Mattia Rizzolo wrote:
> On Sat, Aug 26, 2017 at 02:14:16PM +0200, Julian Andres Klode wrote:
> > I also want this for delta debs, to identify local rebuilds being
> > installed, and prevent delta installation failure in such cases.
> 
> yay another user!
> 
> > > To me it seems that:
> > > * we are mostly interested in the hash of the whole container: all the
> > >   use cases highlighted above would require this;
> > > * If ↑ then the hash can't be pre-computed and stored inside the
> > >   container.
> > 
> > Practically speaking, for your use case you only need a hash of the
> > file tree. My proposal for a package id is to use the md5sum of
> > DEBIAN/md5sums. This can be stored in DEBIAN/control in an id
> > field and generated at build time. 
> 
> That's not true, as we need the hash also (for example) of all the
> maintainer scripts which are not in data.tar (I assume that's what you
> meant by "file tree").  Also, we have seen packages where the only
> difference is the order of entry in the md5sums file, therefore making
> the build not reproducible by our (higher than policy) standards.
> We really need the whole container.

I don't see why you need maintainer scripts. When you are building
a package what you care about is the state you are building in. And
the maintainer scripts are irrelevant in that state. But well, you could
hash them too.

> 
> > We can also use cat DEBIAN/md5sums DEBIAN/control | md5sum (without an
> > Id field in control) as the ID, and then append that to control. This
> > means that dependency relations and stuff is included as well. That's
> > useful for the solver use case; but it's not really relevant for
> > the reproducible build use case - dependencies on the installed
> > system, description, etc should not matter for you.
> 
> Well, DEBIAN/control contains the dependencies generated during the
> package build, and we do are interested in them as well…
> In short: we do care about both data.tar and control.tar.  After all, we
> do compare the hashes of the final .deb container.

I fail to see why you'd be interested in dependencies. Your stated use
case is "allowing to recreate the same build-environment of a past build
we would need to know which packages where installed.". To recreate a 
build environment you do not have to care about dependencies, nor do
you have to care about maintainer scripts (as soon as you involve
upgrades, the result might be different anyway, if you always bootstrap
the exact tree, it might be useful). Because the matter of fact is that
neither maintainer scripts nor dependencies affect how a package is
built (once build-depends are installed).

There's also the question of conffiles. They might affect the environment,
but: The actually installed ones might be different from the ones in the
package (which essentially is the same upgrade problem you have for
maintainer scripts); hence it's not really useful to include them in
the hash (but it does not hurt either).

> As I saw it when I originally thought of the problem the only sane
> solution to this for me would be to have dpkg compute the hash of the
> .deb before unpacking it, and store in it's $admindir/status file, but
> that makes the installation process very CPU-intensive, to the point
> that very probably it's too much to be bareable in many systems.

If you really want everything, the wise choice is to just hash the
entire tree dpkg-deb is packaging up and then add that to the ID
field. You can easily reconstruct the ID by unpacking the package,
removing the ID field from debian/control and rehashing.

Once mtree is in, this also includes stuff like permissions, xattrs.
Then we can also SHA256 everything and use that as an ID.

The only thing not covered by this is the layout of the tar files,
the compression, and the layout of the final ar file. But that's
not really relevant to any of our use cases.

For APT, we specifically need the ID to be in the package and dpkg
not to add any missing IDs, otherwise we cannot rely on the ID as
the installed one might have one but the remote one not.

For deltas, we only care that it has *some* ID, for what it's worth,
it could be a random UUID (but that's not reproducible). I do need
that to be in debian/control, as this will allow us to change the
ID algorithm at any point in time and not require us to recompute
the id when generating the delta.

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Bug#878110: dpkg-buildpackage: Generate _arch.changes for all-only packages

[Julian Andres Klode]

Package: dpkg-dev
Version: 1.18.24
Severity: minor
File: /usr/bin/dpkg-buildpackage

If a package is Architecture: all only (or I guess has some non-all binaries
that don't build for the current arch), then building the package produces a
_arch.changes, rather than an _all.changes.

This is slightly underdocumented in the manual page, it says:
"for  a  build  that  includes  any  the  name  will  be  
source-name_source-version_arch.changes"

without mentioning that build actually means --build. 

It might be a better idea to produce _all.changes files if no non-all binaries
are produced though (and _all buildinfo files).

-- System Information:
Debian Release: buster/sid
  APT prefers unstable
  APT policy: (900, 'unstable'), (900, 'testing'), (500, 'unstable-debug'), 
(100, 'experimental'), (1, 'experimental-debug')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.13.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_IE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages dpkg-dev depends on:
ii  binutils  2.29.1-4
ii  bzip2 1.0.6-8.1
ii  libdpkg-perl  1.18.24
ii  make  4.1-9.1
ii  patch 2.7.5-1+b2
ii  perl  5.26.0-8
ii  tar   1.29b-2
ii  xz-utils  5.2.2-1.3

Versions of packages dpkg-dev recommends:
ii  build-essential  12.4
ii  clang-3.8 [c-compiler]   1:3.8.1-24
ii  fakeroot 1.22-1
ii  gcc [c-compiler] 4:7.2.0-1d1
ii  gcc-5 [c-compiler]   5.4.1-14
ii  gcc-6 [c-compiler]   6.4.0-7
ii  gcc-7 [c-compiler]   7.2.0-8
ii  gnupg2.2.1-1
ii  gnupg2   2.2.1-1
ii  gpgv 2.2.1-1
ii  gpgv22.2.1-1
pn  libalgorithm-merge-perl  
ii  tcc [c-compiler] 0.9.27~git20161217.cd9514ab-3

Versions of packages dpkg-dev suggests:
ii  debian-keyring  2017.08.28

-- no debconf information

-- 
Debian Developer - deb.li/jak | jak-linux.org - free software dev
  |  Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline').  Thank you.

Re: RFC: Support for zstd in .deb packages?

[Julian Andres Klode]

On Tue, May 01, 2018 at 10:36:34AM +0200, Marco d'Itri wrote:
> On Apr 27, Julian Andres Klode <j...@debian.org> wrote:
> 
> > Our major use case is cloud initial setup, image building, CI, buildds, all
> > of which do not require any syncs, and can safely use eatmydata, for 
> > example;
> > hence the enormous speed up.
> I do not believe that it would be wise to optimize our packaging system 
> for the niche target of package development.
> 
> In my experience as a cloud infrastructure provider, new systems are 
> cloned/instantiated from golden images and not from debootstrap or d-i.

Well, yes, you usually have _some_ cloud-provided image. But usually, people
will use cloud-init to initialize their instances and install packages from
in there, often using eatmydata.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: RFC: Support for zstd in .deb packages?

[Julian Andres Klode]

On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote:
> (ZSTD)
> 
> On Fri, Apr 27, 2018 at 07:02:12AM +0200, Guillem Jover wrote:
> > Recently Julian mentioned it again on IRC, and we each started
> > implementing support in dpkg and apt respectively, to allow easier
> > evaluation. I stopped when I realized the code was getting too messy,
> > but after few weeks Julian and Bálint Réczey ended up implementing
> > the support for apt and dpkg [D], so that they could merge it in
> > Ubuntu for their upcoming release, which they did.
> > 
> > Their main driving reason (AFAICT) has been (de)compression speed.
> 
> With default level:
> 
>   compdecomp  size
> xz8.038s  0.356s  4320292
> bz2   2.265s  0.730s  5234516
> zst   0.274s  0.102s  5657626
> gz0.880s  0.152s  6515505
> Z 0.499s  0.133s  8932459
> lzo   0.100s  0.095s  9198874
> 
> _Compression_ speed is insane; decompression is merely nice.  Obviously,
> tuneable level turns this single point into an envelope, but even xz at its
> lowest levels can be multiple times faster than gzip while delivering better
> compression.

We only care about zstd at level 19 for dpkg and apt.

> Don't.  For .debs, that is.
> 
> A .deb that is getting processed goes through dpkg, whose status files
> writes and all those fsyncs are so slow that it's pointless to optimize
> every last bit of decompression.  If someone already goes as far as
> recompressing packages, we got two fine choices: "xz -0" and "none"; the
> former being faster than gzip while compressing better, and the latter being
> as fast as cat.  You don't get any further than cat on the size-vs-speed
> slider...

Our major use case is cloud initial setup, image building, CI, buildds, all
of which do not require any syncs, and can safely use eatmydata, for example;
hence the enormous speed up.

There's still a speed up with syncs, but it's less than 10% IIRC. This will
get faster over time as SSDs increase their IOPS.

> 
> On the other hand, adding .tar.zst [2] would be a much welcome addition for
> files referenced from .dsc.  A machine that installs dpkg-dev has enough
> storage that a half-MB library is beneath any notice, and zstd is a fine
> replacement for gzip and the likes.

this was backed out at guillem's request.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: RFC: Support for zstd in .deb packages?

[Julian Andres Klode]

On Fri, Apr 27, 2018 at 02:01:44PM +0200, Adam Borowski wrote:
> On Fri, Apr 27, 2018 at 01:45:07PM +0200, Adam Borowski wrote:
> > Don't.  For .debs, that is.
> 
> Scratch that.
> 
> apt Depends: libapt-pkg5.0 Depends: libzstd1
> 
> While apt is "merely" priority:required rather than fully essential, a
> Debian system without apt is so deeply embedded it already requires special
> steps, thus there's probably no reason to bother.
> 
> If apt has already taken the plunge, it's reasonable for dpkg to follow.
> The "reduced essential set" guys will be unhappy, but as we're already
> there, it's good to switch other users which need a general-purpose fast
> good compressor (the "slow but strong" slot is providen by xz, "weak but
> extremely fast" by lz4 -- libzstd happens to include a lz4 implementation).

I specifically called it experimental, and it might not be part of buster
if it turns out to be not useful.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Frontend locking in APT clients

[Julian Andres Klode]

On Mon, Jun 18, 2018 at 08:19:17PM +0200, Julian Andres Klode wrote:
> Hi folks,
> 
> With frontend locking in dpkg git, I think it's time I clear up
> some potential confusion as to how this is supposed to work in the
> APT world.
> 
> The idea is that the current _system->Lock() / apt_pkg.SystemLock
> / apt_pkg.pkgsystem_lock() will start to manage _both_ lock-frontend
> and lock, and new methods LockInner() and UnlockInner() will be
> provided to release "lock". Code running dpkg will need to call
> those around dpkg runs, rather than the generic Lock ones.
> 
> python-apt is currently broken in that you need to release the lock
> prior to calling commit() on an apt.Cache. This will change soon
> - no unlocking will be needed. A python-apt (>> 1.7~alpha1~) client
> should behave as following:
> 
> Basically, add Depends: python-apt (>> 1.7~alpha1~), and do:
>   with apt_pkg.SystemLock():
> main()
> - forget about locks if you don't invoke dpkg manually - do not
> release that, ever. If you do run dpkg manually do it like this:
> 
>   with apt_pkg.NoInnerLock():
> subprocess.check_call(["dpkg", "--configure", "-a"])

I want to note that apt in its current merge request state, 
automatically sets DPKG_FRONTEND_LOCKED when invoking dpkg
if the system lock is acquired (after the fork, directly
before the execvp). If you run dpkg yourself, like for calling
dpkg --configure -a you will probably have to set the environment
variable yourself.

So you'll need:

  with apt_pkg.NoInnerLock():
os.putenv("DPKG_FRONTEND_LOCKED", "1")
subprocess.check_call(["dpkg", "--configure", "-a"])


I thought about making Lock() set the variable and unlock unset
it, but I'm not sure about implicatiomns wrt other subprocesses
like hooks.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Frontend locking in APT clients

[Julian Andres Klode]

Hi folks,

With frontend locking in dpkg git, I think it's time I clear up
some potential confusion as to how this is supposed to work in the
APT world.

The idea is that the current _system->Lock() / apt_pkg.SystemLock
/ apt_pkg.pkgsystem_lock() will start to manage _both_ lock-frontend
and lock, and new methods LockInner() and UnlockInner() will be
provided to release "lock". Code running dpkg will need to call
those around dpkg runs, rather than the generic Lock ones.

python-apt is currently broken in that you need to release the lock
prior to calling commit() on an apt.Cache. This will change soon
- no unlocking will be needed. A python-apt (>> 1.7~alpha1~) client
should behave as following:

Basically, add Depends: python-apt (>> 1.7~alpha1~), and do:
  with apt_pkg.SystemLock():
main()
- forget about locks if you don't invoke dpkg manually - do not
release that, ever. If you do run dpkg manually do it like this:

  with apt_pkg.NoInnerLock():
subprocess.check_call(["dpkg", "--configure", "-a"])

or instead of context managers, use the functions
pkgsystem_{un,}lock{,_inner}.

This will ensure that your apt client will be safe against
any other apt client, and any other client implementing frontend
locking. It will not be safe against other frontends that acquire
the dpkg lock themselves, those will need fixing too. It will however,
be safe against concurrent runs of dpkg by users or frontends not
implementing locking.

Thanks,
Julian
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Bug#887629: libc6: bad upgrade path: libexpat1 unpacked and python3 called before libc6 unpacked

[Julian Andres Klode]

On Thu, Jan 18, 2018 at 08:38:02PM +0100, Julian Andres Klode wrote:
> On Thu, Jan 18, 2018 at 06:41:52PM +0100, Aurelien Jarno wrote:
> > control: reassign -1 apt,dpkg
> > control: affects -1 libc6
> > control: affects -1 libexpat1
> > 
> > On 2018-01-18 15:53, Andreas Beckmann wrote:
> > > Package: libc6
> > > Version: 2.26-2
> > > Severity: serious
> > > User: debian...@lists.debian.org
> > > Usertags: piuparts
> > > 
> > > Hi,
> > > 
> > > during a test with piuparts I noticed your package fails to upgrade from
> > > 'stretch'.
> > > It installed fine in 'stretch', then the upgrade to 'buster' fails.
> > > 
> > > >From the attached log (scroll to the bottom...):
> > > 
> > > [...]
> > >   Preparing to unpack .../libexpat1-dev_2.2.5-3_i386.deb ...
> > >   Unpacking libexpat1-dev:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> > >   Preparing to unpack .../libexpat1_2.2.5-3_i386.deb ...
> > >   Unpacking libexpat1:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> > 
> > $ apt-cache show libexpat1
> > Package: libexpat1
> > Source: expat
> > Version: 2.2.5-3
> > Installed-Size: 429
> > Maintainer: Laszlo Boszormenyi (GCS) <g...@debian.org>
> > Architecture: i386
> > Depends: libc6 (>= 2.25)
> > 
> > So libexpat1 correctly depends on libc6 (>= 2.25), which is not
> > even unpacked at that point.
> > 
> > > [...]
> > >   Preparing to unpack .../3-libglib2.0-dev_2.54.3-1_i386.deb ...
> > >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' 
> > > not found (required by /lib/i386-linux-gnu/libexpat.so.1)
> > >   dpkg: warning: subprocess old pre-removal script returned error exit 
> > > status 1
> > >   dpkg: trying script from the new package instead ...
> > >   dpkg: error processing archive 
> > > /tmp/apt-dpkg-install-wfemKS/3-libglib2.0-dev_2.54.3-1_i386.deb 
> > > (--unpack):
> > >there is no script in the new version of the package - giving up
> > >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' 
> > > not found (required by /lib/i386-linux-gnu/libexpat.so.1)
> > 
> > This failure is normal given libexpat1 requires the new libc which has
> > not been unpacked yet.
> 
> Yeah, well, it needs to Pre-Depend on it then I guess, if it's being used
> in preinst actions. The thing is that Depends only after postinst ordering,
> not unpack ordering.
> 

To be more precise: I guess libglib2.0-dev needs to predepend on python3 or on
libexpat1.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Bug#887629: libc6: bad upgrade path: libexpat1 unpacked and python3 called before libc6 unpacked

[Julian Andres Klode]

On Thu, Jan 18, 2018 at 06:41:52PM +0100, Aurelien Jarno wrote:
> control: reassign -1 apt,dpkg
> control: affects -1 libc6
> control: affects -1 libexpat1
> 
> On 2018-01-18 15:53, Andreas Beckmann wrote:
> > Package: libc6
> > Version: 2.26-2
> > Severity: serious
> > User: debian...@lists.debian.org
> > Usertags: piuparts
> > 
> > Hi,
> > 
> > during a test with piuparts I noticed your package fails to upgrade from
> > 'stretch'.
> > It installed fine in 'stretch', then the upgrade to 'buster' fails.
> > 
> > >From the attached log (scroll to the bottom...):
> > 
> > [...]
> >   Preparing to unpack .../libexpat1-dev_2.2.5-3_i386.deb ...
> >   Unpacking libexpat1-dev:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> >   Preparing to unpack .../libexpat1_2.2.5-3_i386.deb ...
> >   Unpacking libexpat1:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> 
> $ apt-cache show libexpat1
> Package: libexpat1
> Source: expat
> Version: 2.2.5-3
> Installed-Size: 429
> Maintainer: Laszlo Boszormenyi (GCS) 
> Architecture: i386
> Depends: libc6 (>= 2.25)
> 
> So libexpat1 correctly depends on libc6 (>= 2.25), which is not
> even unpacked at that point.
> 
> > [...]
> >   Preparing to unpack .../3-libglib2.0-dev_2.54.3-1_i386.deb ...
> >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' not 
> > found (required by /lib/i386-linux-gnu/libexpat.so.1)
> >   dpkg: warning: subprocess old pre-removal script returned error exit 
> > status 1
> >   dpkg: trying script from the new package instead ...
> >   dpkg: error processing archive 
> > /tmp/apt-dpkg-install-wfemKS/3-libglib2.0-dev_2.54.3-1_i386.deb (--unpack):
> >there is no script in the new version of the package - giving up
> >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' not 
> > found (required by /lib/i386-linux-gnu/libexpat.so.1)
> 
> This failure is normal given libexpat1 requires the new libc which has
> not been unpacked yet.

Yeah, well, it needs to Pre-Depend on it then I guess, if it's being used
in preinst actions. The thing is that Depends only after postinst ordering,
not unpack ordering.

> 
> >   dpkg: error while cleaning up:
> >subprocess installed post-installation script returned error exit status 
> > 1
> > [...]
> >   Preparing to unpack .../9-libc6_2.26-2_i386.deb ...
> >   Unpacking libc6:i386 (2.26-2) over (2.24-11+deb9u1) ...
> >   Errors were encountered while processing:
> >/tmp/apt-dpkg-install-wfemKS/3-libglib2.0-dev_2.54.3-1_i386.deb
> 
> And libc6 2.26-2 is only unpacked afterwards.

does not matter.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Bug#887629: libc6: bad upgrade path: libexpat1 unpacked and python3 called before libc6 unpacked

[Julian Andres Klode]

On Thu, Jan 18, 2018 at 06:41:52PM +0100, Aurelien Jarno wrote:
> control: reassign -1 apt,dpkg
> control: affects -1 libc6
> control: affects -1 libexpat1
> 
> On 2018-01-18 15:53, Andreas Beckmann wrote:
> > Package: libc6
> > Version: 2.26-2
> > Severity: serious
> > User: debian...@lists.debian.org
> > Usertags: piuparts
> > 
> > Hi,
> > 
> > during a test with piuparts I noticed your package fails to upgrade from
> > 'stretch'.
> > It installed fine in 'stretch', then the upgrade to 'buster' fails.
> > 
> > >From the attached log (scroll to the bottom...):
> > 
> > [...]
> >   Preparing to unpack .../libexpat1-dev_2.2.5-3_i386.deb ...
> >   Unpacking libexpat1-dev:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> >   Preparing to unpack .../libexpat1_2.2.5-3_i386.deb ...
> >   Unpacking libexpat1:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> 
> $ apt-cache show libexpat1
> Package: libexpat1
> Source: expat
> Version: 2.2.5-3
> Installed-Size: 429
> Maintainer: Laszlo Boszormenyi (GCS) 
> Architecture: i386
> Depends: libc6 (>= 2.25)
> 
> So libexpat1 correctly depends on libc6 (>= 2.25), which is not
> even unpacked at that point.
> 
> > [...]
> >   Preparing to unpack .../3-libglib2.0-dev_2.54.3-1_i386.deb ...
> >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' not 
> > found (required by /lib/i386-linux-gnu/libexpat.so.1)
> >   dpkg: warning: subprocess old pre-removal script returned error exit 
> > status 1
> >   dpkg: trying script from the new package instead ...
> >   dpkg: error processing archive 
> > /tmp/apt-dpkg-install-wfemKS/3-libglib2.0-dev_2.54.3-1_i386.deb (--unpack):
> >there is no script in the new version of the package - giving up
> >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' not 
> > found (required by /lib/i386-linux-gnu/libexpat.so.1)
> 
> This failure is normal given libexpat1 requires the new libc which has
> not been unpacked yet.

Yeah, well, it needs to Pre-Depend on it then I guess, if it's being used
in preinst actions. The thing is that Depends only after postinst ordering,
not unpack ordering.

> 
> >   dpkg: error while cleaning up:
> >subprocess installed post-installation script returned error exit status 
> > 1
> > [...]
> >   Preparing to unpack .../9-libc6_2.26-2_i386.deb ...
> >   Unpacking libc6:i386 (2.26-2) over (2.24-11+deb9u1) ...
> >   Errors were encountered while processing:
> >/tmp/apt-dpkg-install-wfemKS/3-libglib2.0-dev_2.54.3-1_i386.deb
> 
> And libc6 2.26-2 is only unpacked afterwards.

does not matter.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Bug#887629: libc6: bad upgrade path: libexpat1 unpacked and python3 called before libc6 unpacked

[Julian Andres Klode]

On Thu, Jan 18, 2018 at 08:38:02PM +0100, Julian Andres Klode wrote:
> On Thu, Jan 18, 2018 at 06:41:52PM +0100, Aurelien Jarno wrote:
> > control: reassign -1 apt,dpkg
> > control: affects -1 libc6
> > control: affects -1 libexpat1
> > 
> > On 2018-01-18 15:53, Andreas Beckmann wrote:
> > > Package: libc6
> > > Version: 2.26-2
> > > Severity: serious
> > > User: debian...@lists.debian.org
> > > Usertags: piuparts
> > > 
> > > Hi,
> > > 
> > > during a test with piuparts I noticed your package fails to upgrade from
> > > 'stretch'.
> > > It installed fine in 'stretch', then the upgrade to 'buster' fails.
> > > 
> > > >From the attached log (scroll to the bottom...):
> > > 
> > > [...]
> > >   Preparing to unpack .../libexpat1-dev_2.2.5-3_i386.deb ...
> > >   Unpacking libexpat1-dev:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> > >   Preparing to unpack .../libexpat1_2.2.5-3_i386.deb ...
> > >   Unpacking libexpat1:i386 (2.2.5-3) over (2.2.0-2+deb9u1) ...
> > 
> > $ apt-cache show libexpat1
> > Package: libexpat1
> > Source: expat
> > Version: 2.2.5-3
> > Installed-Size: 429
> > Maintainer: Laszlo Boszormenyi (GCS) <g...@debian.org>
> > Architecture: i386
> > Depends: libc6 (>= 2.25)
> > 
> > So libexpat1 correctly depends on libc6 (>= 2.25), which is not
> > even unpacked at that point.
> > 
> > > [...]
> > >   Preparing to unpack .../3-libglib2.0-dev_2.54.3-1_i386.deb ...
> > >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' 
> > > not found (required by /lib/i386-linux-gnu/libexpat.so.1)
> > >   dpkg: warning: subprocess old pre-removal script returned error exit 
> > > status 1
> > >   dpkg: trying script from the new package instead ...
> > >   dpkg: error processing archive 
> > > /tmp/apt-dpkg-install-wfemKS/3-libglib2.0-dev_2.54.3-1_i386.deb 
> > > (--unpack):
> > >there is no script in the new version of the package - giving up
> > >   /usr/bin/python3: /lib/i386-linux-gnu/libc.so.6: version `GLIBC_2.25' 
> > > not found (required by /lib/i386-linux-gnu/libexpat.so.1)
> > 
> > This failure is normal given libexpat1 requires the new libc which has
> > not been unpacked yet.
> 
> Yeah, well, it needs to Pre-Depend on it then I guess, if it's being used
> in preinst actions. The thing is that Depends only after postinst ordering,
> not unpack ordering.
> 

To be more precise: I guess libglib2.0-dev needs to predepend on python3 or on
libexpat1.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Delta debs repository layout v2

[Julian Andres Klode]

This is an advanced proposal for integrating deltas into the archive. It has
been designed after some feedback and some more research.

Criteria

* We don't want to have Deltas indexes like Packages files
  -> we are unlikely to need most of the deltas
  -> about same size as Packages file (when both compressed)
* We don't want to try to fetch deltas for each upgrade. We'll only be 
generating
  deltas for about 2/3rd of all upgrades (as far as current results tell us.
* We don't want to have untrusted deltas/files
* We don't want to generate/verify too many signatures

Terminology
===
* upgrade: A tuple of two .deb; an old one, and a new one.
* delta: A deb containing only ddelta files or similar instead of full files
* bloom filter: A bitarray of $n$ bytes indexed by $k$ hash functions. Has
  no false negatives, but might have false positives.

Layout
==
This approach tries to solve the problem by introducing per-source-package
Delta indices in the pool. This solves the problem of having to download meta
info about all deltas in the archive.

* pool/$component/$prefix/$sourcename/Deltas

  An inline signed index containing, for each delta for that source package,
  the following information:

* Package   - the name of the package
* Old-ID- the id of the package being upgraded from
* New-ID- the id of the version being upgraded to
* Size  - the size of the delta
* SHA256- the hash of the delta
* Algorithm - algorithm used in deltas, currently ddelta

  Including the algorithm is essential, so we can switch algorithms at a later
  point and have old dpkg versions download a delta they understand or no delta.

  Deltas are stored with a fixed pathname to avoid duplicating information:


pool/$component/$prefix/$sourcename/$binaryname_$oldid_$newid_$algorith.deltadeb

  The concrete layout of the Deltas file is unspecified so far. A minimal 
solution is:

SHA256:
 8a09ba1ca92fe5405de7bd90fb06a1ce6cc502917df88f2990d3701a6cc75fa1 590012 
apt_1e9bfd18a899677e70fda42f5192fd169c4a778d7c7426016d1bf0a0d9b5dbc7_1955901c3d7dd72c451274fa72f42539e573a9e883c4e05ad283e2a30a62f13b_ddelta.deltadeb

  this encodes all the information in a fairly minimal way (thus we don't need 
any compression), 
  and is compatible with the existing Release file format. Note that we will 
know the name(s) of the file's we want
  to download beforehand, we just need to figure out if they exist and get 
signed hashes
  and sizes for them.

  The package name is in fact optional, but it might be worth including it to 
be able
  to figure out which package the delta belongs to.

  It might be worth including the distribution in the filename, and call it
  e.g. Deltas.bookworm-security instead of Deltas. That said, there might not be
  enough releases at a time for it to be worth it. NEEDSINVESTIGATION

However, if we only produce deltas for 2/3 of the upgrades, we still have 1/3 of
false positives for which we will needlessly download Deltas indices. Depending 
on
the connection's latency, this might be bad. To solve this, we can introduce a 
new
file:

* dists/$dist/$component/binary-$arch/Deltas.bloom

  A bloom filter of size $n$ bits. For each delta from $old_id to
  $new_id, construct a delta id $old_id | $new_id (the concatenation). Split
  up the delta_id into chunks of $b$ bits. Use each $chunk \mod n$ as an index
  into the bloom filter.

  It seems reasonable to expect us to achieve a bloom filter with a false
  positive rate of about 1%. Experimental results for xenial-updates/main/amd64 
have
  shown a 0.75% false positive rate for a bloom filter of $n=98317$ bits
  and $b=32$ bit chunks for 9132 accepted out of 13500 total deltas (that's 
about
  12 KB for a Packages.xz of 800 KB).

  This does not include the delta algorithm, hence it's probably best not to 
have
  deltas with different algorithms for the same upgrade.

  The bloom filter is signed by the InRelease file.

Delta upgrade process
=

update
--
Acquire Deltas.bloom for each Packages file

upgrade/install - download
--
  for each upgrade in changes:
if upgrade in Deltas.bloom and applicable(upgrade):
  queue upgrade's Deltas index
else:
  queue full deb

  for each deltas index:
if downloaded and delta in index:
  queue delta
else:
  queue deb

  for each delta:
if downloaded failed:
  queue deb

where applicable(upgrade):
  """Verify that an upgrade is applicable.

  For an upgrade to be applicable, the files on the file system must match the 
files
  shipped in the package. A reasonable approximation is to ensure that the file 
has
  not been modified later than the install time and that the size is the 
same."""

  for each file in installed_version:
if file.mtime >= installed_version.install_time or file.mtime > file.ctime 
or file.size != expected_size:
  return False
  return True


Other 

Re: Bug#919543: apt: when installing with deb file, prerm maintainer script doesn't pass new-package-name before it's replaced due to conflict

[Julian Andres Klode]

On Thu, Jan 17, 2019 at 11:03:01AM +0800, Allen wrote:
> Package: apt
> Version: 1.2.19
> Severity: normal
> 
> Dear Maintainer,
> 
> 
> I am creating a deb packages which will replace another package. And before 
> the
> old package are removed, I want to check whether the package is remove due to
> **replace** operation or a simple **uninstall** operation.
> 
> In man page of `deb-prerm`, I found that when a package is replaced due to
> conflict, the prerm script of old package can be called in the following way:
> `prerm remove in-favour new-package new-version`.
> 
> Therefore, I add some script in prerm of old package and opreate with the 
> shell
> script variables. If I use `dpkg` to install these .deb packages, it will work
> perfectly. However, it does not pass any new package information if I use 
> `apt`
> to install these packages.
> 
> Is it an existing bug? I have searched for a while and have not found any
> relevant content.

So, I don't think this ever worked. It's best to ignore this exists. While it
is documented, it cannot really happen in practice, as apt first does the remove
and then the install; in two separate invocations of dpkg.

It's unlikely that this will change in the near future, either.

It's also completely unreliable in case you have multiple conflicts across
packages, so I think we should just get rid of it completely.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Bug#908747: Default -I and -i option should not exclude .ignore

[Julian Andres Klode]

On Thu, Sep 13, 2018 at 12:26:27PM +0100, Ian Jackson wrote:
> Package: dpkg-dev
> Version: 1.19.0.5
> 
> When the source provided to dpkg-source contains a .ignore file,
> eg a .gitignore, .hgignore, .cvsignore, then that file is part of the
> source code as the maintainer works with it.  It should be retained in
> the source package.
> 
> Likewise if the Debian maintainer makes changes to .ignore in
> their vcs, then those changes are part of the source as the maintainer
> works with it, and in `3.0 (quilt)' should be represented as a patch.
> 
> The result of this default is that many source packages in the Debian
> archive are incomplete.  This is IMO a DFSG violation (and where
> relevant a GPL violation).  Although it is probably legally de
> minimis, this should be fixed.

By the same reason, you could argue that not shipping .git is a DFSG/GPL
violation. ignore files are things that integrate the source code with
the version control system.

Heck, you could even argue that the entire disk of the maintainer is
part of the source code with that definition.


-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: RFC: Standardizing source package artifacts build paths

[Julian Andres Klode]

On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote: 
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. [...]
[...]
> The use of a hidden directory is to reduce clutter and stomping over any

Love the hidden directory.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Fwd: Bug#956931: autopkgtest: Build profiles support for autopkgtest

[Julian Andres Klode]

On Thu, Apr 16, 2020 at 11:04:02PM +0200, Jiri Palecek wrote:
> 
> 
> 
>  Forwarded Message 
> Subject:  Bug#956931: autopkgtest: Build profiles support for autopkgtest
> Resent-Date:  Thu, 16 Apr 2020 20:42:01 +
> Resent-From:  Jiri Palecek 
> Resent-To:debian-bugs-d...@lists.debian.org
> Resent-CC:jpale...@web.de, Debian CI team 
> Date: Thu, 16 Apr 2020 22:38:07 +0200
> From: Jiri Palecek 
> Reply-To: Jiri Palecek , 956...@bugs.debian.org
> To:   Debian Bug Tracking System 
> 
> 
> 
> Package: autopkgtest
> Version: 5.12.2~6.gbp89ad39
> Severity: wishlist
> 
> Dear Maintainer,
> 
> with the latest and greatest changes in dpkg, I think it would be nice
> if autopkgtest followed suit. Particularly, it would be advantageous to
> support running and building tests in autopkgtest under build profiles
> (esp. nodoc!). This would allow for smaller test footprint, less
> packages to pull (ie. you don't need texlive on the testbed), and faster
> building of the packages.
> 
> I prepared a patch implementing such a change here:
> https://salsa.debian.org/jpalecek-guest/autopkgtest/-/commit/6275da59305d6e836cb3558f9f442479eb24fc95
> 
> The patch is functional, albeit incomplete due to missing documentation.
> 
> The real issue is the control file format. In my patch, I use an extra
> stanza to specify build profiles, like this:
> 
> Build-Profiles: nodoc
> 
> Tests: run-some-tests
> <<<
> 
> I chose this format, because adding the specification to some of the
> tests would be IMHO misleading: the build profile applies to all of the
> tests indiscriminately, not to any particular one. Also, I chose to
> apply them to all @builddep@ dependencies as well.
> 
> However, there is a problem about this: it makes the control file format
> non-backwards-compatible. Particularly, dpkg needs to be patched or it
> will croak on packages with such d/t/control. Maybe, some artificial
> Tests: line like
> 
> Tests: *
> 
> could be added? That would make the change backwards compatible. Dpkg
> still needs to be patched, but the change would be rather minimal.
> 
> What do you think about this proposal? Please comment here or on
> salsa. I'm also CC-ing the dpkg developers, since it concerns them.

I understand the motivation for testing in-archive, but this creates
a problem for local testing of unbuilt packages. We want to test as
close to the archive as possible, meaning that the packages we test
should be built with the same build profile as in the archive.

I'm not sure how this is currently handled, but I'd really not
want to go through the trouble of building twice, once without
profiles to get the .debs, and then with build profiles for tests
with build-needed.

And of course, I don't want to just build with those build profiles,
because then it's different from the archive, and I don't get the
guarantees I want (namely, I want to be reasonably certain that
if I run autopkgtest on my .dsc, that it will built and test
successfully in the archive as well; including docs, obviously).

So, in summary, I think it's not a good idea.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: dpkg Broken in Debian Buster

[Julian Andres Klode]

On Mon, May 11, 2020 at 07:33:00AM +1200, Darren Hale wrote:
> Hi Guillem,
> 
> I have been advised that the problem is with the behavior of su
> changed when they moved it to a different source package and a fix is
> to old behavior is to put ALWAYS_SET_PATH yes in /etc/default/su.
> (create the file if it doesn't exist).

I'm not sure why you feel the need to explain back to guillem the
very thing he told you about.

In any case, you can also just use apt to install stuff, which
is a lot safer and also sets PATH.
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Optional Build-Depends

[Julian Andres Klode]

Hi,

we were just talking in #debian-dpkg about optional build-deps. guillem
believes that the release team is in the best position to specify if this
is reasonable or not, so here we go:

We have came up with a syntax, one goal being to break parsers and not
silently ignore optional deps:

  Build-Depends: foo? (>= 1) | baz

The behavior being:

   If foo resolves to a valid package name, this is a normal
   dependency. So if it's like version 0.9, the dependency would
   be unsat/depwait

   For tools stripping alternatives, which I think buildds do,
   it becomes slightly more complex, as they need to check if
   foo exists:

 foo exists => drop `| baz`
 foo does not exist => drop `foo? (>= 1.0) |`

   (this is obviously a recursive thing)

Rationales:


1. You can start optionally build-depending on stuff available
   only on some architectures, without having to use arch restriction
   lists.

   Arch restriction lists are tediuous, especially also because in
   the case of libraries, they need to be recursively applied:

 libfoo is only available on bar
 libbaz depends on libfoo

   results in build-depends: libbaz [bar]

   With optional build-depends, you can just write libbaz? and
   not have to update the dep each time libfoo appears on a new
   arch. (apply argument to longer recursive chains)

2. Now I'm not sure if that's a pro or a con, but downstream
   distributions could also add optional dependencies on stuff
   only they ship.

   It's a question of policy whether optional build-dependencies
   on out-of-archive stuff should be allowed or not (e.g. you
   could add an optional b-d while sth is in NEW, and do binNMUs
   after too)

   I only heard this after talking about the proposal though,
   it was not on my mind while creating it ...

Caveats:

1. This will break tools. We want that though, as we don't want
   tools to just ignore optional dependencies.

2. It might cause wrong behavior if you have foo? | bar dependencies,
   and foo? is parsed as a package name, if the tool just installs
   bar. Maybe "foo ?" causes more parser failures, and is preferrable

Not liked proposals:

  Build-Depends-Optional field - it would just be ignored by tools,
  silently, and we'd find about it onyl when it is too late.

  Build-Recommends field - same as previous, but also the semantic
  is different from Recommends (which ignores any unsat dep).

  Build-Depends: foo [any-available] - it's longer, more magical,
  and would also be silently ignored

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Optional Build-Depends

[Julian Andres Klode]

On Thu, Jul 16, 2020 at 07:27:52PM +0200, Julian Andres Klode wrote:
> Not liked proposals:
> 
>   Build-Depends-Optional field - it would just be ignored by tools,
>   silently, and we'd find about it onyl when it is too late.
> 
>   Build-Recommends field - same as previous, but also the semantic
>   is different from Recommends (which ignores any unsat dep).
> 
>   Build-Depends: foo [any-available] - it's longer, more magical,
>   and would also be silently ignored

Build-Depends: foo  was another suggestion, but maybe
too magical.


-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Optional Build-Depends

[Julian Andres Klode]

On Thu, Jul 16, 2020 at 06:56:56PM +0100, Steve McIntyre wrote:
> 
> On Thu, Jul 16, 2020 at 07:27:52PM +0200, Julian Andres Klode wrote:
> 
> ...
> 
> >Rationales:
> >
> >
> >1. You can start optionally build-depending on stuff available
> >   only on some architectures, without having to use arch restriction
> >   lists.
> >
> >   Arch restriction lists are tediuous, especially also because in
> >   the case of libraries, they need to be recursively applied:
> >
> > libfoo is only available on bar
> > libbaz depends on libfoo
> >
> >   results in build-depends: libbaz [bar]
> >
> >   With optional build-depends, you can just write libbaz? and
> >   not have to update the dep each time libfoo appears on a new
> >   arch. (apply argument to longer recursive chains)
> 
> Hmmm. What happens if a build-dep is transiently not available? How
> can you guarantee controllable, predictable behaviour?

It is not transient. If foo exists but depends on bar
which does not exist, then foo still exists. This is
the difference to Recommends, which makes it more
controllable and predictable.



-- 
debian developer - deb.li/jak | jak-linux.org - free software dev 
ubuntu core developer  i speak de, en

Re: The primary interface to dpkg

[Julian Andres Klode]

On Mon, May 17, 2021 at 10:14:24PM +0200, Fabrice Bauzac-Stehly wrote:
> Hello,
> 
> I am puzzled.
> 
> dpkg/man/dpkg.pod:
>  The primary and more user-friendly
> front-end for B is B(8).
> 
> dpkg/README:
> The primary interface for the dpkg suite is the ‘dselect’ program;
> a more low-level and less user-friendly interface is available in
> the form of the ‘dpkg’ command.
> 
> Aren't these statements contradictory?  Which one is correct?

IMO Neither, the primary frontend for dpkg is apt; not sure
if apt-get(8) or apt(8).

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Bug#910377: Inhibit reboot/shutdown if dpkg is running

[Julian Andres Klode]

On Wed, May 19, 2021 at 04:03:16PM +0200, Laurent Bigonville wrote:
> reopen 910377
> reassign dpkg 1.20.9
> thanks
> 
> On Sat, 31 Aug 2019 00:34:32 +0200 Michael Biebl  wrote:
> 
> > On Fri, 5 Oct 2018 21:30:43 +0200 Michael Biebl  wrote:
> > > Am 05.10.18 um 21:28 schrieb Michael Biebl:
> > > > That said, also keep in mind, that the inhibit mechanism does not
> work
> > > > if the reboot request is triggered by privileged users [1], e.g.
> if you
> > > > trigger a reboot as root, an existing inhibitor blocks are ignored.
> > > > [1] https://github.com/systemd/systemd/issues/6644
> > >
> > > This issue describes this even better
> > > https://github.com/systemd/systemd/issues/2680
> > 
> > It seems there is no real interest to change this upstream and even if
> > at some point in the future there was a way to make inhibitors work for
> > the root user, I think such an inhibitor lock shoud be take directly by
> > dpkg. I don't think the hook interface is sufficient for that.
> 
> I'm reopening this issue and reassigning it dpkg package, that would at
> least avoid non privileged users to restart the machine when there is an
> update happening.
> 
> Apparently RPM has this functionality via a plugin. The manpage of the
> plugins available at [0] and says:
> 
>    This plugin for RPM prevents the system to enter shutdown, sleep
>    or idle mode while there is a rpm transaction running to prevent
>    system corruption that can occur if the transaction is
>    interrupted by a reboot.
> 
>    This is achieved by using the inhibit DBUS interface of systemd.
>    The call is roughly equivalent to executing
> 
>    systemd-inhibit --mode=block --what=idle:sleep:shutdown --who=RPM
>    --why="Transaction running"
> 
> The code is available in [1]
> 
> Having something similar in dpkg would be nice, but that would mean that
> dpkg will grow a dependency on libdbus and/or libsystemd, not sure how that
> would work

Or just use apt which already does this instead of manually running
dpkg? I mean, it wouldn't hurt to have dpkg inhibit too, but it's not
really relevant for most users given that apt-pkg library does and hence
apt, aptitude, packagekit.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Bug#910377: Inhibit reboot/shutdown if dpkg is running

[Julian Andres Klode]

On Wed, May 19, 2021 at 04:03:16PM +0200, Laurent Bigonville wrote:
> reopen 910377
> reassign dpkg 1.20.9
> thanks
> 
> On Sat, 31 Aug 2019 00:34:32 +0200 Michael Biebl  wrote:
> 
> > On Fri, 5 Oct 2018 21:30:43 +0200 Michael Biebl  wrote:
> > > Am 05.10.18 um 21:28 schrieb Michael Biebl:
> > > > That said, also keep in mind, that the inhibit mechanism does not
> work
> > > > if the reboot request is triggered by privileged users [1], e.g.
> if you
> > > > trigger a reboot as root, an existing inhibitor blocks are ignored.
> > > > [1] https://github.com/systemd/systemd/issues/6644
> > >
> > > This issue describes this even better
> > > https://github.com/systemd/systemd/issues/2680
> > 
> > It seems there is no real interest to change this upstream and even if
> > at some point in the future there was a way to make inhibitors work for
> > the root user, I think such an inhibitor lock shoud be take directly by
> > dpkg. I don't think the hook interface is sufficient for that.
> 
> I'm reopening this issue and reassigning it dpkg package, that would at
> least avoid non privileged users to restart the machine when there is an
> update happening.
> 
> Apparently RPM has this functionality via a plugin. The manpage of the
> plugins available at [0] and says:
> 
>    This plugin for RPM prevents the system to enter shutdown, sleep
>    or idle mode while there is a rpm transaction running to prevent
>    system corruption that can occur if the transaction is
>    interrupted by a reboot.
> 
>    This is achieved by using the inhibit DBUS interface of systemd.
>    The call is roughly equivalent to executing
> 
>    systemd-inhibit --mode=block --what=idle:sleep:shutdown --who=RPM
>    --why="Transaction running"
> 
> The code is available in [1]
> 
> Having something similar in dpkg would be nice, but that would mean that
> dpkg will grow a dependency on libdbus and/or libsystemd, not sure how that
> would work

Or just use apt which already does this instead of manually running
dpkg? I mean, it wouldn't hurt to have dpkg inhibit too, but it's not
really relevant for most users given that apt-pkg library does and hence
apt, aptitude, packagekit.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

dpkg: normalize description fields (Was: Re: Bug#986840: apt-listchanges fails to parse status files with ^M characters; should use apt_pkg.TagFile, not write its own parser)

[Julian Andres Klode]

Control: clone -1 -2
Control: reassign -2 dpkg
Control: retitle -2 dpkg: normalize description fields

On Mon, Apr 12, 2021 at 08:14:07PM +0200, Julian Andres Klode wrote:
> Package: apt-listchanges
> Version: 3.23
> Severity: normal
> X-Debbugs-Cc: j...@debian.org
> 
> As reported in 
> https://bugs.launchpad.net/ubuntu/+source/apt-listchanges/+bug/1854772, 
> apt-listchanges
> fails to parse status files that contain carriage return characters, as
> Python normalizes the line endings.
> 
> Instead of writing its own ad-hoc parser, apt-listchanges should use
> apt_pkg.TagFile instead.
> 

I think it could be useful if dpkg could normalize description fields
too, to make the database safer to parse. I have not checked if the
output dpkg prints when reporting status is safe or not.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Unclear new string

[Julian Andres Klode]

Guillem Jover  schrieb am Di., 23. Nov. 2021, 22:14:

> Hi!
>
> On Tue, 2021-11-23 at 18:16:30 +0100, Helge Kreutzmann wrote:
> > while updating the translation of the man pages, I stumbled over one
> > sentence I could not make sense of:
> >
> > "The file mode check failed (since dpkg 1.21.0).  This check currently
> > only applies to regular files that have a known digest, and on the
> > filesystem are not regular files."
> >
> > The part after the "and" is unclear. A filesystem without regular
> > files?
>
> Hmm, right when writing that I did pause for a bit thinking whether it
> was clear or not, but I guess my bias tricked me. :) Let me retry:
>
>   The file mode check failed (since dpkg 1.21.0).  This check currently
>   only applies to pathnames that have a known digest and are not regular
>   files on the filesystem.
>


Sorry for the phone answer, but as I had to read that three times, I
figured I'd note that. I think I got it now: There is a conflict between
having a digest and being an irregular file.

First I wondered: Huh is this checking digests of such files, this makes no
sense? I was like, should this be an or?

Second I forgot.

Turn the "and" into a "but" and I think it gets OK. Currently the
contradiction is not easy to  deduct.

Or write explicitly that having a digest conflicts with actually not being
s regular file.

Bug#910377: System-critical package management

[Julian Andres Klode]

On Mon, Sep 18, 2023 at 12:24:20PM +0200, Guillem Jover wrote:
> While dpkg on systems using systemd _could_ by default take an
> system inhibitor lock, and could provide a good enough reason like say
> "Packaging system upgrade" or whatever, my concern has been with the
> added dependency chain, and after reading your mail and thinking about
> this now, I have to agree this seems like a higher level policy.
> (Of course dpkg could also do that and grow a new --no-inhibit,
> or --refuse-inhibit or similar option, but still.)
> 
> But then, I recalled I had a git branch adding a dpkg-db-lock command
> with a --wait-lock option, that I could recover and polish to provide
> an example pre-hook script that would call that via a background
> systemd-inhibit if systemd is running and the command is available,
> where an admin that wanted to do that for their system or fleet of
> systems could hook into the dpkg config. I've done that locally, and
> will check whether that's viable and probably merge it for 1.22.1
> or 1.22.2, so that people that want to do it can easily do so.

I'm not sure how that works because you'd need to respawn yourself
with systemd-inhibit, whereas the API essentially gives you a file
descriptor over dbus that you keep open until it is safe to reboot.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: System-critical package management

[Julian Andres Klode]

On Mon, Sep 18, 2023 at 12:24:20PM +0200, Guillem Jover wrote:
> While dpkg on systems using systemd _could_ by default take an
> system inhibitor lock, and could provide a good enough reason like say
> "Packaging system upgrade" or whatever, my concern has been with the
> added dependency chain, and after reading your mail and thinking about
> this now, I have to agree this seems like a higher level policy.
> (Of course dpkg could also do that and grow a new --no-inhibit,
> or --refuse-inhibit or similar option, but still.)
> 
> But then, I recalled I had a git branch adding a dpkg-db-lock command
> with a --wait-lock option, that I could recover and polish to provide
> an example pre-hook script that would call that via a background
> systemd-inhibit if systemd is running and the command is available,
> where an admin that wanted to do that for their system or fleet of
> systems could hook into the dpkg config. I've done that locally, and
> will check whether that's viable and probably merge it for 1.22.1
> or 1.22.2, so that people that want to do it can easily do so.

I'm not sure how that works because you'd need to respawn yourself
with systemd-inhibit, whereas the API essentially gives you a file
descriptor over dbus that you keep open until it is safe to reboot.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Bug#1007717: Draft resolution for "Native source package format with non-native version"

[Julian Andres Klode]

On Tue, Jun 07, 2022 at 08:19:29PM +0200, Helmut Grohne wrote:
> Please keep in mind that this is about trade-offs. It is a question of
> how we value "package ownership". If we favour the strong ownership
> approach that Debian used for a long time, then yes accommodating the
> needs of maintainers is paramount. If we want to lessen the concept of
> ownership, embrace drive-by contributions and decentralize maintenance,
> then we need a stronger focus on uniformity. I suppose that my own
> opinion on this matter is fairly obvious at this point.

This is also a significant issue for downstreams. With my Ubuntu hat
on, if I have to touch packages downstream, I loathe it everytime I
get a non-descript blob of all the changes.

I suspect this is the same for other downstream distributions that
want to modify packages. We cannot cater to everyone's individual
packaging repository approach, and downstream repositories if they
exist are separate anyhow.

Using 1.0 instead of 3.0 (quilt) is just being a hostile upstream,
like Red Hat is with dumping all their kernel patches together.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Terminology changes for update-alternatives

[Julian Andres Klode]

On Sun, Jan 29, 2023 at 06:00:51AM +0100, Guillem Jover wrote:
> Hi!
> 
> I'd like to move away from the master/slave terminology used in
> update-alternatives for both the external interfaces (CLI options,
> output fields) obviously preserving backwards compatibility, docs
> and for all the internal code symbols. For the same reasons as mentioned
> in . (This
> is bug #884368.)
> 
> This has kind of blocked improvements as there was no desire to expand
> their usage in other places, where depending on the context it would
> imply more painful transitions being involved.
> 
> For debhelper, Niels decided to use for its declarative support the term
> "Dependents", which while not a wrong term, I've always found it too
> close to dependencies which we use in the packaging relationships context,
> potentially adding to the concept overload/confusion, in addition of
> being long, so I've had my reservations about switching to it.
> 
> I've been pondering about other options, and I think the concept that
> seems to describe best the relationship is akin a planet and its moon
> or satellite orbiting around it and being pulled along. But satellite
> seems too long and unrelated as a direct term.
> 
> (Primary/secondary do not seem to represent this well, and I have an
> aversion to the leader/follower pair and they don't seem to fit well
> here anyway.)
> 
> This is the list of words that I've been playing with:
> 
>   * dependent links  --depend  (covered above)
>   * affixed links--affix   (might not be ideal for translation?)
>   * coupled links--couple  (seems redundant given the dictionary
> description)
>   * attached links   --attach  (seems confusing? and long)
>   * ancillary links(too long)
>   * accessory links(idem)
>   * subsidiary links   (idem)
> 
>   * annex links  --annex   (discarded due to git-annex confusion)
>   * bound links  --bound/--bind
>   * laced links  --lace(cryptic?)
>   * tied links   --tie


I want to suggest root and leaf.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Install profiles

[Julian Andres Klode]

Hi,

based on some discussion on IRC I want to propose install
profiles.

Recommends: foo 
Suggests: foo 

The syntax is the same as for build profiles, and it is
allowed in Recommends and Suggests fields only (maybe
Enhances?).

dpkg changes needed:

- Introduce /var/lib/dpkg/profiles with one profile per
  line and commands to manage the list

dpkg --add-profile 
dpkg --remove-profile 
dpkg --print-profiles

- Introduce apt commands:

apt add-profile
apt remove-profile

  These call the dpkg command and then install newly
  activated Recommends or remove Recommends no longer
  active.

Alternatively, dpkg doesn't need to care about which
profiles are active and apt manages the list.

Or one could invert the list and say we add profiles
we do not want. This might come in handy with profile
inheritance, where e.g. foo profile might imply bar
by default, but then you can request "foo !bar" for
your system.

Gentoo use flags have a default state, it seems
reasonable to follow the same approach, so you would
have

enable-profile
disable-profile
reset-profile

or something and the repository could declare default
profiles.

This allows users to customize their systems and avoid
installing recommends they don't want.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en


signature.asc
Description: PGP signature

Re: Install profiles

[Julian Andres Klode]

On Wed, Jun 14, 2023 at 06:00:42PM +0200, Julian Andres Klode wrote:
> Hi,
> 
> based on some discussion on IRC I want to propose install
> profiles.
> 
> Recommends: foo 
> Suggests: foo 
> 
> The syntax is the same as for build profiles, and it is
> allowed in Recommends and Suggests fields only (maybe
> Enhances?).

Alternative backwards compatible approach:

Packages declare Profiles: they are part of

and then either

* apt ignores Recommends for disabled profiles
* apt installs Suggests for enabled profiles

It's not clear to me basically what's more powerful,
declaring profiles on relationships, or the packages.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en


signature.asc
Description: PGP signature

Bug#1040968: dpkg: Less noisy output

[Julian Andres Klode]

Package: dpkg
Severity: wishlist
X-Debbugs-Cc: j...@debian.org

Users are complaining loudly that dpkg's output is too noisy,
and we should be looking at how we can improve that, currently
for each package we print at least 3 lines, and it's very
distracting:

Preparing to unpack
Unpacking
Setting up

I think these should be a rolling status line with a carriage
return at the end, such that it gets overriden by output,
but I'm not quite sure.

Optimally we'd be able to turn the output off, and control
it from apt based on the dpkg status log. And then also capture
output from maintainer script and essentially prepend the line
to the backlog when the maintainer script has output.

But not sure if that's feasible, probably needs some very
annoying manipulation of pty stuff.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Really enable -fstack-clash-protection on armhf/armel?

[Julian Andres Klode]

On Thu, Nov 23, 2023 at 10:45:33AM +0100, Matthias Klose wrote:
> Hi,
> 
> it looks like enabling this flag on armel/armhf is a little bit premature.
> 
> Apparently it's not completely supported upstream, and might cause
> regressions, according to
> https://bugzilla.redhat.com/show_bug.cgi?id=1522678
> 
> Is that a feature that the Debian ARM32 porters and the security team really
> want to support actively, despite the missing upstream support?
> 
> In Ubuntu, people tracked down segfaults due to this change in at least
> valgrind and gnutls, maybe more.

It's 1.5 months later, valgrind is still failing and apt in valgrind
hence segfaults. I am disabling the apt valgrind test on armhf in 2.7.8,
but this situation is somewhat untenable.

I did clone the bug to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1060251
now.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Architecture variants for Debian / Ubuntu

[Julian Andres Klode]

On Fri, Nov 03, 2023 at 02:02:37PM +0100, Guillem Jover wrote:
> Hi!
> 
> On Thu, 2023-11-02 at 15:27:54 +, Michael Hudson-Doyle wrote:
> > On Tue, 31 Oct 2023 at 09:21, Guillem Jover wrote:
> > > This can be used among other things to set up foreign chroots, by
> > > running the host tools, so disallowing installation could be
> > > problematic. Even though I guess there could be a warning about this,
> > > or maybe it could be controlled through a force option, although both
> > > seems like they could be disruptive.
> 
> > Of course in such cases dpkg knows that something funny is going on and
> > could suppress the warning itself.
> 
> Right, also true.
> 
> > I spent a few minutes trying to think hard about this and I honestly don't
> > think I can predict whether trying to prevent installation of incompatible
> > packages is worth it (after all one of the ways users could get into
> > trouble would be moving an installed system to a different CPU and having
> > binaries start to fail and obviously dpkg can't help there).
> > 
> > One result of this thinking was: I had been thinking/assuming the issue of
> > which variants to consider would be apt configuration, but maybe dpkg
> > configuration would make more sense (after all, --add-architecture is a
> > parameter to dpkg). And in this case, dpkg could object when installing a
> > variant that has not been configured.
> 
> Yes, the "plan" has been to add support for run-time CPU attributes,
> so that when adding a new arch, for example you can specify whether
> that arch is runnable, which could help dpkg decide whether to allow
> by default to install M-A:foreign packages.
> 
> I guess this is similar, so such future interface should probably take
> this into account as something to support too. Will check where this
> is tracked and add a note to it.
> 
> And of course that is fine as a guardrail, but if a user hit that out
> of running a frontend, then that would already be too late, which to
> me means that frontends need to be aware of this too (and not pass
> packages that dpkg would/could/might refuse to install), when deciding
> what to pass to dpkg.
> 
> But in any case, as you say, this currently would not be worse than
> configuring a foreign arch, installing some foreign package and trying
> to run it, but it might make it potentially more common. And as
> mentioned above the effecting layer this needs to be decided up seems
> higher anyway (even if dpkg could provide the infra for it).

So I'd like to interject here for a moment with my APT hat, because I
want to have ISA autodiscovery. I want to be able to configure the
system such that APT automatically picks the best ISA to download,
and not have to configure a specific one.

Similar to when you pass -cpu host to qemu, I want a magic alias
for dpkg ISA support to say "host" or whatever and APT can then go
and pick the best one (possibly multiple ones).

It's fine for dpkg to provide a list of all architectures and the
preference ordering in that case I think.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: Architecture variants for Debian / Ubuntu

[Julian Andres Klode]

On Wed, Sep 06, 2023 at 11:27:02AM +0200, Guillem Jover wrote:
> Hi!
> 
> On Fri, 2023-09-01 at 08:43:55 +1200, Michael Hudson-Doyle wrote:
> > Recently the topic of exploiting newer instructions without dropping
> > support for older machines has come up several times inside Ubuntu
> > engineering. I understand this topic has come up several times in the past
> > for Debian as well, but nothing has really come of it to date.
> 
> I also had a chat about this with Matthias Klose (CCed) around 2022-05.
> 
> > I've spent a while thinking through the options and coming up with a design
> > and wrote some notes into a wiki page:
> > https://wiki.debian.org/ArchitectureVariants
> 
> I think we are already doing 1, 2 and 3. I agree 4 is just wrong. And
> something like 5 is what I suggested to Matthias for Ubuntu when we
> last discussed it as the best way to go about this.
> 
> I'm not sure I entirely agree with the requirements you set forth
> though:
> 
>  - I think such optimized builds might need to be done with "special
>toolchains" (these could simply be wrappers over the host compiler
>passing the appropriate flags via command-line or via specs or
>similar, not necessarily full blown toolchains), passing these via
>something like dpkg-buildflags seems currently unreliable, as I don't
>think we have full coverage in packages (neither for all compilers
>available)? Although it would be better as it would centralize the
>management. (For reference this is in part how rpm handles this:
> https://github.com/rpm-software-management/rpm/blob/master/rpmrc.in)
>  - Perhaps that's a limitation from the archive software side, but
>requiring to place the binary packages in the same pool seems
>rather restrictive (it forces different filenames for example).
>  - I guess it might be nice for the ISA to be passed down to the
>dpkg tools, but I don't think this is strictly necessary? A
>frontend like apt could also decide based on metadata in say the
>Release file, although not having the actual installed package
>metadata on whether it was a different ISA build or not would make
>its job more inconvenient. In any case I don't have a big issue
>with recording this via dpkg-gencontrol or similar if necessary.
> 
> On the specific implementation details:
> 
>  - Changing the Architecture format (as in adding colons there) seems
>like a non-starter, and I expect that would break lots of things
>(I mean it could be done but I'm not sure it's worth it for this).
>Recording this mostly as a hint than anything else, via another
>field (if necessary at all) I think would be best.
>  - As covered in previous discussions, dpkg could (but I don't think
>it's necessary) check whether the .deb is runnable on the current
>hw, but that's tricky as chrootless installs need to be taken
>into account, etc. It should certainly not be part of dependency
>resolution.
>  - I'm not fond of having to change the binary package name format
>either for this (name_version_arch.deb) even if at least dpkg
>itself does not care (but I know other tools do care), and
>depending on the format I'd expect things to break (this goes
>back to the shared pool concern).
>  - If dpkg-architecture needs to be aware of this, then this might need
>to be auto-detectable from just the current toolchain being used.
> 
> Some of the above problems could perhaps be avoided if we introduced
> a concept of architecture aliases/ISAs (similar to what rpm has), which
> would side-step the pool sharing issue, the binary package renaming,
> etc. One big issue with this is that it requires for dpkg to have an
> exhaustive table of all such aliases, and if there's ever a new alias
> added, old dpkg versions need to be updated or they will not understand
> what they match with. So this does not seem ideal either. So I guess this
> is a variation over your proposal, but perhaps this could still be used
> in specific contexts, say only at build-time (but not for dependency
> relationships), for repo management (say binary-arm64v9/Packages.xz),
> or binary package names where the field would specify the actual name
> for the filename, say:
> 
>   Architecture: arm64
>   ArchitectureIsa: arm64v9
> 
> or maybe better:
> 
>   Architecture: arm64
>   ArchitectureIsa: v9
> 
> resulting in dpkg-deb generating:
> 
>   binpkg_1.0-1_arm64v9.deb
> 
> but targeting arm64. I also think I prefer naming this explicitly as ISA
> variants, if you will, than just architecture variants as that gives
> way too much room (which perhaps we want, but then that has other
> implications over compatibility), and for the field perhaps just Isa is
> better, to avoid the implicit repetition of
> ArchitectureInstructionSetArchitecture :), but that makes it less easy
> to associate both as related.

I have thought more about this and I'm not particularly fond of the
ArchitectureIsa name. While 

Bug#1067155: debian-policy: prerm scripts cannot actually rely on dependencies

[Julian Andres Klode]

Package: debian-policy
Severity: wishlist
X-Debbugs-Cc: de...@lists.debian.org, debian-d...@lists.debian.org

APT's installation planner does not consider dependencies of packages
being scheduled for removal, so a prerm must fail equally gracefully
as a postrm does in absence of its dependencies.

This does break dpkg's assumptions which it happily tells you about,
but this is the reality we live in.

So e.g. one thing you see is that apt removes libapt-pkg6.0, then
unpacks libapt-pkg6.0t64, then removes libapt-pkg6.0 reverse
dependencies.

Clearly APT should be considering dependencies when removing packages
but even in that case, removals may sometimes need to be forced in the
wrong order because any order leads to broken dependencies, so still,
prerms should not rely on dependencies, but only on essential packages.
-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en