Re: dpkg database during postinst processing
Hi, On 7/29/24 17:27, Simon Richter wrote: Do I need to do anything special to get the current view, or should libdpkg provide a current view in any case? Nevermind, the modstatdb_open() call got lost in a rebase. Oof. Simon
dpkg database during postinst processing
Hi, I'm trying to look at the list of installed files from a program that uses libdpkg and runs as part of postinst processing. Simple test case is a package containing a single file /testfile, and a postinst "exec /bin/bash -i <>/dev/tty", installed into empty installation and database directories. Here, info/status is empty, and the new package can only be seen in updates/000*. When I now call ensure_allinstfiles_available(), the files installed with my package are not visible as namenodes. I'd assume it's the same for all files installed within the same transaction -- so I am seeing a slightly outdated view. For dpkg-divert, that is probably not a big problem (but might still be wrong), but if I want to check for file conflicts caused by aliasing, I absolutely need the view to be up-to-date. Do I need to do anything special to get the current view, or should libdpkg provide a current view in any case? Simon
Re: [PATCH] Symlink src/sh/dpkg-error.sh into build directory
Hi, On 6/12/24 11:54, Guillem Jover wrote: This is needed to run the tests during an out-of-tree build I guess this is for the functional test suite? What runes are you using to run it out-of-tree? $ make installtest DPKG_BUILDTREE=/tmp/dpkg I also have an evil bash script that sets up a namespace where dpkginst/usr is a bind mount of itself, which is great for testing scenarios where /usr is a separate partition (rename gives EXDEV, stat shows different device). I'm not sure yet how to integrate that, given that it's orthogonal to the other test options, my feeling is that I'm building a matrix here. Simon
[PATCH] Symlink src/sh/dpkg-error.sh into build directory
This is needed to run the tests during an out-of-tree build
---
configure.ac | 1 +
1 file changed, 1 insertion(+)
diff --git a/configure.ac b/configure.ac
index 36db52e71..cc2027534 100644
--- a/configure.ac
+++ b/configure.ac
@@ -246,6 +246,7 @@ AC_DEFINE([PACKAGE_RELEASE], [PACKAGE_VERSION " ("
ARCHITECTURE ")"],
AC_CONFIG_TESTDIR([src/at])
AM_MISSING_PROG([AUTOM4TE], [autom4te])
+AC_CONFIG_LINKS([src/sh/dpkg-error.sh:src/sh/dpkg-error.sh])
AC_CONFIG_FILES([
Makefile
dselect/Makefile
--
2.39.2
Salsa CI broken
Hi, Builds on Salsa terminate with PO4A man.stamp Undefined subroutine ::Po4a::Pod::dgettext called at /usr/share/perl5/Locale/Po4a/Pod.pm line 94, <$fh> line 21. I might investigate later, but if someone has seen that problem before and knows what it is, that would possibly save some time. Simon
[PATCH] Add missing
--- lib/dpkg/varbuf.h | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/dpkg/varbuf.h b/lib/dpkg/varbuf.h index 99d9f0691..5cc9718c7 100644 --- a/lib/dpkg/varbuf.h +++ b/lib/dpkg/varbuf.h @@ -23,6 +23,7 @@ #define LIBDPKG_VARBUF_H #include +#include #include #include -- 2.39.2
Re: [PATCH] cleanup: Pull in for memchr
Hi Guillem, On 3/20/24 08:59, Guillem Jover wrote: Thanks! I ended up including this unconditionally (also because we are not explicitly checking for this header which should always be there, and if something else is checking for it, that would be an internal implementation detail that might change in the future), and before "compat.h". Now in git main. strnlen() should also always be there. :> Simon
[PATCH] cleanup: Pull in for memchr
This needs further inspection on more systems --- lib/compat/strnlen.c | 4 1 file changed, 4 insertions(+) diff --git a/lib/compat/strnlen.c b/lib/compat/strnlen.c index d02bb4bbd..6a7eb0454 100644 --- a/lib/compat/strnlen.c +++ b/lib/compat/strnlen.c @@ -22,6 +22,10 @@ #include "compat.h" +#ifdef HAVE_STRING_H +# include +#endif + /* Find the length of STRING, but scan at most MAXLEN characters. If no '\0' terminator is found in that many characters, return MAXLEN. */ -- 2.39.2
[PATCH] cleanup: make memcpy check -Werror safe
--- configure.ac | 16 +++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index b084ade12..1daf4b1de 100644 --- a/configure.ac +++ b/configure.ac @@ -207,7 +207,21 @@ DPKG_CHECK_COMPAT_FUNCS([\ alphasort \ unsetenv \ ]) -AC_CHECK_FUNCS([memcpy lchown], + +AC_MSG_CHECKING([for memcpy]) +AC_LINK_IFELSE( + AC_LANG_PROGRAM( +[[ +#ifdef HAVE_STRING_H +#include +#endif +]], +[[memcpy(NULL, NULL, 0);]]), + [AC_MSG_RESULT([yes])], + [AC_MSG_RESULT([no]) + AC_MSG_ERROR([missing required function])]) + +AC_CHECK_FUNCS([lchown], [], [AC_MSG_ERROR([missing required function])]) AC_CHECK_FUNCS([\ strtoimax \ -- 2.39.2
[PATCH] Add db-fsys-load.c to POTFILES.in
--- po/POTFILES.in | 1 + 1 file changed, 1 insertion(+) diff --git a/po/POTFILES.in b/po/POTFILES.in index b38797ad7..a2386011a 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -17,6 +17,7 @@ lib/dpkg/db-ctrl-upgrade.c lib/dpkg/db-fsys-digest.c lib/dpkg/db-fsys-divert.c lib/dpkg/db-fsys-files.c +lib/dpkg/db-fsys-load.c lib/dpkg/db-fsys-override.c lib/dpkg/deb-version.c lib/dpkg/debug.c -- 2.39.2
Re: [PATCH] Factor out common code for reloading database files
Hi Guillem, On 3/18/24 01:28, Guillem Jover wrote: Attached is what I had, which I've polished a bit more now. Will include in my next push. Nice, will rebase my branch on top of that. The new file probably needs to be added to po/POTFILES.in as well, since it contains translatable strings. Simon
Re: [PATCH] Factor out common code for reloading database files
On 17.03.24 08:27, Simon Richter wrote: --- lib/dpkg/Makefile.am| 1 + lib/dpkg/db-fsys-common.c | 87 + lib/dpkg/db-fsys-divert.c | 43 -- lib/dpkg/db-fsys-override.c | 50 + lib/dpkg/dpkg-db.h | 13 ++ 5 files changed, 120 insertions(+), 74 deletions(-) create mode 100644 lib/dpkg/db-fsys-common.c FWIW, this also needs translation changes, but for that the general direction needs to be useful. I think it is because it merges duplicated code, and it will be even more useful when I add a third file there. Simon
Another usrmerge complication
Hi, because life isn't hard enough as it is: When /bin is a symlink to usr/bin, and I install two packages, where one installs /bin/foo and the other installs /usr/bin/foo, then, if both are installed in the same dpkg invocation, the contents of the first package end up being installed, while in two separate invocations, the second package's contents win. So for any undetected file conflict, which file now exists now depends on whether apt decided to install both packages during the same installation round or not. Writing more test cases, Simon
[PATCH] Valgrind support for test suite
---
tests/Makefile | 4
tests/Test.mk | 17 +++--
2 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/tests/Makefile b/tests/Makefile
index 5d8309608..5d3f30ee2 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -141,6 +141,10 @@ test:: $(test_targets)
$(test_targets)::
$(MAKE) -C $(subst -test,,$@) test
+memtest:: export DPKG_WRAPPER=valgrind --leak-check=full --show-leak-kinds=all
--track-origins=yes
+memtest:: export DPKG_BUILDTREE=$(realpath ..)
+memtest:: $(test_targets)
+
installtest: $(test_targets)
.PHONY: test $(test_targets)
diff --git a/tests/Test.mk b/tests/Test.mk
index 3c020af2c..1f46a59af 100644
--- a/tests/Test.mk
+++ b/tests/Test.mk
@@ -21,6 +21,11 @@ ifdef DPKG_BUILDTREE
DPKG_PATH :=
$(DPKG_BUILDTREE)/src:$(DPKG_BUILDTREE)/utils:$(DPKG_BUILDTREE)/scripts:$(DPKG_PATH)
endif
+## Wrapper support (default none) ##
+
+DPKG_WRAPPER ?=
+export DPKG_WRAPPER
+
DPKG_ENV = \
PATH=$(DPKG_PATH) \
$(DPKG_MAINTSCRIPT_DEBUG)
@@ -77,21 +82,21 @@ endif
DPKG_ADMINDIR = $(CURDIR)/../dpkgdb
DPKG_COMMON_OPTIONS = --admindir="$(DPKG_ADMINDIR)"
-DPKG = dpkg $(DPKG_COMMON_OPTIONS) $(DPKG_OPTIONS)
+DPKG = $(DPKG_WRAPPER) dpkg $(DPKG_COMMON_OPTIONS) $(DPKG_OPTIONS)
DPKG_INSTALL = $(BEROOT) $(DPKG) -i
DPKG_UNPACK = $(BEROOT) $(DPKG) --unpack
DPKG_CONFIGURE = $(BEROOT) $(DPKG) --configure
DPKG_REMOVE = $(BEROOT) $(DPKG) -r
DPKG_PURGE = $(BEROOT) $(DPKG) -P
DPKG_VERIFY = $(DPKG) -V
-DPKG_DEB = dpkg-deb $(DPKG_DEB_OPTIONS)
-DPKG_DIVERT = dpkg-divert $(DPKG_COMMON_OPTIONS) $(DPKG_DIVERT_OPTIONS)
+DPKG_DEB = $(DPKG_WRAPPER) dpkg-deb $(DPKG_DEB_OPTIONS)
+DPKG_DIVERT = $(DPKG_WRAPPER) dpkg-divert $(DPKG_COMMON_OPTIONS)
$(DPKG_DIVERT_OPTIONS)
DPKG_DIVERT_ADD = $(BEROOT) $(DPKG_DIVERT) --add
DPKG_DIVERT_DEL = $(BEROOT) $(DPKG_DIVERT) --remove
-DPKG_SPLIT = dpkg-split $(DPKG_SPLIT_OPTIONS)
+DPKG_SPLIT = $(DPKG_WRAPPER) dpkg-split $(DPKG_SPLIT_OPTIONS)
DPKG_BUILD_DEB = $(DPKG_DEB) -b
-DPKG_QUERY = dpkg-query $(DPKG_COMMON_OPTIONS) $(DPKG_QUERY_OPTIONS)
-DPKG_TRIGGER = dpkg-trigger $(DPKG_COMMON_OPTIONS) $(DPKG_TRIGGER_OPTIONS)
+DPKG_QUERY = $(DPKG_WRAPPER) dpkg-query $(DPKG_COMMON_OPTIONS)
$(DPKG_QUERY_OPTIONS)
+DPKG_TRIGGER = $(DPKG_WRAPPER) dpkg-trigger $(DPKG_COMMON_OPTIONS)
$(DPKG_TRIGGER_OPTIONS)
PKG_STATUS = $(DPKG_QUERY) -f '$${Status}' -W
--
2.39.2
[PATCH] Factor out common code for reloading database files
---
lib/dpkg/Makefile.am| 1 +
lib/dpkg/db-fsys-common.c | 87 +
lib/dpkg/db-fsys-divert.c | 43 --
lib/dpkg/db-fsys-override.c | 50 +
lib/dpkg/dpkg-db.h | 13 ++
5 files changed, 120 insertions(+), 74 deletions(-)
create mode 100644 lib/dpkg/db-fsys-common.c
diff --git a/lib/dpkg/Makefile.am b/lib/dpkg/Makefile.am
index 9ef3a37f7..9ab402cf5 100644
--- a/lib/dpkg/Makefile.am
+++ b/lib/dpkg/Makefile.am
@@ -73,6 +73,7 @@ libdpkg_la_SOURCES = \
db-ctrl-access.c \
db-ctrl-format.c \
db-ctrl-upgrade.c \
+ db-fsys-common.c \
db-fsys-digest.c \
db-fsys-divert.c \
db-fsys-files.c \
diff --git a/lib/dpkg/db-fsys-common.c b/lib/dpkg/db-fsys-common.c
new file mode 100644
index 0..be106aff6
--- /dev/null
+++ b/lib/dpkg/db-fsys-common.c
@@ -0,0 +1,87 @@
+/*
+ * libdpkg - Debian packaging suite library routines
+ * db-fsys-common.c - Common handling for admindb files
+ *
+ * Copyright © 2008-2012 Guillem Jover
+ * Copyright © 2022 Simon Richter
+ *
+ * This is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ */
+
+#include
+#include
+
+#include "dpkg-db.h"
+#include "dpkg.h"
+#include "i18n.h"
+#include "debug.h"
+
+#include
+#include
+
+#include
+#include
+
+FILE *dpkg_db_reopen(
+ struct dbinfo *info,
+ char const *filename,
+ bool *out_unchanged)
+{
+ struct stat sb_next;
+ FILE *file_next;
+
+ assert(info);
+ assert(filename);
+
+ onerr_abort++;
+
+ file_next = fopen(filename, "r");
+ if (!file_next) {
+ if (errno != ENOENT)
+ ohshite(_("failed to open %s"), filename);
+ } else {
+ setcloexec(fileno(file_next), filename);
+
+ if (fstat(fileno(file_next), _next))
+ ohshite(_("failed to fstat %s"), filename);
+
+ /*
+* We need to keep the database file open so that the
+* filesystem cannot reuse the inode number (f.ex. during
+* multiple dpkg-divert invocations in a maintainer script),
+* otherwise the following check might turn true, and we
+* would skip reloading a modified database.
+*/
+ if (info->file &&
+ info->sb.st_dev == sb_next.st_dev &&
+ info->sb.st_ino == sb_next.st_ino) {
+ fclose(file_next);
+ onerr_abort--;
+ debug(dbg_general, "%s: %s is unchanged, skipping",
+ __func__, filename);
+ if(out_unchanged)
+ *out_unchanged = true;
+ return NULL;
+ }
+ info->sb = sb_next;
+ }
+ if (info->file)
+ fclose(info->file);
+ info->file = file_next;
+
+ onerr_abort--;
+ if(out_unchanged)
+ *out_unchanged = false;
+ return file_next;
+}
diff --git a/lib/dpkg/db-fsys-divert.c b/lib/dpkg/db-fsys-divert.c
index e0054bb35..2198685ad 100644
--- a/lib/dpkg/db-fsys-divert.c
+++ b/lib/dpkg/db-fsys-divert.c
@@ -23,7 +23,6 @@
#include
#include
-#include
#include
#include
@@ -45,54 +44,28 @@ static char *diversionsname;
void
ensure_diversions(void)
{
- static struct stat sb_prev;
- struct stat sb_next;
+ static struct dbinfo dbi;
+ bool unchanged;
char linebuf[MAXDIVERTFILENAME];
- static FILE *file_prev;
FILE *file;
struct fsys_diversion *ov, *oicontest, *oialtname;
if (diversionsname == NULL)
diversionsname = dpkg_db_get_path(DIVERSIONSFILE);
- onerr_abort++;
+ file = dpkg_db_reopen(, diversionsname, );
- file = fopen(diversionsname, "r");
- if (!file) {
- if (errno != ENOENT)
- ohshite(_("failed to open diversions file"));
- } else {
- setcloexec(fileno(file), diversionsname);
-
- if (fstat(fileno(file), _next))
-
Scriptable interface for "set package selection to this set"
Hi, I'm trying to build a small script to synchronize the package installation sets of a bunch of machines. My current approach is to simply run dpkg --clear-selections dpkg --set-selections I believe that this is the only interface that allows me to set the package selection without having to compare current and desired list. To get started, I first tried what would happen if I cleared the entire list: root@57ac7d42f84f:/# dpkg --clear-selections root@57ac7d42f84f:/# apt-get dselect-upgrade Reading package lists... Done Building dependency tree... Done Reading state information... Done The following packages will be REMOVED: adduser apt debian-archive-keyring e2fsprogs gpgv libapt-pkg6.0 libcom-err2 libext2fs2 libffi8 libgnutls30 libhogweed6 libidn2-0 libnettle8 libp11-kit0 libseccomp2 libsemanage-common libsemanage2 libsepol2 libss2 libstdc++6 libtasn1-6 libunistring2 libxxhash0 logsave mount passwd tzdata WARNING: The following essential packages will be removed. This should NOT be done unless you know exactly what you are doing! apt adduser (due to apt) gpgv (due to apt) libapt-pkg6.0 (due to apt) debian-archive-keyring (due to apt) libgnutls30 (due to apt) libseccomp2 (due to apt) libstdc++6 (due to apt) e2fsprogs libcom-err2 (due to e2fsprogs) libext2fs2 (due to e2fsprogs) libss2 (due to e2fsprogs) logsave (due to e2fsprogs) 0 upgraded, 0 newly installed, 27 to remove and 0 not upgraded. After this operation, 30.4 MB disk space will be freed. E: Removing essential system-critical packages is not permitted. This might break the system. As far as I can see, there are two things apt and dpkg disagree on: - whether apt itself is Essential This is understandable, apt special-cases itself, and for a good reason. I have no problem hardcoding an "apt install" into my package selection, that solves this nicely. - whether e2fsprogs is Essential This is only Important, not Essential, so dpkg marks it "deinstall", and apt doesn't want to remove it. Outside of a container, obviously there will be more "Important" packages. I can hardcode a sensible package set to have something that works for me for now, but medium term I will probably have to explain this script to someone else. The obvious problem with this interface is that half of it is provided by one tool, and half of it by another, and these implementations have incompatible worldviews that the user of the interface needs to be aware of, because it was never designed to be cohesive. Also, the name "dselect-upgrade" suggests that this is really an atavism. A useful one, but nonetheless probably not meant to be officially supported. Since this functionality is useful, it would be nice to have a proper supported way to do this. Simon
Re: Removing dpkg arch definitions for uclinux-any?
Hi, On 13.11.23 19:08, Guillem Jover wrote: I was checking for dpkg arch definitions to cleanup and stumbled over the uclinux-any ones (added as part of #455501), where I noticed the µCLinux fork got merged into mainline Linux in 2.5.46, and then several of the no-MMU ports got removed from there more recently. I must admit that I haven't really used uClinux, or rather any MMUless architecture in ages, and the company that I originally wrote this for (and which did indeed run an internal Debian mirror with packages for a big-endian MMUless ARM box running uClinux) has since shut down, so I believe there are no users left. I'm not sure if it's entirely dead or just about to get a resurgence because of FPGAs and FPGA SoCs with integrated hard CPUs that are gaining in popularity. On the pure FPGA side, my expectation is that if it runs Linux, it will have an MMU. The major commercial soft CPUs, Microblaze and NiosII, have MMU designs with very little overhead (basically, a TLB and a processor mode for the TLB miss handler to run in), so there is little incentive to run MMUless. RISC-V, ARM and SPARC implementations for FPGAs might be interesting to run MMUless (because their MMU interface standards take a lot of resources to implement), but I'm not aware of anyone actually doing that. On the SoC side, I think most available ICs contain an MMU, and those that don't aren't really used with Linux (there are some MMUless Cortex-R cores that can address enough memory, but I doubt that's a practical concern). So I think these can be removed. Simon OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#910377: System-critical package management
Hi,
On 18.09.23 19:38, Julian Andres Klode wrote:
I'm not sure how that works because you'd need to respawn yourself
with systemd-inhibit, whereas the API essentially gives you a file
descriptor over dbus that you keep open until it is safe to reboot.
popen("systemd-inhibit ... cat", "w");
should work.
Simon
OpenPGP_signature
Description: OpenPGP digital signature
Re: System-critical package management
Hi,
On 18.09.23 19:38, Julian Andres Klode wrote:
I'm not sure how that works because you'd need to respawn yourself
with systemd-inhibit, whereas the API essentially gives you a file
descriptor over dbus that you keep open until it is safe to reboot.
popen("systemd-inhibit ... cat", "w");
should work.
Simon
OpenPGP_signature
Description: OpenPGP digital signature
Re: System-critical package management
Hello, The lack of any system of recognition for packages that are critical to system operation impedes the reliability of Debian-based systems. For example, a reboot during a background package upgrade process on critical system packages unbeknownst to the user may result in the system unable to boot as expected, with little readily-available feedback to the user as to the cause. Locking out reboots while the package manager is active is a policy that needs to be provided by the policy layer that allows ordinary users to reboot -- so this is the responsibility of the desktop environment. The base system and package manager require superuser privileges for both reboot and invoking the package manager. For single-user systems, it is the responsibility of the administrator to not issue a reboot command while a package upgrade is in progress, which is not an onerous requirement because the package upgrade must be manually commanded as well. Packages are often installed in environments where no control over reboots is possible and where system services usually found on desktops are unavailable, such as inside containers during preparation of container images. There is no appropriate place to implement such a lockout at a low level. The kernel is informed of the intention to reboot only after system shutdown is complete, so this is the wrong place, and above that, users have a choice of different policy layers that fit their use case best, including "none". But: because background updates on desktop systems are implemented as a system service that is run through a policy layer, it is possible to implement such a lockout on this layer. Other operating systems like Windows and MacOS manage this by updating system-critical components separately from user-land during shutdown, while clearly giving user-feedback that critical updates are taking place, and that for example the system should not be turned off. No, these systems make no distinction between system and user components. The reason upgrades are performed through a reboot is a historical shortcoming in the file system implementation: Unix separates the contents of a file from its file name, so if a file is open, its name can still be changed or removed, while the file contents are kept until no more names point to it *and* no more open file handles exist. On Windows, open files cannot be renamed or deleted unless the program has specifically allowed this, which (for historical reasons) few programs do, so the upgrade process works by unpacking the new files to a temporary name, making a note to rename the files, then rebooting and performing the rename while no users are logged on and no services are running, and then subsequently starting the system. This process is the same even for user programs, so if you update WinRAR while it is open (so the file cannot be updated), the installation process will ask for a reboot to complete the upgrade. A potential middle-ground solution to this is to allow packages to be marked as "system-critical" to DPKG by external system components - for example a standard desktop Ubuntu system might mark the Gnome Display Manager, Networking drivers, and others in this way during installation. These system-critical packages could then be protected by DPKG in the following ways: - They are automatically reverted to a known good state on upgrade failure (e.g. previous version) Generally, packages are expected to go from one functional state to another in a very quick operation after verifying that the operation can be performed. For example, grub installed into the MBR will check that all components are present, prepare the image to be written in memory, and only in the last step, write the first and second stage bootloaders in one go. Any failure at this stage would be "hardware error", which would also apply to the old version, and until that point, the old version would still work. It is much more likely for a package to indicate success and subsequently fail on reboot because of a missing check, but this is not something the package manager can help with. What already exists is automatic revert if a package fails to unpack because of an I/O error (or the disk being full). - They cannot be removed without being unmarked as "system-critical" We have "Essential: yes", which dpkg protects, and "Protected: yes", which are protected by apt. The latter category is what bootloaders fall in (it also helps that the main author for apt is also a grub maintainer). The dpkg program will allow you to remove the bootloader, because that is what allows changing bootloaders easily, the "Essential" set is basically just what is required for dpkg to function -- so dpkg cannot self-destruct. - The system could check during every shutdown that system-critical packages are in a consistent state, reverting to a known good
Re: Possible workaround/fix for usrmerge issues with dpkg-query -S (#858331 / #848622)
Hi, On 7/8/23 22:55, Eric de Ruiter wrote: The idea is simple: if you specify you want to match on realpath (for now by prefixing the path with "realpath:"); it will search for files matching the basename of the given path and does an extra check to only return matches where the realpath of the given path will match the realpath of the partial match it found. We mostly need to define an API here. The interesting callers do stuff like $ dpkg -S /lib/x86_64-linux-gnu/libselinux.so.1 /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libpcre2-8.so.0 libselinux1:amd64: /lib/x86_64-linux-gnu/libselinux.so.1 libc6:amd64: /lib/x86_64-linux-gnu/libc.so.6 dpkg-query: no path found matching pattern /lib/x86_64-linux-gnu/libpcre2-8.so.0 so multiple file names in a single call, and probably expect the same file name to be returned to match the response back to the query. We'd need a new format, for example like libc6:amd64: /usr/lib/x86_64-linux-gnu/libc.so.6 (/lib/x86_64-linux-gnu/libc.so.6) and a flag to enable that format, for clients to use. I haven't invested much time into that yet, my expectation is that the aliasing infrastructure I've put into the backend handles this nicely, but the frontend is something that needs to be coordinated. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi,
On 6/21/23 20:33, Guillem Jover wrote:
I don't think we disagree (?), I probably didn't express myself clearly.
The fact that no package ships those symlinks *is* and *has* been a
problem, and what I've been saying all along, this will be the only
correct way to let dpkg know whether there will be aliasing in play.
I've looked into building a dpkg-alias tool that would work similar to
dpkg-divert, and currently that looks like it might be a viable solution.
Rules:
- some package will need to register the alias in its preinst and
remove it in its postrm (to make piuparts happy and provide symmetry).
My favourite for that would be systemd-sysv, because that is literally
the package that brings in the requirement, but there might be problems
with that approach for containers, so I suspect that's not a good choice.
- unpacking a symlink over a registered alias is fine if the symlink
and the alias match.
This way, we can ship the symlink in a package for bootstrapping. Terms
and conditions apply.
- dpkg keeps track of the name of files in the .tar.gz, but also
recognizes aliased names as referring to the same file
This can be done inside dpkg's file database -- whenever an entry is
created, additional entries for aliases are also generated along it, so
the file can be looked up using any aliased path
- circular aliases are not allowed
This would break the requirement that it is possible to generate an
exhaustive set of all names a file may be found under.
- the newly created dpkg-alias tool is responsible for moving files,
if necessary
This is a separate tool, so we don't need to extend the unpacking logic,
and we can build an algorithm here that includes error recovery.
- if an alias is registered for a symlink that already exists, that is
not an error
This way, we accept the status quo silently.
- registering an existing alias or unregistering a nonexistant alias
is not an error
This allows future releases to change the list of aliases without
requiring complex logic in maintainer scripts.
- Files remain in the same place during the trixie cycle
We only shift responsibility for moving files and creating the symlinks
during this cycle, but bootstrapping will have to go through an unmerged
phase in the beginning, and files are then moved into merged paths from
the preinst of the key package, after the initial unpack.
Whether bootstrapping tools prefer to create the symlinks themselves
does not really matter at this point -- ideally they wouldn't, because
we'd need to keep track of any symlinks typically created by bootstrap
tools and explicitly remove these if the actual system should not have
them (e.g. if an architecture specific symlink is created on an arch
that doesn't have it).
- Symlinks cannot yet be shipped in data.tar during the trixie cycle
Because the unpack phase during bootstrap creates an unmerged file
system, the symlinks cannot be unpacked here.
- In trixie+1, the symlinks are then created from data.tar, and files
can then be moved.
This allows bootstrap to create merged filesystems directly during the
unpack phase.
- dpkg-alias can fail if there is a conflict during alias registration
This should not actually happen, but protects people who may have local
packages that use unmerged paths.
- dpkg-divert and dpkg-statoverride act on the normalized path
- dpkg-divert and dpkg-statoverride are registered with un-normalized
paths
- it is an error to register a diversion or statoverride with
non-matching data
This should allow most of the handover scenarios. For a diversion, it is
sufficient if the normalized destinations match, so we can have a
handover that registers /usr/lib/x -> /usr/lib/y from the preinst of a
new package and then unregisters /lib/x -> /lib/y from the postrm of an
older one.
The package would need to unregister on upgrade in the postrm though,
but that is standard for removed diversions.
- dpkg-query returns the package name if any aliased name matches
There should also be a flag whether to report the file name from the
data.tar as well, defaulting to "no", because that's what scripts expect.
But given these mentioned constraints
it cannot be made to support (as in accept) unpacking files inside
aliased directories (it should be able to unpack the symlinks creating
those aliased directories though!).
I think that can be done. I have already successfully made it report a
conflict between /bin/testfile and /usr/bin/testfile, with a meaningful
error message, and runtime overhead isn't too bad -- a factor of
log_{262144} 2 on the lookup time for a single path, but inserts got a
bit more expensive because these now have prefix comparisons on the
path. The latter could probably be improved with another hash on the
first N bytes of the path.
dpkg-divert distinguishes between local and package level changes, it
is true that dpkg-statoverride does not have
Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)
Hi, On 5/18/23 18:08, Luca Boccassi wrote: Without it, leaving them in place makes no difference for usrmerged systems, and allows derived distributions that don't need usrmerge to continue using our packages. Not quite. Having packages only ship files under /usr (and possibly /etc) is very much a goal in itself for a lot of us. My point is: that is not an end goal, because it provides no real tangible benefit on its own. It does make sense in the context of building immutable images, which is *also* a bootstrapping problem, and probably worth being supported with proper tooling. - there is no guarantee that usrmerge will be permanent or the last transition of this kind It is permanent, there are several upstream projects that will drop support for legacy layouts very soon, and it will not be re-added back. You are currently building a "legacy" system, it will just take a bit of time to reach that status. The less you anticipate future needs, the faster this will be. I understand that you are also a member of one of these upstream projects, and that you are taking the interests of this project to "pretty much the last relevant holdout" here. Has it occurred to you that you are also wearing a Debian hat, and you could be taking the interests of the Debian project to said upstream project? - it also solves the bootstrap problem It also is the least likely to succeed, and the most likely to cause significant "social" upheavals. The only social problem I see is that you are trying to create a situation in which other people are compelled to do your work for you if they want it to be done properly. So far, you have been throwing out "solutions", and left the analysis of the feasibility of those to other people, then, after three iterations, you demanded a full write-up of all existing use cases and blanket permission to ignore anything not brought up in this list. The thing is: I see more enthusiasm and self-directed problem solving skills from the interns at the company where I work, and at the same time you are one of the top contributors of the upstream project whose ideas of a "supported" configuration we are supposed to follow. Simon
Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)
Hi, On 5/18/23 02:15, Sam Hartman wrote: Helmut> I think at this point, we have quite universal consensus Helmut> about the goal of moving files to their canonical location Helmut> (i.e. from / to /usr) as a solution to the aliasing problems Helmut> while we do not have consensus on precisely how to do this Helmut> (i.e. with changing dpkg or without). If you believe that Helmut> this is not consensus, please speak up. I agree we have strong consensus that we want to move files to their canonical locations. I'm not entirely sure I'd agree that we have consensus that's our solution to the aliasing problem. It's the other way around: moving the files as a solution to the aliasing problem is the strongest argument in favour of moving the files inside the packages. Without it, leaving them in place makes no difference for usrmerged systems, and allows derived distributions that don't need usrmerge to continue using our packages. If for example we accomplish the move to canonical locations by changing dpkg, we might well get some form of aliasing support in dpkg. IMO, that is still the preferred solution: - it is actually safe, because dpkg knows what is going on and can reject conflicting changes - there is no guarantee that usrmerge will be permanent or the last transition of this kind - it also solves the bootstrap problem Simon
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Hi, On 5/12/23 02:51, Luca Boccassi wrote: Or alternatively, we can establish that a documentation/post-facto approach is enough for derivatives, and then that's valid for all changes and transitions. For the context of this bug, the notice *is* the after-the-fact documentation of an interface change, i.e. the bare minimum. It would have been better to get input from derived distributions about this interface change before it happened, but given that the interface change was not caused by a change in dpkg code, but by the introduction of the usrmerge package, there was not much of a remedy available then. Simon
Re: dpkg currently warning about merged-usr systems (revisited)
Hi Luca, [dropping the tech-ctte bug Cc, because most of this mail is irrelevant to that discussion, I'll send a separate mail for that one paragraph] On 5/12/23 02:51, Luca Boccassi wrote: The crux of the issue is that we are hearing how negatively affecting derivatives in any way, even purely theoretically, is a big no-no, in this very same thread and topic. Reaching out and asking for directions/help/whatever is not enough in that context. So it follows that it cannot be enough in this context either, and it must be fixed instead. There are different levels of expectations here: one for things that were explicitly promised, and one for things that are only incidentally the way they are. For example, whether a binary is in /bin or /usr/bin is inconsequential except for the two-stage boot process used by sysvinit, and whether a library is in /lib or /usr/lib is inconsequential except for whether a binary in /bin uses it as part of the first sysvinit boot stage. The systemd boot process has a different early/late split and uses an initramfs as the first stage, so it has no use for the old separation policies[1], and they were only ever promised as part of sysvinit policy, which has been demoted to a status between "probably still useful to someone" and "annoying that it still exists", depending on who you ask. For that reason I think we do have a consensus that moving the files is allowed, and derived distributions may not expect us to follow the superseded policy anymore -- so Devuan will definitely need to provide their own packages for all the early boot stuff, and that is fine, because the split policy has been dropped with the introduction of systemd, even if communication on that has been poor because the people driving it could not even be bothered to update Policy. On the other hand, we need to either keep the interfaces that have been documented and that people still rely on, or deprecate them, document that they have been deprecated, give users time to adapt, and ideally explain the reasoning and alternatives. Anything that is not an interface can be changed, and anyone relying on an implementation detail is on their own. You can get the same information from reading /var/lib/dpkg/diversions or from calling "dpkg-divert --list", but it is immediately obvious which of those is the interface and which is incidental. This discussion is mostly about interfaces that have been broken by the transition: are we supposed to fix them, can we break them more since the sky hasn't fallen, and whose responsibility is it to fix problems when it is disputed whether an interface has been broken. Replacing a directory controlled by dpkg with a symlink is explicitly allowed, but whether the symlink may point to a directory also controlled by dpkg is unclear -- the normative language is silent on this, and the use case in the design document does not anticipate it. If the file system interface on the bottom of dpkg was broken by usrmerge, then it would have been the responsibility of the usrmerge team to coordinate that with the dpkg team. If it was not, then it's the dpkg team's responsibility to fix their data model. Right now, all we have is after-the-fact documentation from the dpkg team that some of the interfaces do not work in the context of usrmerge. That is unfortunate, but not easy to change. #1035904 does not even attempt that, it just aims to remove documentation, that is definitely the wrong approach even in a world where after-the-fact notification instead of pre-transition coordination is the norm. The usrmerge transition so far has been an elaborate game of "I'm not touching you" around existing interfaces to get where it is today[2], but continuing requires changing interfaces. That is a completely valid approach especially in the free software world, but requires a bit more stakeholder management than moving a few files around while keeping them accessible through their old location. You may also have noticed that responsibility for completing it has shifted to different people now that it has hit this wall. We recognize that you are holding the bag here and support you in the search for a minimally invasive solution, but there is also a good chance that none exists and the scope of this change is too large for a bunch of hobbyists without a process. If that is the case, we need to take a step back and remember that we're *also* professionals, so we have the tools and knowledge to get this done, even if it feels a lot like our day job. :P Simon [1] or at least they think so [2] stuck
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Hi, On 5/11/23 10:59, Sean Whitton wrote: Dear ctte, please consider overruling the dpkg maintainer to include the patch from #994388[1]. Currently dpkg contains code to emit the merged-/usr warning, that's dead code on Debian, but which becomes active when packages from the Debian archive are copied unmodified into derivatives. The way I see it (but I'm not a dpkg maintainer), the current implementation is correct, as dpkg does not support aliased directories, but Debian has decided to use it in such an environment nonetheless. The tech-ctte decision not to roll back usrmerge accepts responsibility for this decision, so silencing the warning on Debian is correct, but no one has accepted that responsibility for derived distributions. Any derived distribution can easily go on record and request inclusion in the list of distributions where this warning is suppressed, by typing the phrase "Yes, I understand that this is a bad idea." into an email client. dpkg has an upstream existence that's independent of Debian, and it's perfectly legitimate for that version of dpkg to emit the warning. The problem here might be caused by how the Debian archive is implicitly being used to distribute upstream dpkg. I'm skeptical about splitting development and packaging, though, especially in this context, because we'd need to clarify how far we'd want upstream dpkg and Debian dpkg to deviate. Basically, non-native dpkg has two scenarios: either it is maintained by the same people as now, which causes them extra work for no clear benefit, or we find maintainers willing to deal with complex bug reports that upstream is fully entitled to reject because Debian brought this onto themselves by deciding that one upstream project's "unsupported configuration" needs to be avoided by moving to another upstream project's "unsupported configuration." Those same people who would volunteer to maintain such a package could spend their energy finding a solution that can be merged into "upstream" dpkg, that would be more productive. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 5/8/23 20:38, Luca Boccassi wrote: [local diversions] Sure, they are supported in the sense that they can be enabled, and then you get to keep the pieces. They are supported in the sense that someone actually added an explicit flag for dpkg-divert for specifically this feature and documented it in the manual page as an interface. Maintainer scripts don't need to work around the admin installing arbitrary incompatible tools, because we generally expect admins to know what they are doing -- however requiring admins to perform multiple diversion registrations to have them count is a change of the interface. The /bin/sh diversion is a bit ...special. This should have been an alternative, but cannot be because /bin/sh needs to be functional after unpacking Essential packages, so maintainer scripts work during bootstrap. There is an interesting use case here as well, bootstrapping a foreign architecture, where we only unpack the Essential set, divert /sbin/init to /sbin/init.real and place a shell script as /sbin/init that completes the installation once the resulting tree is mounted as a root filesystem on the target platform. It has been a while since I used that, in principle this should even still work with systemd. Simon
Corner case: can a package owning a diversion disappear?
Hi, There are a few very unlikely corner cases where I'm unsure what would be expected of dpkg: 1. a package not providing a diverted file, and then disappearing pkg-a contains /x and registers a diversion /y -> z. pkg-b contains /y pkg-c Replaces pkg-a and contains /x Installing pkg-c overwrites the last file in pkg-a. Should pkg-a disappear at this point, or does the diversion count? 2. like 1, but no one else provided the diverted file either The same scenario, but only pkg-a and pkg-c are installed, so /y does not even exist and is therefore not redirected to /z. Does pkg-a disappear? 3. a diversion target is replaced pkg-a contains /x and registers a diversion /x -> y pkg-b contains /x and is diverted to /y pkg-c Replaces pkg-b and contains /y Is that allowed? Does pkg-b disappear? Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 5/7/23 18:14, Ansgar wrote: Is there any specific reason why specifically diversions are a problem where "it might work" is not sufficient? That is, why should we divert from the usual standard for dealing with packages outside the Debian ecosystem here? Locally created diversions are a supported feature, and the only way for admins to replace single files in a way that is safe for installing updates. Even within Debian, it is not sufficient to just coordinate uploads of packages that divert each others' files, because the new diversion needs to be in place before a newly-canonicalized package is unpacked, a Breaks relationship does not enforce that ordering, and while a Conflicts without a Replaces does, this adds a lot of constraints to the solver. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 06.05.23 21:28, Luca Boccassi wrote: [shipping usrmerge symlinks in packages] In the far future I'd like for these details to be owned by image builders/launchers/setup processes rather than a package, but this can be discussed separately and independently, no need to be tied to this effort. Ideally I'd like to have this information in a single package rather than distributed over ten different tools, especially as this is also release and platform dependent. If possible, I'd like to go back to the gold standard of - download Essential: yes packages and their dependencies - unpack them using dpkg --fsys-tarfile | tar -x - install over this directory with dpkg --root=... -i *.deb to get something that works as a container. Right now, that only works if I remove "init" from the package list, which is fair since a container doesn't need an init system anyway. The less an image builder needs to deviate from this approach, the better for our users. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 06.05.23 07:11, Luca Boccassi wrote: - every package is forcefully canonicalized as soon as trixie is open for business You will also need to ship at least - /lib -> usr/lib (on 32 bit) - /lib64 -> usr/lib64 (on 64 bit) as a symlink either in the libc-bin package or any other Essential package, to fulfill the requirement that unpacked Essential packages are operational. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: [PATCH] Warn for unknown info files in packages
Hi, On 06.05.23 05:58, Guillem Jover wrote: Debian Policy forbids info files not mentioned in Policy, except if their names start with an underscore to flag them as non-critical. I think you might be mixing up .deb ar members with entries in the control member in the .deb? Oof, yes. -_- Removing this one from my stack, that is probably better handled by a Pre-Depends anyway. Simon OpenPGP_signature Description: OpenPGP digital signature
[PATCH] Warn for unknown info files in packages
Debian Policy forbids info files not mentioned in Policy, except if their
names start with an underscore to flag them as non-critical.
---
src/main/unpack.c | 29 +
tests/t-multiarch/Makefile | 24
2 files changed, 41 insertions(+), 12 deletions(-)
diff --git a/src/main/unpack.c b/src/main/unpack.c
index 9048f6fb0..446f8d4c5 100644
--- a/src/main/unpack.c
+++ b/src/main/unpack.c
@@ -496,9 +496,23 @@ pkg_infodb_update_file(const char *filename, const char
*filetype)
static void
pkg_infodb_update(struct pkginfo *pkg, char *cidir, char *cidirrest)
{
+ char const *const known_info_names[] =
+ {
+"conffiles",
+"control",
+"preinst",
+"prerm",
+"postinst",
+"postrm",
+"triggers"
+ };
+ size_t const num_known_info_names =
+ sizeof known_info_names / sizeof known_info_names[0];
+
struct match_node *match_node;
DIR *dsd;
struct dirent *de;
+ bool known;
/* Deallocate the match list in case we aborted previously. */
while ((match_node = match_head)) {
@@ -567,6 +581,21 @@ pkg_infodb_update(struct pkginfo *pkg, char *cidir, char
*cidirrest)
continue;
}
+/* if the info file name starts with an underscore, then it can be
+ * safely ignored */
+known = (de->d_name[0] == '_');
+
+/* compare against list of files we understand */
+for (size_t i = 0; !known && (i < num_known_info_names); ++i)
+ if (strcmp(de->d_name, known_info_names[i]) == 0)
+known = true;
+
+/* TODO: error? force option? */
+if (!known)
+ warning(_("package %s contains unknown info file '%s'"),
+ pkgbin_name(pkg, >available, pnaw_nonambig),
+ de->d_name);
+
/* Right, install it */
newinfofilename = pkg_infodb_get_file(pkg, >available, de->d_name);
if (rename(cidir, newinfofilename))
diff --git a/tests/t-multiarch/Makefile b/tests/t-multiarch/Makefile
index 31449437b..99e09c9e9 100644
--- a/tests/t-multiarch/Makefile
+++ b/tests/t-multiarch/Makefile
@@ -787,39 +787,39 @@ test-dpkg-output: ma-setup
$(DPKG_PURGE) pkg-ma-shared-files:$(NATIVE_ARCH)
pkg-ma-shared-files:$(FOREIGN_ARCH)
pkg-ma-db-layout_1.0_$(NATIVE_ARCH):: pkg-template
- touch pkg-ma-db-layout_1.0_$(NATIVE_ARCH)/DEBIAN/foo-1
+ touch pkg-ma-db-layout_1.0_$(NATIVE_ARCH)/DEBIAN/_foo-1
pkg-ma-db-layout_2.0_$(NATIVE_ARCH):: pkg-template
- touch pkg-ma-db-layout_2.0_$(NATIVE_ARCH)/DEBIAN/foo-2
+ touch pkg-ma-db-layout_2.0_$(NATIVE_ARCH)/DEBIAN/_foo-2
test-db-layout: ma-setup
$(DPKG_INSTALL) pkg-ma-db-layout_1.0_$(NATIVE_ARCH).deb
test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.list"
- test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.foo-1"
+ test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout._foo-1"
$(DPKG_INSTALL) pkg-ma-db-layout_2.0_$(NATIVE_ARCH).deb
! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.list"
- ! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.foo-1"
+ ! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout._foo-1"
test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).list"
- test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).foo-2"
+ test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH)._foo-2"
$(DPKG_INSTALL) pkg-ma-db-layout_1.0_$(NATIVE_ARCH).deb
test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.list"
- test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.foo-1"
+ test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout._foo-1"
! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).list"
- ! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).foo-2"
+ ! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH)._foo-2"
$(DPKG_PURGE) pkg-ma-db-layout
# Test the upgrade of the db layout
$(DPKG_INSTALL) pkg-ma-db-layout_2.0_$(NATIVE_ARCH).deb
rm -f "$(DPKG_ADMINDIR)/info/format"
mv "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).list"
"$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.list"
- mv "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).foo-2"
"$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.foo-2"
- $(call stdout_has,$(DPKG_QUERY) --control-path pkg-ma-db-layout
foo-2,/pkg-ma-db-layout\.foo-2$$)
+ mv "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH)._foo-2"
"$(DPKG_ADMINDIR)/info/pkg-ma-db-layout._foo-2"
+ $(call stdout_has,$(DPKG_QUERY) --control-path pkg-ma-db-layout
_foo-2,/pkg-ma-db-layout\._foo-2$$)
$(DPKG_CONFIGURE) -a
grep -q "^1$$" "$(DPKG_ADMINDIR)/info/format"
test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).list"
- test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH).foo-2"
+ test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout:$(NATIVE_ARCH)._foo-2"
! test -e "$(DPKG_ADMINDIR)/info/pkg-ma-db-layout.list"
- ! test
[PATCH 2/2] Mark functions in headers as inline
---
lib/dpkg/perf.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/dpkg/perf.h b/lib/dpkg/perf.h
index a46792a43..48e69ccde 100644
--- a/lib/dpkg/perf.h
+++ b/lib/dpkg/perf.h
@@ -47,7 +47,7 @@ perf_ts_sub(struct timespec *a, struct timespec *b, struct
timespec *res)
}
}
-static void
+static inline void
perf_ts_mark_print(const char *str)
{
struct timespec ts;
@@ -57,7 +57,7 @@ perf_ts_mark_print(const char *str)
printf("%lu.%.9lu: %s\n", ts.tv_sec, ts.tv_nsec, str);
}
-static void
+static inline void
perf_ts_slot_print(struct perf_slot *ps, const char *str)
{
struct timespec t_res;
--
2.39.2
[PATCH 1/2] Make headers self-contained
--- lib/dpkg/command.h | 2 ++ lib/dpkg/db-fsys.h | 2 ++ lib/dpkg/parsedump.h | 1 + 3 files changed, 5 insertions(+) diff --git a/lib/dpkg/command.h b/lib/dpkg/command.h index 7d2098a29..09ec92ac7 100644 --- a/lib/dpkg/command.h +++ b/lib/dpkg/command.h @@ -23,6 +23,8 @@ #include +#include + DPKG_BEGIN_DECLS /** diff --git a/lib/dpkg/db-fsys.h b/lib/dpkg/db-fsys.h index cd9abefaf..3a9696aa4 100644 --- a/lib/dpkg/db-fsys.h +++ b/lib/dpkg/db-fsys.h @@ -27,6 +27,8 @@ DPKG_BEGIN_DECLS +struct pkgbin; + /* * Data structure here is as follows: * diff --git a/lib/dpkg/parsedump.h b/lib/dpkg/parsedump.h index 4f03e87f7..c43da7ff6 100644 --- a/lib/dpkg/parsedump.h +++ b/lib/dpkg/parsedump.h @@ -26,6 +26,7 @@ #include #include +#include DPKG_BEGIN_DECLS -- 2.39.2
[PATCH] Ignore TAGS
This file can be optionally built, and is helpful in navigating the source tree, but should never be checked in. --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index a5edb85b8..19bdd9e73 100644 --- a/.gitignore +++ b/.gitignore @@ -17,6 +17,7 @@ .libs/ Makefile Makefile.in +TAGS # Top dir ignores autom4te.cache/ -- 2.39.2
[PATCH] Rename .gitinore -> .gitignore
This seems to be a mistake.
---
tests/dpkginst/{.gitinore => .gitignore} | 0
1 file changed, 0 insertions(+), 0 deletions(-)
rename tests/dpkginst/{.gitinore => .gitignore} (100%)
diff --git a/tests/dpkginst/.gitinore b/tests/dpkginst/.gitignore
similarity index 100%
rename from tests/dpkginst/.gitinore
rename to tests/dpkginst/.gitignore
--
2.39.2
[PATCH] Factor out common code for reloading database files
---
lib/dpkg/Makefile.am| 1 +
lib/dpkg/db-fsys-common.c | 84 +
lib/dpkg/db-fsys-divert.c | 43 ---
lib/dpkg/db-fsys-override.c | 50 +-
lib/dpkg/dpkg-db.h | 13 ++
5 files changed, 117 insertions(+), 74 deletions(-)
create mode 100644 lib/dpkg/db-fsys-common.c
diff --git a/lib/dpkg/Makefile.am b/lib/dpkg/Makefile.am
index 3403338ce..45c3e01ff 100644
--- a/lib/dpkg/Makefile.am
+++ b/lib/dpkg/Makefile.am
@@ -79,6 +79,7 @@ libdpkg_la_SOURCES = \
db-ctrl-access.c \
db-ctrl-format.c \
db-ctrl-upgrade.c \
+ db-fsys-common.c \
db-fsys-digest.c \
db-fsys-divert.c \
db-fsys-files.c \
diff --git a/lib/dpkg/db-fsys-common.c b/lib/dpkg/db-fsys-common.c
new file mode 100644
index 0..fc52e9f8b
--- /dev/null
+++ b/lib/dpkg/db-fsys-common.c
@@ -0,0 +1,84 @@
+/*
+ * libdpkg - Debian packaging suite library routines
+ * db-fsys-common.c - Common handling for admindb files
+ *
+ * Copyright © 2008-2012 Guillem Jover
+ * Copyright © 2022 Simon Richter
+ *
+ * This is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ */
+
+#include
+#include
+
+#include "dpkg-db.h"
+#include "dpkg.h"
+#include "i18n.h"
+#include "debug.h"
+
+#include
+#include
+
+#include
+#include
+
+FILE *dpkg_db_reopen(
+ struct dbinfo *info,
+ char const *filename,
+ bool *unchanged)
+{
+ struct stat sb_next;
+ FILE *file_next;
+
+ assert(info);
+
+ onerr_abort++;
+
+ file_next = fopen(filename, "r");
+ if (!file_next) {
+ if (errno != ENOENT)
+ ohshite(_("failed to open %s"), filename);
+ } else {
+ setcloexec(fileno(file_next), filename);
+
+ if (fstat(fileno(file_next), _next))
+ ohshite(_("failed to fstat %s"), filename);
+
+ /*
+* We need to keep the database file open so that the
+* filesystem cannot reuse the inode number (f.ex. during
+* multiple dpkg-divert invocations in a maintainer script),
+* otherwise the following check might turn true, and we
+* would skip reloading a modified database.
+*/
+ if (info->file &&
+ info->sb.st_dev == sb_next.st_dev &&
+ info->sb.st_ino == sb_next.st_ino) {
+ fclose(file_next);
+ onerr_abort--;
+ debug(dbg_general, "%s: %s is unchanged, skipping",
+ __func__, filename);
+ *unchanged = true;
+ return NULL;
+ }
+ info->sb = sb_next;
+ }
+ if (info->file)
+ fclose(info->file);
+ info->file = file_next;
+
+ onerr_abort--;
+ *unchanged = false;
+ return file_next;
+}
diff --git a/lib/dpkg/db-fsys-divert.c b/lib/dpkg/db-fsys-divert.c
index bf0b7ff5a..e3bdcadf4 100644
--- a/lib/dpkg/db-fsys-divert.c
+++ b/lib/dpkg/db-fsys-divert.c
@@ -23,7 +23,6 @@
#include
#include
-#include
#include
#include
@@ -45,54 +44,28 @@ static char *diversionsname;
void
ensure_diversions(void)
{
- static struct stat sb_prev;
- struct stat sb_next;
+ static struct dbinfo info;
+ bool unchanged;
char linebuf[MAXDIVERTFILENAME];
- static FILE *file_prev;
FILE *file;
struct fsys_diversion *ov, *oicontest, *oialtname;
if (diversionsname == NULL)
diversionsname = dpkg_db_get_path(DIVERSIONSFILE);
- onerr_abort++;
+ file = dpkg_db_reopen(, diversionsname, );
- file = fopen(diversionsname, "r");
- if (!file) {
- if (errno != ENOENT)
- ohshite(_("failed to open diversions file"));
- } else {
- setcloexec(fileno(file), diversionsname);
-
- if (fstat(fileno(file), _next))
- ohshite(_("failed to fstat diversions file"));
-
- /*
-* We need to keep the database file open
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 05.05.23 18:36, Timo Röhling wrote: - it is not an error to register a diversion for an alias of an existing diversion, provided the package and target matches, this is a no-op - it is not an error to unregister a diversion for an alias of a path that has been unregistered previously, that is a no-op as well How do you distinguish between aliased diversions and "real" ones? Because if you allow the registration of duplicate diversions, the following can happen: - Package A is installed, preinst creates a diversion - Package B is installed, preinst creates the same diversion - Package A is uninstalled, postrm removes the diversion That's why the package and target need to match -- so by extension package B may not divert a path that package A already diverts. With dpkg-divert as it is, this works because no aliasing support exists. A diverted dpkg-divert that generates additional diversions would need to catch this case, reject it and restore the previous state, same as an aliasing aware dpkg-divert. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 04.05.23 20:26, Helmut Grohne wrote: From my point of view, the ultimate goal here should be moving all files to their canonical location and thereby make aliasing effects irrelevant. Do you confirm? Yes, that would solve the problem for the current transition without any changes in dpkg. There will be some small aliasing effects still, e.g. ldd will probably output the linker path from the ELF ABI instead of the canonical linker path, but nothing major. What I'm concerned about is that, whether we like it or not, we're still (d|r)efining interfaces here, and that affects future development, including future transitions. As such, I do not see aliasing support in dpkg as complementing the forced file move approach with lots of workarounds such as diverting dpkg-divert. Rather, I see them as exclusive strategies. Yes, absolutely. No one can coordinate two groups that don't even agree on the scope of the work or whether a definition of the scope is even necessary before beginning. Each of these strategies has significant downsides. In combining the different strategies, we combine their downsides, but since their benefit is shared, we do not gain anything in return but increase the price to pay. For this transition, yes, but we need to think beyond that. Any workaround we introduce here will need one release to be rolled out, one release to be stable, and one release to disappear. We can, in many but not all cases, introduce dependencies to enforce a particular ordering and reduce that by one release cycle. We're too late to roll out workarounds in the bookworm cycle, so we can deploy them with trixie at the earliest. Essential packages cannot contain diversions, so we need support from all installers (debootstrap, cdebootstrap, mmdebootstrap) or leave file paths for Essential packages as they are, delaying the end of the transition until forky. Making the workarounds obsolete before that will allow us to finish the transition earlier and thus reduce the effort of future transitions. For the users, nothing changes, they have had the files moved forcefully years ago. At the same time, there is little overlap in the groups that would implement the different strategies, so there is no person who would need to decide what they are working on to the detriment of the other option. On the flip side, if dpkg (and thus dpkg-divert) is to gain aliasing support, I see no reason (and benefit) to diverting dpkg-divert. Me neither. The problem is, we have no commitment from anyone to implement aliasing support. Volunteering Guillem hasn't worked, I don't really have time to do more than "best effort", and I don't know many other people even looking into that, so we cannot decide on that strategy. Can you explain why you see combining these strategies as something worth exploring? It will get something done on the way to finishing the transition until forky, even if dpkg changes don't manifest, but can provide a solution in trixie if they do. then a package containing /bin/foo and a package containing /usr/bin/foo now have a file conflict in dpkg. Not sure if that is a problem, or exactly the This case already is prohibited by policy section 10.1. It can only happen as a temporary state during a file move (from / to /usr and from one package to another). Yes, but from a technical point of view we cannot rely on Policy, it only allows us to assign blame, but that will not fix the error message a user is experiencing. With the status quo, dpkg will not detect the file conflict, which may or may not be the desired result, but it will not flag an error during execution. With two file names being diverted to the same name, it is likely (again, I haven't tested) that installation of the conflicting package will be aborted unless there is a Replaces: in effect, which is something that whatever tool is calling dpkg will need to deal with. behaviour we want. Probably the latter, which would allow us to define a policy "if aliased paths are diverted, the diversion needs to match", which in turn would allow the conflict checker during alias registration to verify that the aliased diversions are not in conflict. If we do not modify dpkg to improve aliasing support, then yes, such a scenario will require a Conflicts declaration or a different measure averting this problem. No, that would require the package declaring the diversion to conflict with itself, that makes no sense. We need to clarify the interface for dpkg-divert here (i.e. create Policy), and that interface will need to be supported by both the diverted dpkg-divert and the one shipped with dpkg. As of now, the one in dpkg will happily register diversions for aliased paths, but to my knowledge, none have been registered, and the diverted dpkg-divert does not exist yet, so we have full freedom to define what the policy should be. My proposal would be to put the onus on
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 03.05.23 19:19, Helmut Grohne wrote: What still applies here is that we can have usr-is-merged divert /usr/bin/dpkg-divert and have it automatically duplicate any possibly aliased diversion and then the diverter may Pre-Depends: usr-is-merged (>=...) to have its diversions duplicated. Of course, doing so will make usr-is-merged very hard to remove, but we have experience here from multiarch-support. For aliasing support in dpkg, that means we need a safe policy of dealing with diversions that conflict through aliasing that isn't "reject with error", because the magic dpkg-divert would always generate conflicts. One thing we need to check is whether diversions to the same target cause file conflicts -- I think they should. So if you divert /bin/foo -> /usr/bin/foo.dontdelete /usr/bin/foo -> /usr/bin/foo.dontdelete then a package containing /bin/foo and a package containing /usr/bin/foo now have a file conflict in dpkg. Not sure if that is a problem, or exactly the behaviour we want. Probably the latter, which would allow us to define a policy "if aliased paths are diverted, the diversion needs to match", which in turn would allow the conflict checker during alias registration to verify that the aliased diversions are not in conflict. The diverted dpkg-divert would probably generate extra register/unregister calls as soon as dpkg-divert itself is aliasing aware, but all that does is generate warning messages about existing diversions being added again, or nonexistent diversions being deleted -- these are harmless anyway, because maintainer scripts are supposed to be idempotent, and dpkg-divert supports that by not requiring scripts to check before they register/unregister. And of course, we can always draw the diversion card and have usr-is-merged divert /usr/bin/update-alternatives to have it canonicalize paths as required to be able to migrate alternatives in a sane way (from a consumer point of view). We get to draw this card exactly once, and any package that would need the same diversion would need to conflict with usr-is-merged, which would make it basically uninstallable. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 30.04.23 03:08, Marvin Renich wrote: My understanding from following this thread (and others) is that dpkg has a bug that can easily be triggered by a sysadmin replacing a directory with a symlink (and some other necessary conditions that don't happen very often), which is explicitly allowed by policy. No, it becomes "easily" triggered because usrmerge aims to move the files in the packages, which would cause /bin to become empty at some point in the future, at which point dpkg correctly needs to remove it. As long as /bin/ls exists in coreutils, the /bin -> usr/bin symlink will be safe from deletion. The --add-alias solution that has been suggested in this thread seems like it would fix the general problem iff policy was changed to require sysadmins to use it if they replaced a directory with a symlink. Yes, we want something that is basically --add-alias that returns us to a state where dpkg's model is an accurate representation of reality. I do not understand why the dpkg maintainer has rejected this solution; I've understood this to be mostly an issue with the quality of the patches submitted, not the general idea. Dpkg has a really extensive test suite, I have not so far seen any patch being submitted that even adds test cases there. My branch adds six, that is still ridiculously low. it would still be a fix for the general bug after the usrmerge transition has completed. And it would be at least one order of magnitude more performant than scanning the filesystem for directory symlinks. That, and as far as I can see, fixing the general problem will also be a prerequisite to finishing the transition in the first place because there are lots of subtle little things caused by the model mismatch, and the workarounds solve the problem at hand but further entrench the mismatch. Simon PS: Current state of my branch: Selecting previously unselected package pkg-usrmerge. (Reading database ... 6 files and directories currently installed.) Preparing to unpack pkg-usrmerge.deb ... D01: process_archive oldversionstatus=not installed /home/geier/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb/tmp.ci/preinst install D01: ensure_canonical: new, (re)loading D01: ensure_canonical: found mapping /bin -> /usr/bin D01: ensure_canonical: mapping /bin -> /usr/bin seems to be new D01: create_alias: alias /bin/test-conflict -> /usr/bin/test-conflict conflicts with pkg-b(0) OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Thu, Apr 27, 2023 at 12:34:06AM +0200, Helmut Grohne wrote: > Ok, let's move on. I've proposed diversions as a cure, but in reality > diversions are a problem themselves. Consider that > cryptsetup-nuke-password diverts /lib/cryptsetup/askpass, which is > usually owned by cryptsetup. If cryptsetup were to move that file to > /usr, the diversion would not cover it anymore and the actual content of > askpass would depend on the unpack order. That's very bad and none of > what I proposed earlier is going to fix this. Another question: how would this interact with a patch that teaches dpkg to do aliases properly, because such a patch would affect diversion handling as well -- specifically, aliases mean that diversions are also aliased (I believe my patch implicitly does this right, but I think I just got 60 more testcases to write), and that diversion targets are implicitly redirected to a resolved form (I don't do that yet, but it's simple to add to my patch). I think the "divert, but do not rename" approach itself should be fairly safe, because all it does is make a deletion fail. Registering the diversion to the new package should be sufficient to make sure the newly unpacked file is not diverted. This probably needs some additional undo operations for failed installs/upgrades, so the diversion is properly removed in these cases (there is no guarantee that postinst will be called after preinst, we could also end up in postrm). > So how do we fix diversions? Let's have a look into the dpkg toolbox > again. I've got an idea. Diversions. What you say? How do you fix > diversions with diversions? Quite obviously, you divert > /usr/bin/dpkg-divert! And whenever dpkg-divert is instructed to add a > diversion for a non-canonical path, you forward that call to the real > dpkg-divert, but also call it with a canonicalized version such that > both locations are covered. When initially deploying the diversion of > /usr/bin/dpkg-divert, we also need to transform existing diversions. Ouch, if you deploy that, I will definitely need to add diversion merging code to alias registration. That's another 60 testcases, but we need defined behaviour for that anyway. Transforming existing diversions: yes, if you can find out about them without looking at dpkg internal files. It may very well be necessary to update the file format on one of these, and if that would cause your script to create nonsense diversions, that'd be another thing we'd have to work around while building real aliasing support. My current mood is "I'd rather focus on a proper solution, not another hack that needs to be supported by the proper solution." Anything we build here that is not aliasing support for dpkg, but another "shortcut" will delay aliasing support for dpkg because it adds more possible code paths that all need to be tested. Keep in mind that we're here because someone took a shortcut, after all. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Thu, Apr 27, 2023 at 12:34:06AM +0200, Helmut Grohne wrote: > At some point the question becomes: Do we want that complexity inside > dpkg (aka DEP 17 or some variant of it) or outside of dpkg (i.e. what > we're talking about here). It seems clear at this time, that complexity > is unavoidable. My gut feeling is that returning to "dpkg's model is an accurate representation of the file system" will be less complex to manage long-term. For this to work, the model needs to be able to express reality, so I guess we can't avoid updating dpkg. I'm also not convinced that the current filesystem layout will remain as it is, for example I can see a use case for installing kernel modules outside of /usr. It would be great to have a generic mechanism here, and be able to do transitions like these without inventing new tools every time. Also, the more we can do in a descriptive framework, the better we can do static checks. The main reason we can argue about what packages are affected is that we have a database of what files are installed where, and that still accurately reflects reality, so we can apply a transformation onto this data and check for conflicts -- but we cannot see diversions in this database as these are created from imperative code. So my fear is that if we create a workaround here that is implemented as imperative code in pre/postinst, this will be invisible to whoever plans the next transition, so this would create immense technical debt. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Tue, Apr 25, 2023 at 09:07:28PM +0200, Helmut Grohne wrote: > This and /bin/sh is the kind of files I'd consider important. And then > upon thinking further it became more and more difficult for me to make > sense of the objection. On a merged system, we can just move that file > to its canonical location without having any trouble even with an > unmodified dpkg. So from my pov, the question about important files can > be disregarded. I hope Simon Richter agrees. Yes, the relevant code at https://github.com/guillemj/dpkg/blob/main/src/main/unpack.c#L749 already handles moving a file inside the same package, and that has existed for some time, that's why I use two packages for the PoC. I have not looked for more issues beyond that, so there might be others lurking in the depths of this code. What I'm mostly concerned about (read: have not verified either way) with /lib/ld.so and /bin/sh is what happens when dpkg learns of /bin and /lib as symlinks -- because right now, the symlinks created by usrmerge are protected by the rule that if dpkg expects a directory and finds a symlink, that is fine because that is obviously an action taken by the admin. But if dpkg sees a package containing these as symlinks, then this is entered into the dpkg database, and subject to conflict resolution, and for that, a separate rule exists that directory-symlink conflicts are resolved in favour of the directory, so the interaction between a newer base-files packages shipping /lib as a symlink and an older or third-party package containing /lib as a directory (e.g. a kernel package from a hosting provider) could overwrite the /lib symlink. It might be possible to avoid that by never shipping /lib as a symlink and always creating it externally, but I still think that's kind of wobbly. > If we look deeper into the dpkg toolbox, we pause at diversions. What if > the new package were to add a (--no-rename) diversion for files that are > at risk of being accidentally deleted in newpkg.preinst and then remove > that diversion in newpkg.postinst? Any such diversion will cause package > removal of the oldpkg to skip removal of the diverted file (and instead > deleted the non-existent path that we diverted to). Thus we retain the > files we were interested in. O_O Yes. Hahahahaha yes. Brittle, but it could work. What is a bit annoying is that we'd have to keep this for an entire cycle. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi Helmut, On 22.04.23 10:44, Helmut Grohne wrote: Dpkg already has defined behaviour for directory vs symlink: the directory wins. In principle a future version of dpkg could change that, but /lib/ld-linux.so.2 is just too special, we'd never want to have a package that actually moves it. Your argument of the dynamic linker being too special is interesting. I certainly agree that we must take great care at moving it, but on the flip side I do not consider the transition complete until we reach the point where we have moved it. I think that if dpkg gains a useful level of support for aliases, it will no longer matter which path is used in the libc package, and I'd consider the transition complete then because it no longer requires any kind of coordination. Do you actually see us coping with some aliases (e.g. the /lib one) eternally? I think that dpkg will need to support them eternally, so package maintainers don't have to. That's why I went with "this needs to be a separate mechanism." Or do you want to say that we need a new mechanism to be able to move such important files? I meant a new mechanism to make sure creation of aliases is not accidentally triggered because of the difference between mkdir -p $(DESTDIR)foo ln -s ../bar $(DESTDIR)foo and mkdir -p foo ln -s ../bar $(DESTDIR)foo which is a totally understandable mistake to make in an install script. I did not have this goal in mind, but now that you mention it, it seems important to me. It is not clear to me though how that control file actually gets us there. In your picture, which component is in charge of actually creating the symbolic links on the filesystem? Can you go into detail as to how you imagine that bootstrap without special-casing? The symlink would still need to be in data.tar, but the control file is responsible to get the symlink also registered as "this is intentionally meant as an alias." The reason it needs to be in data.tar is the two-stage installation used for (c)debootstrap --foreign, where we stop after the unpack stage, and replace init by a shell script that runs dpkg to actually install the packages. This works even when the host cannot execute the target architecture -- just the first boot takes a little longer. The appropriate requirement in Policy would be that Essential packages need to be functional after unpack only, when no maintainer scripts have been run, so if ld.so is in /usr/lib, then /lib -> usr/lib needs to be a symlink in a data.tar, and if /ld.so is in /lib, then /lib needs to be a directory, so some coordination is needed there for the installer to remain working. The advantage is that it works for Essential packages, like the one shipping /lib/ld-linux.so.2. In the --add-alias variant, I think we would still move the dynamic linker to /usr and ship the /lib symlink in base-files eventually. I admit that it is not entirely clear how such a move could be performed safely, but that seems like a solvable problem to me. In that way, I fail to see the control file being an advantage. When the alias is registered, we need to look at the existing state, because we might be inside a new installation (which has exactly what is in data.tar), an upgrade of a merged installation, or an upgrade of an unmerged or partially merged installation, and we want to get into a consistent state as quickly as possible and continue from there. I think the appropriate time would be preinst/postrm, or close to that. Having a control file would mainly give us a declarative interface that always works at the right time (and TBH I'd like to move diversions and alternatives to a declarative interface at some point as well, so we can have archive-wide static analysis, and fewer maintainer scripts overall, which should give a nice speed boost, but that's not part of the usrmerge thing). Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 22.04.23 11:36, Helmut Grohne wrote: For clarity let me propose the following requirements for the definition of done: * All files shipped in Debian packages are shipped in their canonical location rather than an aliased path. With proper support in dpkg, that is even somewhat optional. * The symbolic links are part of some essential package. Yes, but simple symbolic links in packages would lose against directories in other packages due to the symlink-vs-directory rule. Treating these as a conflict in the future instead of silently resolving them is not sufficient for transition handling, and making symlinks win over directories and cause aliases to be created automatically will cause curious and interesting effects if someone makes a mistake in a .links file while packaging. I'd ship the aliases as part of base-files, because that is Essential and has very few reverse dependencies, so giving it a Pre-Depends on a recent dpkg version does not cause ordering issues on upgrade. The Pre-Depends is ignored during bootstrap, but that is fine as the dpkg that is about to be installed is already new enough if it is from the same distribution. In addition, derived distributions that don't want usrmerge will likely ship their own base-files package. * Therefore no special handling code is needed in bootstrapping tools for future releases. For the requirement that unpacking Essential packages gives enough of a running system that installing packages is possible, this unpack phase needs to leave us with a working ld.so in the place the ABI standard requires it, so we probably need to ship the symlinks as symlinks in data.tar.gz, and register them as aliases from metadata so they are not overwritten by a directory. The alternative would be to keep shipping ld.so in /lib, and let dpkg do usrmerge during the phase where the unpacked Essential packages are overwritten using dpkg. We won't really be able to avoid at least some ugliness there, given the interaction between usrmerge and the ABI standard. The alternative would be a consensus that dpkg is simply not expected to always leave the system in a useful state if it encounters certain invalid situations My main reason for disliking it is that it shifts the effort from the proponents to everyone else. The aliasing effects consume so much mental capacity of regular package maintainers Exactly that's what I'm feeling as well, and why I want dpkg to be able to reject invalid packages safely so if a maintainer makes a mistake, this can simply be fixed by an updated package. Testing alone will be an absolute nightmare because we can enter invalid states through multiple avenues, for example, if I have a conflict a.deb: /bin/test b.deb: /usr/bin/test c.deb: /bin -> /usr/bin Arguably, we can rule out a lot of test cases by policy. We already have policy that forbids the combination of a.deb and b.deb in Debian. This property can relatively simply be checked on a distribution level and therefore I think we can entirely skip this case. No, dpkg needs to be able to handle situations forbidden in Policy. File conflicts without declaring "Conflicts:" are also forbidden in Policy, but detecting these is one of the main functions of dpkg. # move file to /usr, install symlink, then remove symlink, move back dpkg -i a.deb c.deb dpkg --remove c.deb Again, I think c.deb would likely be essential and since removal of essential packages is undefined, we can remove such cases from our test matrix. Essential packages can become non-Essential in a later release, can change or be replaced, including piecewise. If we ship the symlinks inside the data.tar, then at least the latter is very unlikely, but I'd think that there needs to be a way to remove aliases. The only thing we could argue that this can be delayed for a release or two, but this would introduce additional error paths into package removal, and possibly create situations that the package resolver does not understand, so I'd rather not. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi,
On 21.04.23 15:03, Raphael Hertzog wrote:
Here you are considering all files, but for the purpose of our issue,
we can restrict ourselves to the directories known by dpkg. We really
only care about directories that have been turned into symlinks (or
packaged symlinks that are pointing to directories). That's a a much lower
number of paths that we would have to check.
Having all paths in the database is cheaper, because doubling the number
of paths multiplies the (average) cost by log_{262144} 2 only, and we do
significantly more lookups than inserts.
The other problem is that we do not know all of these paths, because the
file system has been modified externally without informing dpkg. The
closest thing we can do is scan everything that is supposed to be a
directory.
As an additional complication, dpkg silently resolves
symlink-vs-directory conflicts in favour of the directory (which happens
seldom, but third-party tools sometimes generate broken packages like
that, so it is useful to keep it that way).
Thus this time-consuming operation would be done once, the first
time that the updated dpkg starts and when /var/lib/dpkg/aliases
does not yet exist.
That is still a public interface. :/
In any case, now that you have a database of aliases, you can do the other
modifications to detect conflicting files and avoid file losses.
How does that sound?
Alas, that is the easy part. My branch already implements most of that,
including the logic to trigger a reload after a maintainer script if the
stat information changed (like for diversions).
The proposal I made above is not a real database in the sense that we
don't record what was shipped by the .deb when we installed the files...
it's rather the opposite, it analyzes the system to detect possible
conflicts with dpkg's view of the system.
That is going to be slow, and it changes dpkg's public interface to a
more complex one where our tight loop that handles unpacking files gains
additional error states.
It can be seen as complimentary to it. In any case, I don't see how
implementing metadata tracking would help to solve the problem that we
have today. dpkg would know that all .deb have /bin as a directory and
not as a symlink, and it would be able to conclude that the directory
has been replaced by a symlink by something external, but that's it.
It should still accept that replacement and do its best to work with it.
That means there are two sources of truth: packages and the file system.
We then need a (lowercase) policy how to resolve conflicts between
these, which becomes our public interface, and thus part of (uppercase)
Policy.
I'd also single out the usrmerge transition here. This package operates
in a grey area of Policy where technically a grave bug is warranted
because it manipulates files belonging to other packages without going
through one of the approved interfaces, but since we accidentally
shipped that, we need to deal with it now. That does not mean this is
acceptable, it just wasn't enforced.
To me it would also be acceptable to just hardcode "if usrmerge or
usr-is-merged is installed, take over the known aliases and silently
discard that package", then salt the earth in dak that no package of
this name can ever be shipped again until bookworm+3.
That would be significantly easier than finding a generic solution that
covers all existing use cases.
Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Fri, Apr 21, 2023 at 02:15:54PM +0200, Raphael Hertzog wrote: I'd like to express some disappointment that nobody replied publicly sofar. There were a few replies on the dpkg mailing list. Last year's developer survey concluded that "Debian should complete the merged-/usr transition" was the most important project for Debian [1] (among those proposed in the survey). That's what we are trying to do here and it would be nice to build some sort of consensus on what it means in terms of changes for dpkg. The first thing we need consensus on, IMO, is the definition of "complete". The maintainers of the usrmerge package consider the status quo an acceptable technical solution, so their definition of "complete" is to roll out the change to the remaining users. Other people view the transition as complete when the transition handling code can be removed, which, with the current implementation, is bookworm+2 because that is the first time we could ever have ld.so in /usr/lib. The handling code for bookworm can never be removed from the various bootstrapping tools, because users will rightfully expect to be able to install older versions. I believe cross debootstrap is also currently broken. The problem for dpkg is that there is already a considerable amount of cases it handles. Paths are owned by specific packages, ownership can be transferred, they can be marked as conffiles and diverted. That alone gives us a massive testsuite already, including such elaborate cases as a package providing two files that are subsequently replaced by files in two other packages so no files remain, which automatically makes the first package disappear. Extending these cases to include aliases is a monumental task, and limiting the allowed combinations does not reduce that, it only inverts the expected return code from that test case. We need defined behaviour for what happens if someone tries to register an alias for /etc (not allowed because there are conffiles inside), and we need error handling paths for cases like /lib -> /usr/lib, where creation of the /lib symlink fails and the original /lib needs to be restored before exiting with an error. There are several possible designs here, including a variant that is similar to dpkg-divert, which is called from a maintainer script, safely performs the filesystem transformation and updates dpkg's database, which is then reloaded after the maintainer script finishes. This variant has the drawback that even in future releases we can never move ld.so to /usr in the packages, because this would be safe only if we could guarantee that the alias is registered. We can approximate that by giving ownership of the alias to a package, which allows declaring a dependency, but then the maintainer scripts need to be able to handle transfer of ownership at some point or we're again creating technical debt. Another variant would be a declarative approach, which would, conveniently, work for initial installation for foreign architectures, and permit moving files to /usr eventually (one or two releases later). The drawback there is that we need a defined point when that declaration is read, and at this point the main dpkg binary (which for cross installation is the host binary) is responsible for moving the files over. This also gives strong ownership of the alias to a package, but now it is dpkg that is responsible for ownership transfer and removal of the alias when the last package providing it vanishes. That solution again spans up a massive space of possible edge cases. Either we handle the alias through the normal file handling code, which gives us ownership transfer for almost free, but introduces lots of special cases into that code, or we add separate handling, which introduces the special case where all files in a package providing an alias are gone and the package should be deinstalled -- do we deinstall when the alias is gone as well, or when all regular files are? This is the problem with doing it right: there are so many corner cases that need to be handled, even if the handling consists of an error message about an unsupported configuration and a rollback to the last valid state. The alternative would be a consensus that dpkg is simply not expected to always leave the system in a useful state if it encounters certain invalid situations, and hoping that we will also be able to point to a few million installations where that has not exploded and call it a success, but that would need to be communicated. I know that Guillem (dpkg's maintainer) is generally opposed to the approach that Debian has followed to implement merged-/usr but I have yet to read his concerns on the changes proposed here (besides the fact that they ought to not be needed because we should redo the transition in another way, but that's a ship that has sailed a long time ago...). A sensible solution needs to be able to perform the
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 08.04.23 04:35, Guillem Jover wrote: A little later, Simon Richter also looked into the problem (https://lists.debian.org/debian-dpkg/2022/12/msg00023.html), but remained silent after the initial post. Yes, I am quite busy, but it's not forgotten. I keep adding new test cases. * dpkg cannot safely and atomically perform such switches (and I don't see it ever being able to portably do so, so I don't see ever supporting that). The dpkg core cannot, but the approach by the diversions mechanism can be replicated here: a dedicated tool that adds an alias, and performs the necessary filesystem change. Dpkg already stat()s the global diversions list after executing a maintainer script, for precisely this reason. The difference to diversions is that cleanup after reload is more complex, but that only happens when the list of aliases changes, and that is pretty much not a hot path. Maintaining all the aliased paths in the database is not expensive, we double the number of entries, but it's still a hashmap with 262144 buckets. The most annoying thing is that it finally breaks statistics as the number of nodes now no longer corresponds to the number of files. * No packages ships those symlinks (and none should! as that would currently imply having the same pathname contain different file types on the same system, introducing ordering issues and file type conflicts). Dpkg already has defined behaviour for directory vs symlink: the directory wins. In principle a future version of dpkg could change that, but /lib/ld-linux.so.2 is just too special, we'd never want to have a package that actually moves it. That's why I went with "this needs to be a separate mechanism." The reason to use a control file instead of a tool would be to install the alias from an Essential package, so the old-school "unpack essential packages, then overwrite with dpkg" approach to system installation would work again without special-casing usrmerge in debootstrap * Wants to be a generic change but it is really targeted to this specific mess. This mechanism could have made the transition easier, so it is a generally useful mechanism. Also, this could possibly be built so unregistering the aliases will effektively do an usrunmess. * This information can get out of sync with reality, as it adds an additional and unconnected with anything source of truth, that dpkg cannot do anything about if it diverges (in contrast to diversions or statoverrides f.ex.). This can never happen when that information comes from the real source of truth (the fsys metadata via the .deb). Yes, we need to go back to dpkg as source of truth, but pragmatically we need to pick up existing installations where they are, including those with incomplete usrmerge transformations (I have a few of those, because usrmerge got intermittently installed there). Basically, until the alias is registered, we wouldn't be worse off than the status quo, and picking up the existing install is as simple as "if the symlink already exists when the alias is registered, we don't need to do anything." * This also adds undue complexity, by supporting those as admin aliases. The admin generated redirecting symlinks are already annoying, I'd rather not add further to that pile. I don't really want to support admins doing this (dpkg-divert does not even support diverting a directory). That's a policy decision. I think it's completely valid to have a mechanism that is only to be used by other packages (but that's also part of the reason why my proposal was to use a control file). [ As an aside, I think ideally eventually nothing distro provided should be allowed to be installed within an aliased dir, and dpkg should eventually just error out in those cases, which eventually would get rid of the aliasing problems and any such complexity (I'm not sure how or when that would be feasible though, but obviously in Debian at least not until nothing ships files there). ] There'd be no sensible upgrade path though. It was suggested that the mapping could be managed via a special control file `canonical`. Given that aliasing is not a common operation, the benefit of handling it declaratively is minor. Beyond that, aliasing can also happen as an customization issued by an administrator. Therefore, a command line based approach is preferred. The advantage is that it works for Essential packages, like the one shipping /lib/ld-linux.so.2. As long as the package does not provide the symlinks, shipping this type of information declaratively would also be conceptually wrong. In principle, the package could also ship the symlinks. With current dpkg, the file conflict causes the symlinks to be ignored, and that makes sense for safety purposes (imagine a broken symlink in a package moving all installed files), so a mechanism that sa
another usrmerge branch
Hi, I've also started work on getting usrmerge back into a sensible state, current progress is at https://salsa.debian.org/sjr/dpkg/-/tree/wip-canonical-paths All of these commits obviously need to be rewritten a few times, but the general idea should be clear: - all paths that can be aliased should be added to the in-memory database under each alias name - the alias name is set up by a special control file in a package - file conflicts are to be resolved normally I'd like to avoid a separate tool like dpkg-divert, because we need to be able to set up aliases without being able to run maintainer scripts, so these are useful from (c)debootstrap when setting up a foreign architecture. Ideally we'd be able to get rid of any special usrmerge handling in other tools that way as well, and gain some infrastructure that will also be useful for declarative diversions. The obvious interface breakage would be dpkg-query, which would probably behave like $ dpkg-query -S /usr/bin/ls coreutils: /bin/ls but we have all the relevant file names handy at this point so we can have a different format as well. Transition would be a dpkg version that conflicts with usrmerge, and a systemd package that declares the alias and pre-depends on a recent enough version of dpkg, that will work for both upgrades and new installations. If dpkg external to the new installation is used, the files will be moved during unpacking when the systemd package is encountered, if the files are first manually unpacked, and then overwritten later during a second round, that should still work (but we need to be aware of this case). The most annoying part will be identifying all the corner cases, like crossgrades from ppc32 to ppc64, or from Debian to Devuan and back, and writing test cases for those. Simon OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Bug#951479: [files list file for package 'base-passwd' is missing final newline] [debian buster 10.3]
Hi, On Mon, Feb 17, 2020 at 04:49:44PM +0100, Jean-Marc LACROIX wrote: > >Could you attach the old base-passwd.list file? > yes, in attached file. That file has a sensible size, but consists of only zeroes. This typically happens with file systems that have metadata only journaling after a power loss, as the metadata update is covered by the journal and can be replayed, while the data write itself is lost. That can happen e.g. with ext4 in "data=writeback" mode, which is discouraged precisely for that reason. Simon
Bug#951479: [files list file for package 'base-passwd' is missing final newline] [debian buster 10.3]
Hi, On Mon, Feb 17, 2020 at 10:31:25AM +0100, Jean-Marc LACROIX wrote: > dpkg: unrecoverable fatal error, aborting: > files list file for package 'base-passwd' is missing final newline > E: Sub-process /usr/bin/dpkg returned an error code (2) That file is generated by dpkg on installation, so it's not the package's fault. The "final newline" check is a safety check to see that the file name is complete, guarding against write errors. You need to reinstall the base-passwd package, possibly after deleting /var/lib/dpkg/info/base-passwd.list, and I'd also run a file system check for good measure, because that state should only happen on write errors (although I remember it also sometimes happened when /var was full, but IIRC that got fixed). Simon
Bug#846405: dpkg-shlibdeps: scales badly for many binaries
Package: dpkg Version: 1.18.15 Severity: minor Hi, when building the piglit package, the dpkg-shlibdeps invocations take upwards of 30 minutes (on an i7, building inside a tmpfs). Most of the time seems to be spent spawning several thousand instances of dpkg-query. Is there a way to speed this up, e.g. with a cache? Simon -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (990, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386, armhf Kernel: Linux 4.8.0-1-amd64 (SMP w/8 CPU cores) Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages dpkg depends on: ii libbz2-1.0 1.0.6-8 ii libc62.24-5 ii liblzma5 5.2.2-1.2 ii libselinux1 2.6-3 ii tar 1.29b-1.1 ii zlib1g 1:1.2.8.dfsg-2+b3 dpkg recommends no packages. Versions of packages dpkg suggests: ii apt 1.3.1 -- no debconf information
Bug#840308: dpkg: extraction of additional orig archives into nested subdirectory
Package: dpkg Version: 1.18.10 Severity: wishlist Hi, I have several packages that use additional .orig archives for program modules, but these are almost always expected in a nested subdirectory for the package build, e.g. "modules/foo". It would be nice if there was a way to ship this as a separate archive instead of repacking the source. Simon -- System Information: Debian Release: stretch/sid APT prefers testing APT policy: (990, 'testing') Architecture: amd64 (x86_64) Foreign Architectures: i386, armhf Kernel: Linux 4.6.0-1-amd64 (SMP w/8 CPU cores) Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages dpkg depends on: ii libbz2-1.0 1.0.6-8 ii libc62.23-5 ii liblzma5 5.1.1alpha+20120614-2.1 ii libselinux1 2.5-3 ii tar 1.29b-1 ii zlib1g 1:1.2.8.dfsg-2+b1 dpkg recommends no packages. Versions of packages dpkg suggests: ii apt 1.3~rc4 -- no debconf information
*** SPAM LEVEL 4.823 *** dpkg-shlibdeps: Semi-private symbols
Hi, In #712903, a symbol not intended as public is used by a dependent library. The symbol appears to be exported for the benefit of other libraries built from the same source, so it cannot be easily hidden, but I wonder whether it would make sense to have a mechanism that allows dpkg-shlibdeps to fail if a symbol is referenced from a package outside a certain set. Opinions? Simon signature.asc Description: OpenPGP digital signature
Bug#591522: dpkg-shlibdeps still fails when cross-compiling
Hi, On Tue, Aug 03, 2010 at 04:01:48PM -0400, Loïc Minier wrote: I wonder whether it makes sense to look into /usr/lib at all when cross-building? Not really, I think. If that's not an option, I think the routine checking the format of executable could run the cross-objdump and if it fails try the host objdump as a fallback. I have an use case here where the native objdump will fail on the target binary, so if there is a fallback, it needs to be per-file. I also wonder whether objdump fails to read it might already be a strong hint that the file is the wrong version. If another file exists that can be read using the targetted objdump, and has the correct architecture, then I'd expect the build to continue, and ideally, to not complain too loudly. Simon -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#591522: dpkg-shlibdeps still fails when cross-compiling
Package: dpkg Version: 1.15.8.2 Severity: normal Hi, while using the target objdump is a Good Thing(tm), it uncovered another bug: for dependent libraries, the host's version is passed to the target objdump. While this looks like a regression as builds that used to work now fail, it isn't, because the builds were only correct by chance before. Symptom is that during a cross build of a library that links against libutil, the following is output: m68k-linux-gnu-objdump: /lib/libutil.so.1: File format not recognized dpkg-shlibdeps: error: m68k-linux-gnu-objdump gave error exit status 1 The correct file would be /usr/m68k-linux-gnu/lib/libutil.so.1 . The path can be retrieved from the target gcc using the -print-search-dirs option. Simon -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable'), (500, 'stable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.32-5-amd64 (SMP w/2 CPU cores) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages dpkg depends on: ii coreutils 8.5-1 GNU core utilities ii libbz2-1.01.0.5-4high-quality block-sorting file co ii libc6 2.11.2-2 Embedded GNU C Library: Shared lib ii libselinux1 2.0.96-1 SELinux runtime shared libraries ii xz-utils 4.999.9beta+20100713-1 XZ-format compression utilities ii zlib1g1:1.2.3.4.dfsg-3 compression library - runtime dpkg recommends no packages. Versions of packages dpkg suggests: ii apt 0.7.25.3 Advanced front-end for dpkg -- no debconf information -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: (not) simplifying dpkg-shlibdeps with readelf
Hi, On Thu, Apr 29, 2010 at 11:24:35AM +0200, Hector Oron wrote: I see no reason why embedded platforms can't use ELF. ELF is very common in the embedded world even entirely apart from Linux. There is some effort to move from bFLT to something called FDPIC ELF, which is basically ELF with runtime relocation of binaries and libraries, and an ABI extension that allows per-process instantiation of writeable segments. The difficulty there is that this format is severely underdocumented, and no C library supports it out of the box in their dynamic linker. I think in the long run everything will be ELF. Sometimes ELF is not best for XIP (eXecute In Place) and other formats do better. I wonder if Debian will be able to cope with those sometime. XIP is a subset of the per-process instantiation problem. What about mingw32 targets? (I have not worked with) Some people build Debian packages for such target. The current objdump based implementation is already dependent on ELF, as only ELF uses the string NEEDED. Simon -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100429123803.gb7...@honey.hogyros.de
Bug#558095: dpkg: please accept :native multiarch qualifier in build dependencies
Package: dpkg Version: 1.15.5.2 Severity: wishlist User: debian-embed...@lists.debian.org Usertags: multiarch-cross Hi, We would like to allow people to prepare their packages for cross-compiling. Most functionality is already in place, however we are still lacking a way to distinguish between build dependencies that are required as tools (e.g. debhelper) or for building a tool (e.g. zlib-dev for building a jar archiver which is then used to build a JAR file), and build dependencies required for linking (e.g. zlib-dev for building a java interpreter which can then unpack the JAR file). The multiarch specification solves this to some extent, as tools are marked Multi-Arch: foreign and libraries Multi-Arch: same, which allows us to have reasonable defaults by requiring foreign packages to be installed in working order (i.e. preferentially DEB_BUILD_ARCH, but any working one will do) and same packages in the version required for linking (i.e. DEB_HOST_ARCH). The cases not handled by the multi-arch specification are: - A library package (M-A: same) is to be installed for DEB_BUILD_ARCH. - A hybrid package (M-A: allowed) is to be installed for DEB_BUILD_ARCH. - A hybrid package (M-A: allowed) is to be installed for any architecture. The last case is somewhat implied in the original specification (:any qualifier in Depends lines), but never explicitly allowed for Build-Depends; the others require introducing a new qualifier that is only meaningful in build dependencies. In phase one, which needs to happen before the squeeze release for this to be useful, it would therefore be good if :any and :native qualifiers in build dependencies were accepted. As Debian itself does not yet cross-compile packages, actually supporting the specified resolution is optional (but would be nice to have in dpkg-checkbuilddeps). A minimal implementation would accept those qualifiers in all tools that handle Build-Depends fields, but interpret the field as if the qualifiers were not present (thus falling back to the strictest possible interpretation for a native build). A full implementation would resolve the dependencies according to the table on the specification page[1]. Simon [1] https://wiki.ubuntu.com/MultiarchCross -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#539038: please add support for ad-hoc architecture definitions
Package: dpkg Severity: normal Hi, any news on this? Simon -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) Kernel: Linux 2.6.30-2-amd64 (SMP w/2 CPU cores) Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Versions of packages dpkg depends on: ii coreutils 7.5-6 GNU core utilities ii libc6 2.9-27 GNU C Library: Shared libraries ii lzma 4.43-14Compression method of 7z format in dpkg recommends no packages. Versions of packages dpkg suggests: ii apt 0.7.24 Advanced front-end for dpkg -- no debconf information -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#533916: C++ symbol mangling difference between arches
Hi, On Sat, Jun 27, 2009 at 09:23:29AM +0300, Modestas Vainius wrote: While it is a good idea worth consideration but I think demangled symbol names are somewhat too ambiguous to be used in general. See below: [Examples] Not a problem IMO -- we need a new package name anyway if gcc's ABI changes, and the double/triple symbols are there precisely because the ABI says so. Except for a vtable/typeinfo/name group (which reeks of actual doubled code) all lines in your output are constructors, destructors and thunks to destructors. Simon -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#534831: wrong quotation marks in German translation
Package: dpkg Version: 1.15.2 Severity: minor Tags: l10n Hi, dpkg occasionally prints the warning dpkg: Warnung: veraltete Option »--print-installation-architecture, bitte verwenden Sie »--print-architecture« stattdessen.« The correct quotation marks for German are „quote“ or «quote», the latter being rarely used in print. Apart from that, quotes in the above sentence are mismatched. Simon -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: powerpc (ppc) Kernel: Linux 2.6.26-1-powerpc Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages dpkg depends on: ii coreutils 7.4-2 The GNU core utilities ii libc6 2.9-18 GNU C Library: Shared libraries ii lzma 4.43-14Compression method of 7z format in dpkg recommends no packages. Versions of packages dpkg suggests: ii apt 0.7.21 Advanced front-end for dpkg -- no debconf information -- To UNSUBSCRIBE, email to debian-dpkg-bugs-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: [RFC] [Cross Toolchain] Multiarch and sysrooted toolchain
Hi, On Thu, Mar 12, 2009 at 07:54:49PM +0100, Hector Oron wrote: install anything but libraries and headers into /usr/triplet -- so I don't think there is a pressing need to replicate a filesystem hierarchy standard below a triplet directory. That is what GNU people expect when building cross tools, they use a switch called sysroot for it and it is the recommended way to build such cross tools. As far as I've understood, --with-sysroot gives the path to the target OS installation so the build can run fixincludes and add it to the search paths; since it should work with a largely unmodified snapshot of the target OS, they are pretty tolerant with the directory layout there. I don't believe that is meant as an endorsement of a particular layout, and it should work fine with --with-sysroot=/usr/triplet even with the current layout. Simon -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: [RFC] [Cross Toolchain] Multiarch and sysrooted toolchain
Hi, , since they have entirely different objectives Not entirely different - the objective for the packaging tools is actually the same, to have packages install cleanly without changes on systems with a different architecture triplet. I'm not sure this can be achieved at all, as we still need a root filesystem that isn't prefixed with the architecture triplet. The other issue is that generally, the 32/64 bit distinction does not necessarily mean that we use a different triplet. i386/amd64 does, at least in Debian, but that is optional as well and could be changed in the gcc package, which would give us a situation where only clearly incompatible arches need to be installed into a triplet directory. , and there is generally no need to install anything but libraries and headers into /usr/triplet -- so I don't think there is a pressing need to replicate a filesystem hierarchy standard below a triplet directory. True, however, that is not a sufficient reason to not move /usr/triplet to /usr/lib/triplet and /usr/include/triplet if it means getting such support into the core Debian packaging tools. Indeed, however this makes building stuff without the packaging tools a lot harder -- for example I need to patch gcc to recognize these paths if I build gcc by hand. It might be a lot easier to have the packaging tools handle the current layout than to patch all the software that assumes that software is generally installed into include and lib dirs under a common prefix (such as most GNU software that uses the autoconf macros to find installed software). Simon -- To UNSUBSCRIBE, email to debian-dpkg-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#457371: dpkg: please add the DEB_VARIANT build environment variable
Package: dpkg
Version: 1.14.12
Severity: wishlist
Tags: patch
Hi,
the attached patch adds a new variable DEB_VARIANT to the build
environment, which can be used to build different binary packages out of
the same source package in different circumstances.
The most obvious application would be derived distributions; there is
currently no reliable way to communicate cross-derivatives building of a
package that uses the same source tree in multiple distributions but
should provide different binaries on each; the most notorious example is
gcc which uses different compiler defaults on Ubuntu, and detects
whether the build system is a Debian or Ubuntu system and configures the
target compiler appropriately (so it is difficult to build a compiler
that will create binaries targeted at a Debian system on an Ubuntu
system, as the resulting compiler will assume that the target's libc
contains SSP support).
A different application will be embedded systems, where a variant string
could be used to build a finer grained package split or leave out large
optional bits of the package.
The system is self-perpetuating; as soon as a deriver built dpkg with
this patch once, telling it which DEB_VARIANT to use, this information
will be passed on to all packages built from there on, including newer
versions of dpkg. This will allow for branding changes to be merged back
into Debian, and thus reduce merging work.
Simon
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.22-3-amd64 (SMP w/1 CPU core)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages dpkg depends on:
ii coreutils 5.97-5.6 The GNU core utilities
ii libc6 2.7-5 GNU C Library: Shared libraries
dpkg recommends no packages.
-- no debconf information
diff -Nru /tmp/CTGRa1SVPh/dpkg-1.14.12/configure.ac
/tmp/Zg4BEoCa5A/dpkg-1.14.12/configure.ac
--- /tmp/CTGRa1SVPh/dpkg-1.14.12/configure.ac 2007-11-29 05:58:22.0
+0100
+++ /tmp/Zg4BEoCa5A/dpkg-1.14.12/configure.ac 2007-12-21 23:46:29.0
+0100
@@ -49,6 +49,15 @@
esac])
AC_SUBST(admindir)
+AC_ARG_WITH(variant,
+ AS_HELP_STRING([--with-variant=VARIANT],
+ [Debian system variant [[debian]]]),
+[case $with_variant in
+ ) AC_MSG_ERROR([invalid variant specified]) ;;
+ *) variant=$with_variant ;;
+esac],
+[variant=debian])
+AC_SUBST(variant)
# Checks for programs.
AC_PROG_CC
diff -Nru /tmp/CTGRa1SVPh/dpkg-1.14.12/debian/rules
/tmp/Zg4BEoCa5A/dpkg-1.14.12/debian/rules
--- /tmp/CTGRa1SVPh/dpkg-1.14.12/debian/rules 2007-10-18 00:30:14.0
+0200
+++ /tmp/Zg4BEoCa5A/dpkg-1.14.12/debian/rules 2007-12-22 00:04:27.0
+0100
@@ -32,6 +32,9 @@
confflags += --with-selinux=static
endif
+DEB_VARIANT ?= debian
+
+confflags += --with-variant=$(DEB_VARIANT)
# Create configure script if necessary, automake handles rebuilding it.
configure:
diff -Nru /tmp/CTGRa1SVPh/dpkg-1.14.12/scripts/dpkg-buildpackage.pl
/tmp/Zg4BEoCa5A/dpkg-1.14.12/scripts/dpkg-buildpackage.pl
--- /tmp/CTGRa1SVPh/dpkg-1.14.12/scripts/dpkg-buildpackage.pl 2007-10-18
00:41:02.0 +0200
+++ /tmp/Zg4BEoCa5A/dpkg-1.14.12/scripts/dpkg-buildpackage.pl 2007-12-22
00:09:07.0 +0100
@@ -48,6 +48,7 @@
-usunsigned source.
-ucunsigned changes.
-aarch Debian architecture we build for (implies -d).
+ -VvariantDistribution variant we build for
-b binary-only, do not build source. } also passed to
-B binary-only, no arch-indep files. } dpkg-genchanges
-S source only, no binary files. }
@@ -106,6 +107,11 @@
my $diffignore = '';
my $binarytarget = 'binary';
my $targetarch = my $targetgnusystem = '';
+my $variant = debian;
+
+if (defined($ENV{'DEB_VARIANT'})) {
+ $variant = $ENV{'DEB_VARIANT'};
+}
while (@ARGV) {
$_ = shift @ARGV;
@@ -139,6 +145,8 @@
} elsif (/^-a(.*)$/) {
$targetarch = $1;
$checkbuilddep = 0;
+} elsif (/^-V(.*)$/) {
+$variant = $1;
} elsif (/^-s[iad]$/) {
$sourcestyle = $_;
} elsif (/^-s[nsAkurKUR]$/) {
@@ -299,6 +307,8 @@
}
close $arch_env or subprocerr('dpkg-architecture');
+$ENV{'DEB_VARIANT'} = $variant;
+
my $arch;
unless ($sourceonly) {
$arch = mustsetvar($ENV{'DEB_HOST_ARCH'}, _g('host architecture'));
diff -Nru /tmp/CTGRa1SVPh/dpkg-1.14.12/scripts/Makefile.am
/tmp/Zg4BEoCa5A/dpkg-1.14.12/scripts/Makefile.am
--- /tmp/CTGRa1SVPh/dpkg-1.14.12/scripts/Makefile.am2007-11-24
06:14:44.0 +0100
+++ /tmp/Zg4BEoCa5A/dpkg-1.14.12/scripts/Makefile.am2007-12-21
23:35:20.0 +0100
@@ -90,8 +90,10 @@
dist_pkglib_SCRIPTS = \
controllib.pl
+variant = @variant@
do_perl_subst = sed -e
Bug#398625: adapted patch against current dpkg
Package: dpkg
Version: 1.14.7
Followup-For: Bug #398625
Hi,
I have written a new implementation of the patch proposed earlier,
against the current shell script.
This patch tries to address all the concerns raised so far, however it
introduces some ugliness in doing so: behaviour depends on the system
clock.
Basically, what this patch does: if -B was used, it will attempt to use
the build-arch target. if that fails, and either -nc was set or it
took longer than 300 seconds to fail, the build is aborted there,
otherwise, the tree is cleaned and the build retried with the build
target.
Rationale: if it takes longer than 300 seconds for the package to find
out whether it has a build-arch target or not, the situation is FUBAR
anyway. Either the package build scripts are horribly broken, or the
machine is completely overloaded -- in neither case it makes real sense
to continue here. This does, however, solve the problem where a package
takes long to build in the build-arch target and fails later -- those
builds should not be retried -- and where a package does not have a
build-arch target -- these should. As (hopefully) many packages start
implementing these currently optional targets, the number of normal
FTBFS cases is high enough that we don't want to waste autobuilder time
on trying these packages twice.
A special string I: optional-target-failed followed by the name of the
target is output if the build is being retried -- this will allow
generating a database of packages that do not have the optional targets,
so interested individuals can contribute patches.
Simon
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: powerpc (ppc)
Kernel: Linux 2.6.20-1-vserver-powerpc (SMP w/1 CPU core)
Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Versions of packages dpkg depends on:
ii coreutils 5.97-5.4 The GNU core utilities
ii libc6 2.6.1-6GNU C Library: Shared libraries
dpkg recommends no packages.
-- no debconf information
--- scripts/dpkg-buildpackage.pl.orig 2007-10-08 06:12:43.0 +0200
+++ scripts/dpkg-buildpackage.pl2007-11-04 14:18:31.0 +0100
@@ -102,6 +102,7 @@
my $signsource = 1;
my $signchanges = 1;
my $diffignore = '';
+my $buildtarget = 'build';
my $binarytarget = 'binary';
my $targetarch = my $targetgnusystem = '';
@@ -160,6 +161,7 @@
} elsif (/^-b$/) {
$binaryonly = '-b';
@checkbuilddep_args = ();
+ $buildtarget = 'build-arch';
$binarytarget = 'binary';
if ($sourceonly) {
usageerr(sprintf(_g(cannot combine %s and %s), '-b', '-S'));
@@ -167,6 +169,7 @@
} elsif (/^-B$/) {
$binaryonly = '-B';
@checkbuilddep_args = ('-B');
+ $buildtarget = 'build-arch';
$binarytarget = 'binary-arch';
if ($sourceonly) {
usageerr(sprintf(_g(cannot combine %s and %s), '-B', '-S'));
@@ -365,7 +368,30 @@
chdir($dir) or failure(chdir $dir);
}
unless ($sourceonly) {
-withecho('debian/rules', 'build');
+# This handles the (currently) optional build targets. In case they ever
+# become mandatory, replace this entire if block with the else branch.
+# The timeout is ugly -- but it will stop packages that have the optional
+# target but fail to build from wasting more than five minutes of
+# autobuilder time -- and honestly, if a package takes five minutes to
+# find out that it actually does not have an optional target, failing to
+# build once is the least of its problems.
+if($buildtarget ne 'build') {
+my $starttime = time;
+eval { withecho('debian/rules', $buildtarget); };
+ my $stoptime = time;
+if($@) {
+ if($noclean || (($stoptime - $starttime) 300))
+{
+die $@;
+}
+print [EMAIL PROTECTED];
+print I: optional-target-failed $buildtarget\n;
+withecho($rootcommand, 'debian/rules', 'clean');
+withecho('debian/rules', 'build');
+}
+} else {
+withecho('debian/rules', $buildtarget);
+}
withecho($rootcommand, 'debian/rules', $binarytarget);
}
if ($usepause
Re: Bug#229357: Can we require build-arch/indep targets for lenny?
Hi, Frank Lichtenheld schrieb: 3) Autodetection My approach would be to have the autobuilders use build-arch, and if that fails within 60 seconds, clean and build. If build-arch is not implemented, it fails rather quickly, so we use build and make a note in the build log. Later, one can grep for that note. If it is implemented, but fails within 60 seconds, not much is lost. If it takes longer to fail, then it's a normal FTBFS. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#440094: dpkg-dev: allow specifying the source version explicitly
Hi, Guillem Jover wrote: What you describe is a binNMU but with a different suffix. Maybe we could come up with a generic syntax for binNMU-style versions, which could include backports and the like. That would be difficult. For example, for my cross-toolchain backports I use ~debian4.0+b1 as a suffix, with debian being replaced by ubuntu for Ubuntu builds etc. In principle the only way to be sure would be to add another special character (# springs to mind), but that would be a pretty invasive thing. Although I'm not sure this is worth it, in most cases, you'll need to modify the source anyway, and in the others having an additional diff.gz is not that much (that does not take into account native packages, but there should not be that many of those anyway). Well, all current toolchain packages auto-backport without any source changes; I'd expect a lot of packages already work fine. [patch adding DEB_SOURCE_VERSION] I don't think this is a good idea, it makes automatically building those packages rather cumbersome, and difficult to reproduce. For normal binNMUs, nothing changes, as the current behaviour is still the default. When (automatically) building a backport with a nonstandard version number, all that has to be done is save the version number before the addition of the extra changelog entry, and pass that as the contents of DEB_SOURCE_VERSION during the build process. Omitting this step leaves us at the status quo, so there is no regression in the patch either. Also the patch is wrong, Source-Version != source:Version, otherwise we'd not have introduced the new variables. Well, at this specific point, they are initialized to the same value, so I'd expect them to be much the same at least at this point in time. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#229357: Can we require build-arch/indep targets for lenny?
Hi, Andreas Metzler wrote: --- Somehow make dpkg-buildpackage -B make use of the build-arch target if present. Either by detecting it automatically or by something like #229357. --- The entire issue circles around not being able to reliably detect whether the target is present using a simple script. But who said it has to be a script? Proposed alternative solution: 1. hack GNU make to support a --has-target option that returns an appropriate return code (hey, it's free software, after all!). Proposed return codes are 0 (yes), 1 (no) and 2 (error). 2. make dpkg-buildpackage -B use this facility to determine whether the target is present. If yes, use the build-arch target to build; if no, use the build target after printing a warning. 3. grep the build logs for warnings about missing build-arch target, add an entry to the TODO list on the package overview page on qa.debian.org and to DDPO. The only remaining question is what should happen with build failures in packages that provide a non-functional build-arch target. In my opinion, this is a genuine bug, even if policy can be read in a way that makes it disagree. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Bug#229357: Can we require build-arch/indep targets for lenny?
Hello, Goswin von Brederlow wrote: The seconds part requires that tools like sbuild and pbuilder know beforehand if build or build-arch will be used. For packages that do not implement build-arch, it is acceptable to call the build target with only B-D installed, because that is the way the autobuilders handle it now. So no problem there; packages that implement build-arch can move the dependencies not needed for building arch-dependent packages from B-D to B-D-I as soon as the autobuilders start using build-arch. Getting rid of unneeded build dependencies is mostly orthogonal to the issue at hand, though. Running debian-rules can always have side effects and can actively rely on them so a --has-target can not be implemented cleanly in make. I am proposing hooking into the logic that ultimately decides that there is no such target in the Makefile and goes on to print Don't know how to make 'foo'. Stop.. This means that Makefiles are rebuilt before that test is performed, we stop immediately before the point where we would go towards the first goal target. Yes, that means running commands that possibly have side effects. But we are going to run these commands anyway. Simon -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
Re: Secret changes for binNMUs
Hi, Henrique de Moraes Holschuh schrieb: We really need another substvar with different semantics. http://lists.debian.org/debian-devel/2002/09/msg01251.html Simon signature.asc Description: OpenPGP digital signature
Bug#35691: dpkg: New package dependency: affects
Package: dpkg Version: 1.4.0.34 Severity: wishlist I'd like to propose a new dependancy type which could possibly be described best with the keyword affects. The problem behind this is that with slink, many packages that register new mimetypes do not depend on mime-support, and thus get configured before mime-support. Since the /etc/mailcap file is not present at configuration time, they assume that mime-support is not installed, and skip this step. Thus, about 50% of the available mimetypes are not registered. I would propose that these packages could affect mime-support. As soon as something about the installation status of mime-support changed, the postinst script could be called with appropriate arguments, and the script would make sure the package is still registered properly. This would also allow to rebuild the database if it got lost, by simply re-installing mime-types. BTW, the same goes for the documentation. Until doc-base (I believe) is installed, install-docs simply ignores every request to register documentation. Simon -- System Information Debian Release: 2.1 Kernel Version: Linux phobos 2.0.36 #1 Sun Mar 21 15:07:33 CET 1999 i486 unknown Versions of the packages dpkg depends on: ii libc6 2.0.7.19981211 GNU C Library: shared libraries ii libncurses4 4.2-3 Shared libraries for terminal handling ii libstdc++2.92.91.60-5 The GNU stdc++ library (egcs version)
Bug#35691: dpkg: New package dependency: affects
On Wed, 7 Apr 1999, Santiago Vila wrote: Yet another thing could be done: rewrite the install-mime mechanism and convert it into something like the menu system. The menu system does not have this problem, because it collects the menu entries of each package on the background. The menu system simply builds its database out of a bunch of files in /usr/lib/menu. This is indeed an alternative, although it would require that the package that keeps the database provides such the necessary scripts, which means predicting patches etc. (Another interesting application: Make sure that a patch is installed). Also, it requires the user to keep databases he did not request to install (The menu files are there, whether the menu package is installed or not). This is only a small problem, because the postinst script would have to be there, too, but it is easier to mount /var/lib/dpkg over the net if it doesn't fit than all the database directories. The menufile approach is certainly a thing I could live with, although it is not as flexible as the affects keyword. Simon
Bug#33991: Automatic library install/deinstall
Stephane Bortzmeyer wrote: My suggestion on how to achieve this would be a new installation request: Install if other packages depend on it, remove otherwise. This could be the default value for just about any package starting with lib*, because they seldom add functionality to the system by themselves. Of course, the user should be able to explicitly request installation. Very bad idea (making it the default value). What about libraries which are used only by /usr/local/* stuff, thus not visible for dpkg? For everything that is not handled by dpkg, it is still up to the administrator which packages he/she/it needs to install. It would be a first step, and it would speed up binstats checks (you only need to check the libraries that are installed although no other packages depend on them and libraries that are not contained in a package [cruft could find them] against all executables not found in any package. That is, instead of testing 200 libraries against 50,000 executables, we're now testing 10 libraries against 100 executables), so it would make running binstats every week or so a feasible option. Simon
Bug#33991: Automatic library install/deinstall
Package: dpkg Version: 1.4.0.33 Severity: wishlist When I cleaned up my disk of stale files, I found out that I had just about 100MB of unneeded libraries floating around, and thought it would be nice if dpkg could remove all the libraries that are no longer needed together with the package I just removed. My suggestion on how to achieve this would be a new installation request: Install if other packages depend on it, remove otherwise. This could be the default value for just about any package starting with lib*, because they seldom add functionality to the system by themselves. Of course, the user should be able to explicitly request installation.
