Re: ifupdown maintenance
Hi Ansgar, On 7/11/24 06:15, Ansgar wrote: It is supported *now*, but the roadmap is unclear -- that support could be discontinued at any moment, and it would not be the first time a feature Debian relied on was removed. I understand your fears about the uncertainty of future developments. No, you don't. What I'm concerned about is not packages as a whole being discontinued. This is highly unlikely for systemd, and for simpler packages, it is not a problem as long as we are able to react to security issues. My concern is *interfaces* and *features* being discontinued in new versions, requiring adaptation on our end before we can update a package that we want to upgrade. Specifically I'm looking at the interface for passing in system-wide network configuration, because that interface will likely stand in the way of future systemd-networkd development: 1. Plain text files with passwords in them are "unix-like", and systemd has a builtin credential store mechanism, systemd-creds. I expect that there will be a bit of push to move things like passwords into the credential store (which is unavailable from the d-i environment). 2. So far, systemd-networkd does not support run-time configuration from unprivileged accounts (so on laptops it makes sense to keep using N-M). I am not convinced that there are no plans to add that feature though, and at that point, the configuration and credential storage will need to be reworked. This is why I think that this interface will be considered a liability down the line by upstream developers. It's what I would do in their place. After all ifupdown is without doubt in a bit problematic state due to isc-dhcp-client no longer being supported, a feature Debian relied on, and as far as we know the support for alternatives like dhcpcd-base could end at any time as well. That is not a feature, but a package. Unless we actively remove it, it stays. The thing that is special about systemd is that they actively remove things that are in the way of future progress, such as support for unmerged /usr, and we need to deal with that whenever it happens. That is not a fault either: it costs resources to keep legacy interfaces working, and with the ambitious feature set they have, those are better spent elsewhere, but whenever we rely on a legacy interface, we need to be prepared to have it pulled out at some point, like the unit generator for init scripts. Debian already relies on a fairly large set of base software like gcc, linux, glibc, systemd, ... So it seems reasonable to try to not grow that list much further to address your concerns. Linux and glibc are going to great lengths to never break compatibility during an upgrade, these are full of compatibility code. GCC we use an older version that we do not upgrade during a release, and we get full support from upstream for that. The experience of reporting a bug against gcc 12 is vastly different from reporting a bug against systemd 252. Again, that is not a fault, but a difference in project management -- but one that we need to account for when we are doing our own planning. The more policy work we outsource to external projects that work at a faster pace than we do, the more resources we need to keep on hand to catch up. This is not a technical problem, but a project management one. The technical aspects of this are the easy part, although it is also possible to bungle them if they are left to overconfident people. Simon
Re: ifupdown maintenance
Hi, On 7/10/24 05:36, Marco d'Itri wrote: That's my question, essentially: is this an interface with full support from upstream, or something that may change in an incompatible way later that will require us to deploy additional infrastructure to support? Multiple people, one of the systemd upstream maintainers among them, already told you that creating configuration files is a normal and fully supported interface. It is supported *now*, but the roadmap is unclear -- that support could be discontinued at any moment, and it would not be the first time a feature Debian relied on was removed. The key feature of both sysvinit and ifupdown, from Debian's perspective, has always been control: with sysvinit, there were no "upstream" definitions, it was all defined by us as part of Debian Policy, and if we What you mean is that it has always been that they were basically abandoned upstream so there were no new features were coming that way and we had to implement in Debian everything that we needed. If you want to ignore my point, sure, why not. The thing is, we are a volunteer project. We cannot afford to be in perpetual firefighting mode, because none of us is paid well enough for that. We are still wiping up the technical debt of the usrmerge transition, which binds additional resources. I'm not saying this change is bad. I'm saying we need to ensure that this time we have the necessary manpower with enough motivation and available time to treat this as an ongoing commitment to actually maintain and support this after the initial implementation, and the first step is to estimate the amount of effort required, which is largely dependent on upstream commitment to *long term* interface stability. I understand that progress in Debian being slow is frustrating, but forcing progress in the form of practical necessities arising out of previous decisions and calling for volunteers whenever change must be implemented is not a sustainable project steering method. The initial implementation is easy, and I have no doubt this can be done within weeks. However unlike ifupdown, which at least continues to work if no one changes it (which is perfect for a volunteer project), this has a factor that is outside Debian's control. Basically, if a newer version of systemd-networkd breaks the interface d-i used, can we get a process in place to 1. detect that this is a breaking change 2. hold off on deploying the new version until d-i has been updated 3. find a volunteer quickly to implement the required adaptation? Of course, with a long-term support commitment from upstream we can significantly reduce the effort required for this process. Simon OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: ifupdown maintenance
Hi, On 7/9/24 23:01, Luca Boccassi wrote: As per smcv's point, if you need to manually write a static configuration then you can also install your favourite tool to use it. This is not the default case - the default case is "I have ethernet and/or wifi and I want DHCP v4+v6 on anything that can talk to a router kkthxbye" For a server installation, I absolutely need the option to configure a static IP from d-i text mode interface or a preseed file, and this configuration to be taken over into the installed system. The last three machines I've installed I've used the IPMI console to set up the IP address, then used the "continue installation via ssh" function to do the main installation. On one of these networks was a rogue DHCP server. :P I'm not sure if systemd-networkd is much happier long-term if we write its configuration from a shell script, we'd need to get some commitment from upstream that this is an interface, not an implementation detail, at least. I'm not sure what this means, you can write networkd configuration files from wherever you like, the tool doesn't care, it wouldn't even be able to tell the difference anyway, just drop them in the appropriate location in /etc/ and off you go. That's my question, essentially: is this an interface with full support from upstream, or something that may change in an incompatible way later that will require us to deploy additional infrastructure to support? The key feature of both sysvinit and ifupdown, from Debian's perspective, has always been control: with sysvinit, there were no "upstream" definitions, it was all defined by us as part of Debian Policy, and if we needed to change anything, we could do that without introducing an incompatibility; with ifupdown, we are also upstream, and therefore upstream is aware that if they want to change the configuration file format, they have to talk to all the other stakeholders inside the project and coordinate that change. With systemd, we no longer have that level of control and large parts of Policy are now defined externally, and on an external schedule that does not match our release schedule. That is not necessarily a bad thing, but it is a huge deviation from our slow but reliable processes of the past where we could delay a change until it was ready for release. So we need to weigh all the benefits of switching to systemd-networkd against the drawback that we are again tying part of our system to an externally defined release schedule, so we will need to nominate someone who will be responsible for integrating any changes that will be required as a result of upstream changes in a timely manner, and in a way that does not introduce any more technical debt. The issue is not whether we can hack together a perl script *now*, but how big of a commitment that change can potentially be. If we can get a guarantee that any changes to that interface will be communicated two years in advance, that calculation is entirely different than when it can change with any upstream release and we need to provide an upgrade path with the next backport package or risk breaking "stable" systems that for some reason have backports of core system packages installed. I fully expect some breaking changes, as we are storing the WiFi credentials into a configuration file, when they should be stored in some secrets manager -- so this is already "legacy", and I'm not sure if it is a "legacy interface" or a "legacy implementation detail." Simon
Re: ifupdown maintenance
Hi, On 7/9/24 17:57, Matthias Urlichs wrote: Agreed: either it's drop-in compatible or we may as well switch the default to NM and/or systemd-networkd. Well, here's a heretical thought: why don't we do that anyway, at least for new installations? Both are overly complex for a static-IP-only server installation, where there is no requirement for an unprivileged user to modify the network configuration, or any background process at all, really. At least in expert mode I would expect a way to generate a static, unmanaged-by-anything configuration. What would be needed for new installations is d-i support, mainly, and the difficulty there is saving the configuration. I believe NM does not have a fixed configuration format, but only a dbus API. Our best bet there would be a firstboot unit, but I have no idea whether accessing NM from a unit attached to firstboot is safe or leads to a dependency loop[1]. I'm not sure if systemd-networkd is much happier long-term if we write its configuration from a shell script, we'd need to get some commitment from upstream that this is an interface, not an implementation detail, at least. Or we accept the UX regression of having to configure the network again on the first graphical login, when per-user credential stores are available through the appropriate services. Simon [1] there should really be documentation in Policy what dependencies are allowed.
Re: Seeking consensus on file conflict between python3-proto-plus and nanopb
Hi, On 6/11/24 07:40, Yogeswaran Umasankar wrote: It appears that nanopb's use of the module name "proto" does not align with the conventional identification of a Python module. Given this, it might be plausible to make this module private within the nanopb package. This adjustment could potentially resolve the conflict without affecting the dependent packages. From a quick glance at the package, not completely. It seems the "proto" module is used by the generator only, which is called as a plugin from Google protoc, so I'm not sure it can be taken fully private. I think it should be possible to rename this module though with little adverse effect. The standard invocation certainly is convoluted[1], and given that CMake is usually not the first choice for cross compiling in embedded environments, I also expect that there will be a lot of users rolling their own -- but I don't see the name of the module used on that command line, only the name of the plugin. The script tries to do a relative import and provides fallback code if that is not yet supported in the Python interpreter (which it is since 2007), so that might explain why the module is not up to current conventions. Simon [1] https://github.com/nanopb/nanopb/blob/master/extra/FindNanopb.cmake
Re: Mandatory LC_ALL=C.UTF-8 during package building
Hi, On 6/11/24 00:26, Simon McVittie wrote: Reproducibility outside of sterile environments is however a problem for us as a distribution, because it affects how well people are able to contribute to packages they are not directly maintaining If our package-building entry point sets up aspects of its desired normalized (or "sterile") environment itself, surely it's equally easy for those contributors to build every package in that way, whether they maintain this particular package or not? Yes, but building the package is not the hard part in making a useful contribution -- anything but trivial changes will need iterative modifications and testing, and the package building entrypoint is limited to "clean and build entire package" and "build package without cleaning first", with the latter being untested and likely broken for a lot of packages -- both meson and cmake utterly dislike being asked to configure an existing build directory as if it were new. For my own packages, I roughly know how far I can deviate from the clean environment and still get meaningful test results, but for anything else, I will still need to deep-dive into the build system to get something that is capable of incremental builds. if my package is not required to work outside of a very controlled environment, that is also an impediment to co-maintenance I'm not sure that follows. If the only thing we require is that it works in a controlled environment, and the controlled environment is documented and easy to achieve, then surely every prospective co-maintainer is in an equally good position to be contributing? That seems better, to me, than having unwritten rules about what environment is "close enough" and what environment doesn't actually work. I will need to deviate from the clean environment, because the clean environment does not have vim installed. Someone else might need to deviate further and have a graphical environment and a lot of dbus services available because their preferred editor requires it. Adding a global expectation about the environment that a package build can rely on *creates* an unwritten per-package rule whether it is permissible to deviate from this expectation during development. I expect that pretty much no one uses the C.UTF-8 locale for their normal login session, so adding this as a requirement to the build environment creates a pretty onerous rule: "if you want to test your changes, you need to remember to call make/ninja with LC_ALL=C.UTF-8." Of course we know that this rule is bullshit, because the majority of packages will build and test fine without it, but this knowledge is precisely one of the "unwritten rules" that we're trying to avoid here. We already do expect maintainers to be building in a specified environment: Debian unstable (not stable, and not Ubuntu, for example). I develop mostly on Debian or Devuan stable, then do a pbuilder build right before upload to make sure it also builds in a clean unstable environment. The original requirement was mostly about uploading binary packages, which we (almost) don't do anymore. (I also do agree that it is an anti-pattern if we have a specified environment where tests or QA will be run, and serious consequences for failures in that environment, without it being reasonably straightforward for contributors to repeat the testing in a sufficiently similar controlled environment that they have a decent chance at being able to reproduce the failure. But, again, that isn't this thread.) This is largely what I think is this thread -- narrowing the environment where builds, tests and QA will be run, and narrowing what will be considered a bug. Indeed -- however this class of bugs has already been solved because reproducible-builds.org have filed bugs wherever this happened, and maintainers have added workarounds where it was impossible to fix. Someone (actually, quite a lot of someones) had to do that testing, and those fixes or workarounds. Who did it benefit, and would they have received the same benefit if we had said "building in a locale other than C.UTF-8 is unsupported", or in some cases "building in a non-UTF-8 locale is unsupported", and made it straightforward to build in the supported locales? I'd say that developers who don't have English as their first language have directly benefited from this, and would not have benefited if it was not seen as a problem if a package didn't build on their machines without the use of a controlled environment. I also think that we have indirectly benefited from better test coverage. Turning this workaround into boilerplate code was a mistake already, so the answer to the complaint about having to copy boilerplate code that should be moved into the framework is "do not copy boilerplate code." If you don't want package-specific code to be responsible for forcing a "reasonable" locale where necessary, then what layer do you want to be
Re: Mandatory LC_ALL=C.UTF-8 during package building
Hi, On 6/8/24 00:42, Simon McVittie wrote: Having an UTF-8 locale available would be a good thing, but allowing packages to rely on the active locale to be UTF-8 based reduces our testing scope. I'm not sure I follow. Are you suggesting that we should build each package *n* times (in a UTF-8 locale, in a legacy locale, in locales known to have unique quirks like Turkish and Japanese, ...), just for its side-effect of *potentially* passing through those locales to the upstream test suite? To an extent, that is what reproducible-builds.org testing does. According to [1], they test LC_ALL unset vs LC_ALL set to one of et_EE.UTF-8, de_CH.UTF-8, nl_BE.UTF-8, or it_CH.UTF-8. They also test other locale variables. What I'm concerned about is not so much tests inside packages behaving differently depending on locale, because that is an upstream problem. Reproducibility outside of sterile environments is however a problem for us as a distribution, because it affects how well people are able to contribute to packages they are not directly maintaining -- so for me this hooks into the salsa discussion as well: if my package is not required to work outside of a very controlled environment, that is also an impediment to co-maintenance. What we say is a bug, and what we say is not a bug, is a policy decision about our scope: we support some things and we do not support others. Exactly, and a lot of the debates we've had in the past years is who gets to decide what is in scope, and what is "legacy" code that should be excised to reduce the workload of the people driving change forward. What Giole proposed at the beginning of this thread can be rephrased as declaring that "FTBFS when locale is not C.UTF-8" and "non-reproducible when locale is varied" are non-bugs, and therefore they are not only wontfix, but they should be closed altogether as being out-of-scope. Indeed -- however this class of bugs has already been solved because reproducible-builds.org have filed bugs wherever this happened, and maintainers have added workarounds where it was impossible to fix. Turning this workaround into boilerplate code was a mistake already, so the answer to the complaint about having to copy boilerplate code that should be moved into the framework is "do not copy boilerplate code." For locales and other facets of the execution environment that are similarly easy to clear/reset/sanitize/normalize, we don't necessarily need to be saying "if you do a build in this situation, you are doing it wrong", because we could equally well be saying "if you do a build in this situation, the build toolchain will automatically fix it for you" - much more friendly towards anyone who is building packages interactively, which seems to be the use-case that you're primarily interested in. No, automatic fixes are absolutely not friendly -- these *add* to my mental load because I need to be aware of them if I want to understand what is happening. This is already annoying enough for LC_ALL=C.UTF-8 inside a package, but at least that usually happens in prominent enough places that I can find it, and it is part of the package. Magic code inside the framework that performs an action automatically for me is extra debugging effort, because I need to either know the exact rules that the framework applies, or I need to debug into the framework. Simon [1] https://tests.reproducible-builds.org/debian/index_variations.html OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Mandatory LC_ALL=C.UTF-8 during package building
Hi, On 6/7/24 22:40, Alexandre Detiste wrote: Maybe a compromise would be to at least mandate some UTF-8 locale. Having an UTF-8 locale available would be a good thing, but allowing packages to rely on the active locale to be UTF-8 based reduces our testing scope. Basically, we need to define the severity of locale bugs -- if we define that package builds can expect LC_ALL=C.UTF-8, and reproducible-builds testing finds out that passing a non-UTF-8 locale in causes a deviation or a build failure, is the maintainer free to close this "wontfix", or should this still be addressed? Besides locales, there are other things that might affect outcomes, and we need to find some reasonable position between "packages should be reproducible even if built from the maintainer's user account on their personal machine" and "anything that is not a sterile systemd-nspawn container with exactly the requested Build-Depends and no Recommended packages causes undefined behaviour." Personally my preference would be as close as possible to the former, because if I ever need to work on someone else's package, the chance is high that I will need incremental builds and a graphical debugger, and both of these are a major hassle in containers. Simon OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Mandatory LC_ALL=C.UTF-8 during package building
Hi, On 6/6/24 19:56, Johannes Schauer Marin Rodrigues wrote: At the same time though, I also get annoyed of copy-pasting d/rules snippets from one of my packages to the next instead of making use of a few more defaults in our package build environment. Same here -- I just think that such a workaround should be applied only when the package fails to build reproducibly, so this is definitely something that should not be cargo-culted in. What we could also do (but that would be a bigger change) would be another flag similar to "Rules-Requires-Root" that lists aspects of the package that are known to affect reproducibility -- that would be declarative, so the reproducible-builds project can disable the test and get meaningful results for the remaining typical problems, and could be checked and handled by dpkg-buildpackage as well. Simon
Re: Mandatory LC_ALL=C.UTF-8 during package building
Hi, > Would it be possible to set in stone that packages are supposed to > always be built in an environment where LC_ALL=C.UTF-8, or, in other > words, that builders must set LC_ALL=C.UTF-8? This would be the opposite of the current rule. Setting LC_ALL=C in debian/rules is an one-liner. If your package is not reproducible without it, then your package is broken. It can go in with the workaround, but the underlying problem should be fixed at some point. The reproducible builds checker explicitly tests different locales to ensure reproducibility. Adding this requirement would require disabling this check, and thus hide an entire class of bugs from detection. Simon
Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk
Hi, On 6/3/24 21:05, Colin Watson wrote: From the d-i side we've generally preferred to have all the UI be part of the installer (especially for translations etc.). Makes sense, thanks! Simon
Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk
Hi, On 6/3/24 15:33, Philipp Kern wrote: * Package name : systemd-boot-installer Can this be merged into the normal systemd source package? I feel like from a d-i perspective that'd be highly unusual? Having the purely d-i-specific components be owned by d-i is the common setup. If it doesn't make sense to merge it (i.e. because it is a glue package between two unstable interfaces), then that is fine, but I think that the interfaces on both the d-i and the systemd side are stable enough. Simon
Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk
Hi, On 6/3/24 09:33, Luca Boccassi wrote: * Package name: systemd-boot-installer Can this be merged into the normal systemd source package? Simon
Re: MBF: drop dependencies on system-log-daemon
Hi,
On 5/27/24 22:18, Simon McVittie wrote:
So I think your syslogd-is-journald could not be a Provides on the
existing systemd-sysv package, and would have to be a separate package.
I'm not sure that the benefit is worth it (and I see that Luca is sure
that the benefit *isn't* worth it).
I agree -- that's why I suggested changing the dependency to
"systemd-sysv | system-log-daemon"
This does not require an extra package, leaves out the system-log-daemon
on most systems, still leaves the option of co-installing a flat logging
daemon parallel to journald, and the packages work out-of-the-box on
derived non-systemd distributions, so we don't waste developer time on
maintaining a fork just for this issue.
Along similar lines, I have wondered about
dbus-{system,session}-bus-is-external packages as non-default providers
of dbus-{system,session}-bus, for use in containers that arrange for a
suitable socket to be bind-mounted in from outside, but I'm unsure how
much appetite there is for a proliferation of packages whose metadata
is larger than their content for purposes like these - especially if
they could accidentally get installed (possibly on real systems rather
than in containers) by users who have not actually made the documented
arrangements, resulting in spurious bug reports and extra support burden.
Indeed, but we only have the two options "make it declarative in the
package namespace" and "make it an essential service that packages may
rely on without making a declaration."
The latter would imply that we'd need to officially state that using
Debian packages for Docker containers or inside CI runners is
unsupported, because these don't fulfill the requirements for essential
services.
Simon
Re: MBF: drop dependencies on system-log-daemon
Hi, On 5/27/24 19:59, Luca Boccassi wrote: This MBF is not about removing the virtual provides where they are defined, they can stay as-is, but downgrading/removing the hard dependencies so that we can make Debian minimal images. So the policy becomes "a logging service is present even if not explicitly declared." -- that sounds kind of like an Essential package, except that the interface is injected externally, and not visible in the package namespace. My train of thought here is that there should be a "syslogd-is-journald" package that Provides/Conflicts system-log-daemon, so I can explicitly select this during container image creation with the existing package selection mechanism -- basically if I build a container image with syslogd-is-journald, then it becomes my responsibility to fulfill that promise, just as it is my responsibility to start rsyslog from my entrypoint when I build with rsyslog. We cannot really avoid a regression for container builders here: the implicit dependency on (effectively) rsyslog will go away, and if their entrypoint starts it explicitly (because no one uses sysvinit in a container) without having it listed in the package list, that is broken anyway. From systemd's immutable system images point of view, I think at least systemd must be installed into the immutable image anyway (for the default units in /usr/lib to be defined), and systemd-sysv on top of that doesn't use much space -- hence the idea to use systemd-sysv instead of a new empty package as the alternative. Simon
Re: Bug#1071991: ITP: fastobj -- Simple single-header library OBJ loader
Hi, On 5/27/24 17:08, Federico Ceratto wrote: Description : Simple OBJ loader Can this somehow include the word "Wavefront", or a hint that this is about 3D models? OBJ is also used to refer to relocatable object files on MS-DOS. Simon
Re: MBF: drop dependencies on system-log-daemon
Hi, On 5/27/24 11:29, Luca Boccassi wrote: With the default system installation including persistent journald by default, it doesn't seem useful anymore to have such dependencies. They are leftovers from an era where not having a system logging setup that just worked by default was a thing, and fortunately we are long past that today. There is no need to advertise for the default setting. :) For the benefit of Devuan, we could change this to "systemd-sysv | system-log-daemon". The only breakage resulting from this change would be Debian based container images that pull in a (nonfunctional, because not entrypoint) systemd instead of the logging daemon they expect, but that also breaks when we demote the dependency, so that's not worse. And anyone using a different entrypoint is unsupported by Debian anyway. Simon
Re: finally end single-person maintainership
Hi, On 5/23/24 04:32, Andrey Rakhmatullin wrote: It could be argued that testing migration is a CI process. >> Its a CI process at a way too late stage. Also, uploading to test a merge request is not the right thing to do. If the archive is a VCS then uploading an untested package to experimental to run tools is pushing a commit to run CI. *shrug* Yes, but unironically: experimental is a side branch, unstable is a MR, and testing is the main branch. It is entirely valid to be dissatisfied with the turnaround time of the existing CI, but what we're seeing here is the creation of a parallel structure with as-of-yet unclear scope. Can we define the scope as "quick-turnaround unofficial validation", because that's the niche that is currently underserved? Simon
Re: finally end single-person maintainership
Hi, On 5/22/24 20:34, Bill Allombert wrote: You can run autopkgtests locally, you do not need Salsa for that. Also, Debian runs autopkgtests on all packages that provide them, and makes passing them on all supported architectures a requirement for testing migration. It could be argued that testing migration is a CI process. Simon
Re: Salsa - best thing in Debian in recent years? (Re: finally end single-person maintainership)
Hi, On 5/21/24 15:54, Andrey Rakhmatullin wrote: The Debian archive itself is a VCS, so git-maintained packaging is also a duplication, and keeping the official VCS and git synchronized is causing additional work for developers, which is why people are opposed to having it mandated. The Debian archive itself is a *bad* and *poor* VCS. It should be obvious in which areas it's poor, and it's bad e.g. because force pushes are enabled, the history is periodically truncated (though there is no easy way to get it anyway) etc. All of these things are *also* explicit features. We need a way to unpublish things, and mirrors only want to keep a shallow subset. Representing the Debian archive in git would probably look like a massive amount of submodules, because that's the only way to represent state across multiple projects without extending it -- and extending is a big no-no, because then we'd lose all the forge support. Submodules cannot represent the pristine-tar branch though. It makes sense that some people want to replace it with a good VCS, but I agree that adding a good VCS only gives us two VCSes because replacing will never happen. There are two axes here -- "good" and "fits the use case." What I'm arguing is that git does not fit our use case, despite being good, because it is designed for something else. We can make it fit our use case by wrapping it, but then we have a choice between a thin veneer that breaks easily as people use git commands inside a source tree, when they should only be using gbp commands, or a completely opaque wrapper that loses us all the git integration from third-party tools like forges. Because git treats packaging metadata as data, no correctness of metadata is enforced at this level. We could add this in the form of upload hooks, but then people would start complaining that they want to use the tools like they want. They would be wrong, but that is what happens with leaky abstractions. This is the exact opposite of the mandatory-debhelper debate: requiring declarative packaging (and therefore calcifying all custom sequences into debhelper plugins) is precisely the opposite of "using git because it is familiar" -- the benefit of using the abstraction comes from denying access to the layer below and therefore hiding its complexity. Simon
Re: finally end single-person maintainership
Hi, On 5/21/24 10:43, Luca Boccassi wrote: The ecosystem, however, does not support our workflows, and adapting it to do that is even more effort than maintaining our own tools. [...] That's a problem of our workflows, which are horrible. The solution is not to double down on them. Our workflows are largely influenced by what we do here: we maintain packages. This is different from "developing software", or "operating infrastructure." If we can improve the workflows, then I'm all for it. What does not work, however, is to replace them with workflows from a different task. All we are doing here is providing a mapping of the packaging workflow onto a git software development workflow. It still remains a packaging workflow, because the *goal* of the workflow is the completion of packaging tasks. Providing an implementation layer does not change the goal layer, and GitLab does not provide any tools on the goal layer. If we had buttons in GitLab "import new upstream release" or "show issues affecting the stable release branch", I would consider that as support for a packaging workflow. As it is now, GitLab only supports the workflows that the implementation layer below the goal layer typically uses, including workflows that break the upper layer. At the very least, GitLab integration would allow me to automate such a simple thing as changelog handling in a more comfortable way than displaying instructions how to download the conflicting branch into local git, resolve the conflicts manually, and force-push the branch to solve the problem. gbp dch --commit --release Where is that button in GitLab? That's my point: GitLab is a step *back* from the commandline tools for git integration. Simon
Re: finally end single-person maintainership
Hi, On 5/21/24 07:43, Bernd Zeimetz wrote: Also its a gitlab instance. There are all kinds of documentation, tutorials, videos and software for/about gitlab, including lots of beginner friendly options. There is a whole ecosystem around gitlab, it does not depend on a few (two?) developers. The ecosystem, however, does not support our workflows, and adapting it to do that is even more effort than maintaining our own tools. We would not even save effort in onboarding, because we'd still need to explain the Debian specific aspects, only we would now also need to explain which parts of the git workflow we can keep and which parts don't work for us. The workflows of GitHub (more deployment focused) and GitLab (more development focused) are already different enough that I have seen organizations struggle after making the wrong choice. git-buildpackage is not a good mapping of packages to git, but the best you can do without modifying git itself. Actually using it requires more than beginner-level knowledge of git and suspension of any previously held convictions that every single commit should be buildable (because gbp likes to generate ones that aren't). A better approach would not treat Debian metadata as git data. Even the most vocal advocate of switching everything to Salsa writes in his MR that the changelog should not be touched in a commit, because it creates conflicts, and instead a manual step will need to be performed later. At the very least, GitLab integration would allow me to automate such a simple thing as changelog handling in a more comfortable way than displaying instructions how to download the conflicting branch into local git, resolve the conflicts manually, and force-push the branch to solve the problem. Simon
Re: Salsa - best thing in Debian in recent years? (Re: finally end single-person maintainership)
Hi, On 5/20/24 04:32, Otto Kekäläinen wrote: I agree that duplication is bad - but I disagree that use of version control duplicates the use of the Debian archive for source code storage, or that use of GitLab for code reviews would duplicate Debbugs. Outside of DM uploads, I'm not sure that there is much of a need for a code review on packaging -- really what we want is to reduce the amount of code for packaging, by moving it into declarative frameworks whenever possible, so the vast majority of packaging changes should be trivial ones like upgrading a dependency version. Would you be kind and try to understand the opposing viewpoint by trying it for one day? I am using it for most of my packages. It has not made my life easier, it's just another layer that I need to communicate my intentions through. I generally do test builds of my packages in a local pbuilder instance, with a hook script to drop me into a shell if the build fails, so the workspace is kept for my investigation. The only CI system that offers a similar feature is Jenkins, but even there I can only inspect the files through a clunky web interface, as soon as I need to look at a binary file or search for a string, I need to download it as a zipfile, and re-running commands inside the same environment to test them is completely out. You could go to https://salsa.debian.org/debbugs-team/debbugs/-/merge_requests/19 and conduct a code review? At first glance, looks good to me. Looking at the changes: 1. The outdated build dependency is not in the package currently in Debian. If it was, it would have been spotted by Debian's archive analysis tools already, without the need for a build attempt. This static analysis is cheaper than a rebuild, so to achieve the same level of coverage, Salsa would need to perform a full archive rebuild daily, and it would still not catch the broken Suggests: in the binary. 2. The missing file in debian/docs was already reported as #903413. 3. The other changes are "upstream" changes, which should have a separate CI that is more extensive than "still builds." Native packages should only be used for things where it does not make sense to maintain a separate upstream package because they only exist within the package ecosystem, like the "menu" package. Debbugs should really be split into separate "upstream" and "packaging" efforts. You might discover that GitLab is useful and is not duplicating Debbugs or anything else in Debian Well, there is an issue tracker (where tickets go unresponded for a year), that is certainly a duplication of debbugs. It would make sense to maybe track "upstream" bugs there and forward them from debbugs (a feature not present in GitLab's issues handling, but important for package maintenance). - it is currently the only platform to conduct code reviews on in a way that has automatic testing and comment-response-resolved -tracking. Doing code reviews patches attached in email does not have that. Well, I take the diff, prepend each line with > and insert my comments, then send it back. The author then responds to that email, and once the discussion is over, I get a new proposed patch. Not much difference. If you try it out, and still think Salsa is bad for Debian, then I am more willing to accept your stanze. It's not *bad*, but for a lot of our workflows, it's the wrong tool because the use cases it was designed for are different from ours, and there is little we can do to make them meet. Debian's workflow for collaboration on things that are not yet release-ready is clunky to the point that almost no one uses it that way, but in principle it is there: one can always upload a package to experimental and get it autobuilt on the major architectures, and other DDs can download it from there if they want to take a look at it. This workflow is what packaging in git mostly replaces: in pkg-electronics, we quite often have changes that are not ready for release that we want to distribute to the other team members. Quite often, these changes do not build straight away, and the reason they are shared is specifically so other people can take a look at them. Git is a lot better for fostering this collaboration than uploads to experimental, because we get change tracking for individual files, which is invaluable when dealing with a behemoth like ghdl that takes a few hours to build and run tests. The review process still takes place via mail here, because part of the process is that everyone involved needs to be able to build the package locally and investigate failures. We can quickly incorporate changes from others using git and do a minimal rebuild locally, that is useful, but this essentially means that we are pushing commits to an offside branch. Attaching the discussion to individual changes is not that useful for us: 1. changing an annotated line in a commit hides the annotation when looking at
Re: Salsa - best thing in Debian in recent years? (Re: finally end single-person maintainership)
Hi, On 5/19/24 16:11, Jonas Smedegaard wrote: My concern about Gitlab is not its *additions* to existing services, but its *duplications* of core services already in Debian. I agree, that's the key problem. The Debian archive itself is a VCS, so git-maintained packaging is also a duplication, and keeping the official VCS and git synchronized is causing additional work for developers, which is why people are opposed to having it mandated. The Debian archive is purpose-built for packaging, while git is general-purpose, and does not exactly fit our existing workflows. The main thing it adds is collaboration without releases. It is quite obvious we want something like that, so we need to go back to the design of the Debian system and look if we can add flows to facilitate that. The current debate feels a lot like "we need collaboration without releases as a feature, git provides that, let's use git", but we have more requirements than that, and my feeling is that these are falling by the wayside. ...instead of lumping all those discussions into a discussion of ease of user interface for a single catch-all code forge that maybe make all those other complications go away by affecting all those questions and that way implicitly provides *some* answer to them all. Also, there is a difference between ease of use and intuitivity. GitLab does not provide any tools that really make packaging easier. It is initially more accessible to non-maintainers, because of familiarity, but actual work happens outside of it. Simon OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: systemd-dev package in bookworm?
Hi, On 5/15/24 10:31, Sirius wrote: Where is the systemd-dev package for regular Bookworm? The only package that show up is systemd-dev/stable-backports 254.5-1~bpo12+3 all and if I try and install that, it seems like it wants to uninstall most of my system in the process. The version shipped in bookworm did not yet build a separate -dev package. This package was introduced in systemd (253-2) experimental; urgency=medium * Add systemd-dev package for pkg-config files. Move systemd.pc and udev.pc to systemd-dev, so that packages can build-depend on a small package instead of the whole set of binaries. (Closes: #945922, #917549) -- Luca Boccassi Mon, 12 Jun 2023 00:22:52 +0100 For older versions, you can find systemd.pc and udev.pc in the main systemd package; the other files like interface definitions are not shipped at all in the packages in bookworm. If you just need the .pc files, just add the old systemd package as an alternative to the build dependencies. Build-Depends: systemd-dev | systemd (<< 253-2) Simon OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature.asc Description: OpenPGP digital signature
Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]
Hi, On 5/8/24 07:05, Russ Allbery wrote: It sounds like that is what kicked off this discussion, but moving /tmp to tmpfs also usually makes programs that use /tmp run faster. I believe that was the original motivation for tmpfs back in the day. IIRC it started out as an implementation of POSIX SHM, and was later generalized. The connection to SHM is still there -- POSIX SHM only works if a tmpfs is mounted anywhere in the system. Canonically, a tmpfs is mounted on /dev/shm for that purpose, but if /tmp is a tmpfs, then /dev/shm doesn't need to exist. I agree that it makes a lot of things run faster (especially compiling, which creates temporary files in /tmp), but it has also caused situations that required pressing SysRq to resolve (also during compiling). For /var/tmp, I think the primary motivation to garbage-collect those files is that filling up /var/tmp is often quite bad for the system. It's frequently not on its own partition, but is shared with at least /var, and filling up /var can be very bad. It can result in bounced mail, unstable services, and other serious problems. When /var runs full, the problem is probably initrd building. Taking a quick look around all my machines, the accumulated cruft in /var/tmp is on the order of kilobytes -- mostly reportbug files, and a few from audacity -- and these machines have not been reinstalled in the last ten years. Simon
Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]
Hi, On 5/6/24 19:57, Michael Biebl wrote: Afaik, /var/tmp has never been cleaned up on /boot. So I'm not sure what you mean with "no longer"? Oof, you're right, it was /tmp, /var/run, /var/lock: [ "$VERBOSE" != no ] && echo -n "Cleaning" [ -d /tmp ] && cleantmp [ -d /var/run ] && cleanrun [ -d /var/lock ] && cleanlock [ "$VERBOSE" != no ] && echo "." Would it make sense to make it a bug for a package to use /var/tmp (on my system, I can see files from audacity and reportbug there) and declare that this directory is for the use of the sysadmin only? Simon
Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]
Hi, On 5/6/24 20:19, Luca Boccassi wrote: Is that the default layout, or a selectable option? When you create a partition manually, it asks for the mount point, and makes a number of suggestions in a dropdown, and /tmp is one of these. There is also a "enter manually" option. If the latter, to me it makes sense to keep it, maybe with updated wording. Making things customizable is not really an issue, as long as it's clear what happens where. The only wording there is "/tmp - Temporary files" or something like that (has been a while since I saw that particular menu) -- it's an option in a menu, so space is limited. FWIW, I don't see much of a use case for leaving /tmp on the root partition, so if no /tmp mount point is configured, mounting a tmpfs is a good choice. Basically anything that can boot the installer has enough RAM. Simon
Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]
Hi, On 5/6/24 17:40, Michael Biebl wrote: If we go with a/, then I think d-i should be updated to no longer create /tmp as a separate partition. I think if the admin explicitly configures tmpfs as a separate file system, then that should be honored -- if there is memory pressure, tmpfs+swap is pretty much the worst file system there is. For example, I have a VM with 32 threads but only 16 GiB of memory allocated to it, which is normally sufficient for compiling large projects, but occasionally starts thrashing really hard, and /tmp on ext4 with very relaxed ordering rules is a good way to make it behave. (What we're really want is a file system that behaves like FILE_ATTRIBUTE_TEMPORARY on Windows, where backing store is allocated, and pages are evicted first on memory pressure, but are not flushed automatically) Regarding b/: The current setup as used in Debian is to only clean /tmp on boot (which is pointless with /tmp-on-tmpfs) and never clean up /var/tmp There is also the tmpreaper package, which does periodic cleanup based on atime. I'd expect that to be installed on a lot of long-running machines, it is certainly installed on mine. Also if /var/tmp is no longer cleaned on boot, that is a bug. I'm not sure if we have software on long running servers which place files in /tmp and /var/tmp and expect files to not be deleted during runtime, even if not accessed for a long time. This is certainly an issue to be aware of and keep an eye on. IIRC it was an explicit decision not to install tmpreaper by default and leave cleanup policies to the local admin, for precisely this reason. It makes sense to revisit this decision 20 years later, but a lot of the original rationale remains valid: it affects only a handful of machines (because everyone else reboots often anyway), and we kind of expect those machines to be operated by skilled people. Simon
Re: finally end single-person maintainership
Hi, On 13.04.24 00:19, Marc Haber wrote: 'Require' is probably the wrong word. I simply have heard from several potential young contributors that they feel blocked by the tooling and specifically not everything in Git. That does not only apply to young contributors. I am an old fart and I still shy back from packages where I need to familiarize myself with an uncommon packaging toolchain. We cannot help people who want Debian packaging to work like Git, because Git is not a packaging tool, and neither are the forges. We're not even doing anyone a favour by introducing the git based workflows first, because about half of the techniques people know from git will conflict with something git-buildpackage or dgit does, and without a mental model of how Debian packaging is supposed to work standalone, they have no chance of solving even the simplest problem. For example, any repository that does not list debian/files and debian/*.substvars in the gitignore will fail to build twice in a row, because these files are created and are subsequently untracked. Only Debian packaging knowledge can tell you that these should never be checked in and can be ignored -- or we make people reliant on a magic tool to set it up properly for them. Once people are familiar with how Debian packaging works, we can introduce the git interfaces on top. Before that, git is more of a hindrance than a benefit to new contributors, precisely because it looks familiar, but the knowledge is not transferable. Simon OpenPGP_signature.asc Description: OpenPGP digital signature
Re: About Package Maintenance
Hi, On 4/9/24 01:48, Marc Haber wrote: Otoh, I am fully aware of the "you can't force a volunteer to do things", knowing that I myself would be the first one to violently oppose a decision that I don't like while - at the same time - being mad at some core package maintainers who have forced the project to jump through even more burning hoops to finally get the usrmerge migration The project has always had, and continues to have the two options "submit a patch" and "replace the maintainer." We have processes for that. No one, especially not the people driving the usrmerge migration, wants to do that, because that would require taking on a long-term commitment and the responsibility that comes with it, when they would prefer to do a drive-by contribution. Hence we are having a debate again whether there should be a mechanism to force maintainers to accept any patch thrown at them by someone "authorized" to perform a global change. No result can be reached from this debate, because there is no difference between force-accepting a contribution and replacing a maintainer. Simon
Re: finally end single-person maintainership
Hi Julien, On 4/8/24 22:45, Julien Puydt wrote: I only use salsa's git. That begs two questions: - What do I miss by not using the web interface? > - What does that web interface have that people don't like? It's a normal GitLab install. For anything that is a normal software project (like dpkg), you get all the features of a normal forge. But: forges do not map packaging workflows, so in practice you can only browse the source code. Simon
Re: finally end single-person maintainership
Hi, On 4/8/24 05:42, Wookey wrote: I don't mind what other people do, but I worry that conversations like this seem to take the new thing as so self-evidently better that no-one can reasonably complain about them being made a requirement. Well, we don't all think it's better, and I wouldn't enjoy seeing 'packages must be in Salsa' made a requirement, for example. What people seem to be missing is that the Debian archive *is* a version control system. Stacking another VCS on top of it leads to a lot of annoying artifacts, because there is no sensible metadata mapping between them -- what is metadata in Debian becomes data in git, and another metadata layer is added on top of that, and what is data in Debian must be run through a filter to make it accessible to git for efficient handling, so part of it gets converted to metadata. The result is... not ideal, because the resulting workflow is neither a Debian nor a git workflow, but a mix that requires users to be aware of the state of two systems at the same time. Testing a package requires me to commit everything into git first, so I have to remember to squash all these commits later. As long as there is no clear benefit, for example in a team maintained package where multiple team members regularly collaborate on pre-release states without uploading a Debian package in between, it mostly adds to my workload -- especially if no upload is made for a change, whoever does the upload will need to remember to review all of these changes. In the context of the xz debate, I'd expect this to increase, not decrease attack surface. The web interface presented by salsa also does not have the necessary data<->metadata filters to provide an actual insight into what is really happening in the repository. Because metadata is stored as data, building such filters would also be nontrivial. The existing gbp repositories pretty much all use a merge commit to integrate a new upstream version, but do not update debian/changelog in the same commit, so any tool analyzing this commit would see all the upstream changes for the new version as Debian specific changes against the old. Practically the only commits that are safe to analyze in this way are the tagged releases. What would make sense would be a git based format that is designed around the requirements for package maintenance, and where internal consistency can be enforced by upload hooks -- for example by storing metadata out-of-band in a separate branch. Simon
Another usrmerge complication
Hi, because life isn't hard enough as it is: When /bin is a symlink to usr/bin, and I install two packages, where one installs /bin/foo and the other installs /usr/bin/foo, then, if both are installed in the same dpkg invocation, the contents of the first package end up being installed, while in two separate invocations, the second package's contents win. So for any undetected file conflict, which file now exists now depends on whether apt decided to install both packages during the same installation round or not. Writing more test cases, Simon
Re: Requesting help with the t64 transition
Hi, On 3/5/24 17:56, John Paul Adrian Glaubitz wrote: For m68k, there is mitchy.debian.net and for powerpc, there is perotto.debian.net. As soon as the container with my stuff arrives, I have another A1200 with 68060 and 603e+. Alas, Linux does not support asymmetric multiprocessing so I cannot run both at the same time. Simon
Re: Bug#1065022: libglib2.0-0t64: t64 transition breaks the systems
Hi On 2/29/24 14:57, Steve Langasek wrote: Furthermore, this is a downgrade from a replacing package to a replaced package. Unless you also --reinstall the package at the end, missing files are quite to be expected. I wonder if this could be improved -- e.g. by ignoring Replaces: relationships from about-to-be-removed packages. Simon
Re: Bug#1061336: ITP: chatterino -- Chatterino is a chat client for Twitch chat. It aims to be an
Hi, On 1/23/24 05:25, matt wrote: * Package name: chatterino I might be interested in that. - I plan on maintaining it by compiling the latest package or use the deb they provide, inside a packaging team 樂 I would need a need a sponsor I currently have a 60 hour workweek, so I can sponsor packages if that is low enough effort for me. To be blunt, "us[ing] the deb they provide" sounds like you are not all that familiar with packaging and package maintenance, so this would require a lot more mentoring than I can currently provide. What is the current state of your packaging efforts, is anything online that I can take a look at? Simon
Re: Policy: versioning between releases
Hi, On 1/21/24 21:35, Matthias Urlichs wrote: Now … does that apply to crossing release boundaries? Specifically, if foo/testing requires bar >=1.1 to work but just states "Depends: bar >=1", and bar/stable is 1.0.42 … is that a bug? If so which severity? I'd say yes, with normal severity. There is no guarantee that releases will be upgraded atomically. Depending on the package in question, a too-loose dependency might even break release upgrades, and this information is also important for backports. There is also no good reason not to update the package in testing/unstable. Simon
Re: Policy: should libraries depend on services (daemons) that they can speak to?
Hi, On 1/16/24 03:55, Simon McVittie wrote: I would personally like to see *more* privilege separation across IPC boundaries rather than less, if that can reduce the total attack surface of the setuid/setcap executables in the trusted computing base. Yes, however there is a downside to building IPC microservices this way: we are now dynamically building trust chains, and the entire chain is not visible as the orchestrator can only witness individual connections between black boxes (same as socket activation hides dependencies between services). So we might have a component that has no special capabilities, but it is privileged by the trust relationship from other components, which may in turn have capabilities. Such components with incoming trust relationships would _also_ have to be written with minimal dependencies, and lots of mitigation code -- so while we're reducing the impact of a breach, I fear we're increasing attack surface by adding more privilege domains, and declaring some of them as "less critical". Simon
Re: Drawbacks of lack of mandated packaging workflow (Was: Re: Bug#1036884: 64-bit time_t: updated archive analysis, proposed transition plan with timeline)
Hi, On 06.01.24 22:15, Gioele Barabucci wrote: Aren't all these problems just inherent in Debian's lack of a mandated packaging tooling and workflow [1,2]? We have a mandated tooling and workflow. The tooling follows an interface that is defined in Policy. The interface is deliberately designed to be as flexible as possible. Most packages do not require this flexibility, which is why a majority use a library of helper functions that trades that flexibility for ease of use. This works because it is a solution that solves 95% of cases, and does not impose requirements on the remaining 5%. If you wanted 100% of packages to use this, and turn this into the new interface, then all these corner cases would need to be handled as well, and the interface extended. We also have a version control system -- the Debian archive. It, too, has a different focus than other version control systems, because it also includes a mirroring strategy. Switching to git would, again, require replication of the missing functionality, and it would require a lot of work to properly define all these interfaces and make sure they are extensible in the future. Simon
Re: Deprecation of /etc/alternatives? (Re: Reaction to potential PGP schism)
Hi, More metapackages will make transitions harder though, I believe we want to avoid that. In what way would transitions become harder? The alternatives system has "manual" and "automatic" modes for each group, these would probably correspond to "manually installed" and "automatically installed". The latter can only happen if there is also a "defaults" package that pulls in the metapackage we currently designate as default, and the "defaults" package must allow the dependency to be fulfilled by any of the choices. APT will prefer not to uninstall packages if it doesn't have to. As a result, any user will stick to whatever was default when they installed the package, even if they have not explicitly expressed a preference, until that option is no longer available -- the same happens with the alternatives system only in manual mode, which is an explicit choice -- and when it happens, the alternatives system will explicitly tell the user that their choice is no longer available and the setting has reverted to automatic mode. The only way to speed up a transition is to add an explicit hard dependency somewhere and override the user's choice -- which then adds an installation ordering constraint that might be the exact opposite of the order we want. Simon
Re: Deprecation of /etc/alternatives? (Re: Reaction to potential PGP schism)
Hi, On 12/28/23 04:28, Luca Boccassi wrote: if you want to activate a new alternative, you have to download a new package that provides it anyway, so there's no difference. Subsequent switches will use the cached package, and if you have issues downloading a 3 kilobytes metapackage then just ensure the cache is never cleared. More metapackages will make transitions harder though, I believe we want to avoid that. Simon
Re: /usr-move: Do we support upgrades without apt?
Hi, On 21.12.23 23:19, Antonio Terceiro wrote: I think so, yes. I don't think it's likely that there are people doing upgrades on running systems not using apt. What about aptitude and the various other frontends, like the DBus based package management tools? I'd expect quite a few people to use these for upgrades (I will certainly use aptitude). FWIW, I've made some progress with my dpkg patches, the attached log is the relevant snippet from running the tests[1]. It's still far from complete though. Simon [1] https://salsa.debian.org/sjr/dpkg/-/pipelines /usr/bin/make -C t-file-conflicts-usrmerge test make[1]: Entering directory '/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge' mkdir -p \ pkg-usrmerge/usr/lib64 \ pkg-usrmerge/usr/libx32 \ pkg-usrmerge/usr/lib32 \ pkg-usrmerge/usr/sbin \ pkg-usrmerge/usr/bin touch stamp-empty-directories dpkg-deb -b pkg-usrmerge pkg-usrmerge.deb dpkg-deb: building package 'pkg-usrmerge' in 'pkg-usrmerge.deb'. dpkg-deb -b pkg-a pkg-a.deb dpkg-deb: building package 'pkg-a' in 'pkg-a.deb'. dpkg-deb -b pkg-b pkg-b.deb dpkg-deb: building package 'pkg-b' in 'pkg-b.deb'. env PATH=/builds/sjr/dpkg/src:/builds/sjr/dpkg/utils:/builds/sjr/dpkg/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DPKG_DATADIR="/builds/sjr/dpkg/src" dpkg --admindir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb" --force-script-chrootless --force-not-root --force-unsafe-io --instdir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkginst" --no-debsig --log=/dev/null -P pkg-a pkg-b pkg-usrmerge dpkg: warning: ignoring request to remove pkg-a which isn't installed dpkg: warning: ignoring request to remove pkg-b which isn't installed dpkg: warning: ignoring request to remove pkg-usrmerge which isn't installed env PATH=/builds/sjr/dpkg/src:/builds/sjr/dpkg/utils:/builds/sjr/dpkg/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DPKG_DATADIR="/builds/sjr/dpkg/src" dpkg --admindir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb" --force-script-chrootless --force-not-root --force-unsafe-io --instdir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkginst" --no-debsig --log=/dev/null -i pkg-a.deb pkg-b.deb Selecting previously unselected package pkg-a. (Reading database ... 0 files and directories currently installed.) Preparing to unpack pkg-a.deb ... Unpacking pkg-a (0) ... Selecting previously unselected package pkg-b. Preparing to unpack pkg-b.deb ... Unpacking pkg-b (0) ... Setting up pkg-a (0) ... Setting up pkg-b (0) ... # # test if usrmerge fails because it introduces a conflict # ! env PATH=/builds/sjr/dpkg/src:/builds/sjr/dpkg/utils:/builds/sjr/dpkg/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DPKG_DATADIR="/builds/sjr/dpkg/src" dpkg --admindir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb" --force-script-chrootless --force-not-root --force-unsafe-io --instdir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkginst" --no-debsig --log=/dev/null -i pkg-usrmerge.deb Selecting previously unselected package pkg-usrmerge. (Reading database ... 6 files and directories currently installed.) Preparing to unpack pkg-usrmerge.deb ... + dpkg-alias --package usrmerge --add /bin /usr/bin (Reading database ... 6 files and directories currently installed.) dpkg: error processing archive pkg-usrmerge.deb (--install): new pkg-usrmerge package pre-installation script subprocess returned error exit status 1 Errors were encountered while processing: pkg-usrmerge.deb env PATH=/builds/sjr/dpkg/src:/builds/sjr/dpkg/utils:/builds/sjr/dpkg/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DPKG_DATADIR="/builds/sjr/dpkg/src" dpkg --admindir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb" --force-script-chrootless --force-not-root --force-unsafe-io --instdir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkginst" --no-debsig --log=/dev/null -P pkg-a pkg-b pkg-usrmerge (Reading database ... 6 files and directories currently installed.) Removing pkg-a (0) ... Removing pkg-b (0) ... dpkg: warning: ignoring request to remove pkg-usrmerge which isn't installed env PATH=/builds/sjr/dpkg/src:/builds/sjr/dpkg/utils:/builds/sjr/dpkg/scripts:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin DPKG_DATADIR="/builds/sjr/dpkg/src" dpkg --admindir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb" --force-script-chrootless --force-not-root --force-unsafe-io --instdir="/builds/sjr/dpkg/tests/t-file-conflicts-usrmerge/../dpkginst" --no-debsig --log=/dev/null -i
Re: /usr-move: Do we support upgrades without apt?
Hi, On 21.12.23 23:31, Marc Haber wrote: Do those GUI frontends that work via packagekit or other frameworks count as "using apt"? I now that WE recommend using apt in a text console outside of X, but even many of our own users do what their Desktop Environment does, and our downstreams like *b*nt* recommend other ways to upgrade as well. Oof, which reminds me: if you run nVidia closed source drivers, there is a good chance that dist-upgrade will kill your X server and send SIGHUP to apt and dpkg, leaving you in "you need to run 'dpkg --configure --all'" state. If that happens, you get another resolver run on a partially upgraded system. Simon OpenPGP_signature.asc Description: OpenPGP digital signature
Re: RFC: advise against using Proton Mail for Debian work?
Hi, On 11/15/23 08:40, Nicholas D Steeves wrote: 1. I've received a report that this provider is not appropriate for DM and DD use, because the key pair is stored on their servers. Ie: The applicant doesn't control the means to validating identity and authorship. Correct. I'd even go as far and say that using it should be a disqualifying factor. Upload permissions are tied to a gpg key, and the holder of the key needs to at least demonstrate good practices in using gpg, ideally also have enough understanding to be able to derive good practices and not just follow a set of rules. The gpg signature is not just a formal requirement where it doesn't matter how it is generated, but an integral part of the trust chain. A) Continue to explain this to new contributors on a one-by-one basis. I think that is part of Tasks C) Proton Mail begins to do something differently on their end, such as offering some features to Debian contributors that currently require a subscription. Unless that feature is "you control your own secrets", that makes no difference, it remains unsuitable and anyone using it in this manner demonstrates that they are not yet ready for the responsibilities of Debian membership. And "control your own secrets" as a perk of paid membership would be an even bigger level of wrong. Simon
Re: New Essential package procps-base
Hi, On 11/14/23 18:42, Andreas Henriksson wrote: Instead I think pidof can just be part of procps package. The sysvinit-utils package will then pull in procps via a dependency (once sysvinit-utils stops being Essential), which would smooth the transition for all sysvinit users until LSB pidofproc has been implemented in all init scripts. Makes sense, but this effectively means that "pidof" is no longer in the essential set unless you are using sysvinit. This will need to be communicated to Devuan at least (because they also support runit and openrc, and I believe neither pulls in sysvinit-utils explicitly, relying on its Essential status. We should also check systemd units for pidof invocations. A number of years ago I tested booting a regular debootstrapped system (with all priority important packages, etc) with sysvinit-utils excluded and that did not show a single warning about missing pidof. Someone might want to repeat that experiment. This check should probably use a dummy "pidof" on the path, I'd expect a lot of people use "if pidof x >/dev/null 2>&1" because of muscle memory, when they should not be redirecting stderr. Simon
Re: Linking coreutils against OpenSSL
Hi, On 11/11/23 03:34, Julian Andres Klode wrote: 1) we use libgcrypt in libapt-pkg, which needs global initialization. Libraries should not be doing the initialization, we're basically misusing it. I remember that KiCad had problems with OpenSSL for this exact reason -- we were using libcurl from multiple threads from different plugins, and OpenSSL required us to initialize some locking mechanism from the KiCad main program so plugins could operate safely, breaking the abstraction. For this reason, KiCad had a direct dependency on OpenSSL for years, even if libcurl was provided by libcurl-gnutls -- can't know whether libcurl pulls in openssl, so we need to initialize it. We removed that code a while ago after the offending plugins were removed, and it seems the OpenSSL API has removed the CRYPTO_set_locking_functions call since then so there is a good chance that they've cleaned up a bit. There are still some globals, but mostly for malloc replacements, so I guess these can be ignored. Simon
Re: Linking coreutils against OpenSSL
Hi, On 11/10/23 21:07, Stephan Verbücheln wrote: In my opinion, this is yet another reason to use a proper cryptography library (openssl, gnutls or gcrypt) instead of a custom implementation for this kind of algorithm. Yes and no. The reason several of our core tools bring their own functions is to specifically reduce the Essential package set. Hence this thread: we need to weigh the benefits against the drawbacks here. In coreutils' case, I think the benefits are kind of qualified by the number of direct users. Basically, few people have use cases that require them to routinely verify checksums with the tools from coreutils[1]. The main benefit of this move is that container images will shrink because libssl becomes part of the base layer, so fewer copies of it will be kept in stacked layers. I would disregard this as a benefit, otherwise we could make a case that more packages should be Essential. The actual drawbacks for users are minimal too: - systemd pulls it in anyway - apt will pull it in on the remaining systems I don't quite see the appeal of OpenSSL as a dependency for apt either. I have 2 Gbps Internet at home, and a laptop I bought in 2012, and apt is never CPU bound. I could see the benefit of gzip offloading into the kernel crypto engine, that would be noticeable to me and at least two other people. We already have two other such libraries in the quasi-essential set: libcrypt, and the Linux kernel. libcrypt: + already in the quasi-essential set (no extra space) - still slow kernel: + No extra space needed + Support for offload engines that use privileged access - Invisible dependency OpenSSL: + Handwritten optimized assembler functions for a set of architectures - Horrible code The optimized assembler function brings a massive speedup on amd64, which is what triggered this thread. The ARM NEON assembler code gives a moderate speedup for hashing compared to autovectorized generic code, but in general vector units are the wrong tool for cryptographic hashes, so I'm not surprised it isn't an order of magnitude. Over time, when these libraries add support for cryptography acceleration instructions for more architectures, all programs will benefit from it. Yes, but crypto acceleration in instruction form is difficult to implement in RISC architectures -- which is why these usually have separate DMA capable devices, and work queues managed in the kernel. I would expect that many rich ARM SoCs for phones, laptops and servers already have something and that openssl supports it already. What device did you run your benchmark on? I used a Zynq SoC, and just tested a random file I had that fit into memory, running sha256sum and kcapi-dgst -c sha256 five times each. OpenSSL is a bit faster (going from 1 minute to 45 seconds), but in a real world application I'd give this an offload engine on the FPGA side if it was a hot path, and ignore it if it was not. In summary, I don't believe this change has any measurable effect beyond growing the Essential set and improving artificial benchmarks, so I'm pretty lukewarm about this. Simon [1] "debootstrap george" is an outlier and should not have been counted
Re: Linking coreutils against OpenSSL
Hi, What would you think about having coreutils Depend on libssl3? This would make the libssl3 package essential, which is potentially undesirable, but it also has the potential for serious user time savings (on recent Intel CPUs, OpenSSL’s SHA-256 is over five times faster than coreutils’ internal implementation). That is only on amd64 though. On ARM and riscv64, OpenSSL is slightly slower than coreutils' sha256sum, so this would introduce an additional dependency and degrade performance. The best choice there is the kernel crypto API, which knows about offload hardware and special CPU instructions, both of which are common. Alternatively, what would you think about making sha256sum etc. divertible and providing implementations both with and without the OpenSSL dependency? Makes sense to me, especially if it is a diversion only, not full alternatives (which could not have sensible priorities, because what is fastest is system dependent. Simon
Re: [RFC] locking down rsyslog.service
Hi, On 10/17/23 01:24, Michael Prokop wrote: # Restrict access to the various process namespace types the Linux kernel provides RestrictNamespaces=true There is one plugin that uses namespaces. I wonder if it would make sense to split it out into a separate package, and have that package override the default configuration if it's installed. The capability set for rsyslog could be reduced quite a lot further if we could lobby the Linux kernel maintainers to add the open file limit (in CAP_SYS_ADMIN) and the socket buffer size limit (in CAP_NET_ADMIN) to CAP_SYS_RESOURCE), my expectation would be that these are the most common reasons these capabilities are set in other services as well. Could systemd be taught that certain capabilities are required depending on kernel version? Simon
Re: [RFC] locking down rsyslog.service
Hi, On 10/11/23 19:14, Michael Biebl wrote: - CAP_NET_ADMIN: use of setsockopt() - CAP_SYS_ADMIN: exceed /proc/sys/fs/file-max, the system-wide limit on the number of open files, in system calls that open files (e.g. accept execve), use of setns(),... I see, thanks! I looked over the code quickly, it seems the only privileged setsockopt() calls are to set larger buffer sizes. It may be good to lobby for these and the open file limit to be added to CAP_SYS_RESOURCE, that would allow locking it down further in the future. Simon
Re: Bug#1053782: RFP: node-vite -- Next Generation Frontend Tooling
Hi, On 10/11/23 16:00, Andrius Merkys wrote: Yes, but what does it do? Why would I pick this out of a package list if I didn't know the name of the package already? The following line in the RFP says: Vite is a frontend build tool, including development server and build command bundling code with Rollup, pre-configured to output optimized static assets for production. That is the long description. I have to view the details for a package to access the long description, and it still does not tell me anything about the package that would help me make a decision whether I want to install it. The most information is contained in the package name, which suggests that this belongs to the NodeJS ecosystem, and this is obvious to me only because I know that ecosystem exists and uses this naming convention within Debian. I have no idea how it fits into that system, and which specific problems it solves, so even if I were a NodeJS user, I still could not use this to decide whether I want to install it. The target audience for descriptions is "users who do not know the software yet." Simon
Re: Bug#1053782: RFP: node-vite -- Next Generation Frontend Tooling
Hi, On 10/11/23 15:30, Andrius Merkys wrote: Description : Next Generation Frontend Tooling Yes, but what does it do? Why would I pick this out of a package list if I didn't know the name of the package already? Simon
Re: [RFC] locking down rsyslog.service
Hi, On 10/11/23 03:22, Michael Biebl wrote: I intend to lock down rsyslog.service in Debian in one of the next uploads using the following systemd directives CapabilityBoundingSet=CAP_BLOCK_SUSPEND CAP_CHOWN CAP_LEASE CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_SYS_ADMIN CAP_SYS_RESOURCE CAP_SYSLOG Does it actually need CAP_NET_ADMIN and CAP_SYS_ADMIN? Everything else looks good to me. Simon
Re: apt-listchanges: appropriate heuristic for ignoring sub-packages with symlinked /usr/share/doc directories
Hi, On 10/10/23 06:24, Jonathan Kamens wrote: * binary package name different from source name * deb contains no changelog or NEWS files in /usr/share/doc * creates a symlink in /usr/share/doc with the binary package name as its name, pointing at another directory in /usr/share/doc I'd go an easier way: if the changelog is a symlink, its target is the changelog to use, so the same changelog (with the same file name) will be used by multiple packages, and all that's left is to deduplicate this list. Simon
Re: /usr-merge and DEP17 update: what happens next and how you can help
Hi, On 10/10/23 03:16, Helmut Grohne wrote: For one thing, dh_installsystemd generates maintainer scripts for restarting services. Before version 13.11.6, it did not recognize the /usr location. If you were to backport such a package, bookworm's debhelper would not generate the relevant maintainer scripts. You can mitigate this by issuing "Build-Depends: debhelper (>= 13.11.6~)". Thus, you'll be using a backported debhelper (unless the backporter carelessly deletes this dependency). If that would be the only reason to require a backported debhelper, then it would probably be better to make the file move conditional on the target distribution (from "dpkg-parsechangelog -S distribution") -- the intention of that is immediately readable, while a versioned build dependency needs to be explained or a backporter might want to carelessly delete it. gcc does something similar, although that uses "lsb_release -cs", which is technically not entirely correct since it looks at the build system, not the current package, but for the purpose of backports is good enough and avoids corner cases like "stable-proposed-updates" not matching "bookworm" or "bookworm-backports". Simon
Re: Is there a generic canonical way for a package script to check network connectivity?
Hi, On 09.10.23 11:49, Jonathan Kamens wrote: I need to be able to tell from one of my package scripts whether the host has networking connectivity. 無. There is no such thing as "networking connectivity." There is "has access to a particular service, at this precise moment in time" and "is usually connected to an unmetered Internet connection so there is a reasonable expectation that a particular service can be reached and should be made part of a workflow." The latter, for example, is how reportbug deals with this: during configuration, it asks whether the host is typically connected to the Internet, and if you respond yes, it will query Debian services automatically when creating a bug report, and if you respond no, it will ask every time. In general, I'd rather see an interactive process than background network access by default. When I got a shiny new LTE modem in my laptop, Windows Update burned through my monthly quota within three seconds, and I have very little desire to see Debian repeat _all_ the mistakes of Windows. Background activity is fine if explicitly configured by the user, which can be a configuration setting (reachable/preseedable through debconf) or even take the form of separate packages. That this is done so seldom is that there have been no really good use cases for it -- basically the only one we have in Debian is unattended upgrades, and that one is a separate package that additionally provides a policy interface. Everything else either has an interactivity requirement anyway (reportbug needs to know the name of the package first) or does not benefit substantially (either it's fast, then we don't gain much, or it's slow, then it's usually metered and we don't want speculative downloads). Last but not least: if this is related to apt-listchanges, then maybe think about the offline case first (upgrades distributed via CD-ROM or USB stick), because pretty much everything that works there can also be implemented through the mirror network and use the existing load balancing and site policy infrastructure. Adding an extra index to the archive is cheaper than running an API endpoint for millions of clients. Simon OpenPGP_signature.asc Description: OpenPGP digital signature
how-can-i-help by default [Re: lpr/lpd]
Hi, On 9/25/23 14:08, Paul Wise wrote: The problem with that approach is that the help needed information changes independently to packages, so the information will get very out of date in between point releases, which is why how-can-i-help does online checks. If desired, it would be easy to have an apt update hook that would download the data from UDD so that how-can-i-help can work offline when the network is available. Yes, that's why I thought of using indices/ -- there are already Maintainers and Uploaders files in there that I'd expect are also meant to change independently from Packages. The indices/ directory seems to be available on most if not all mirrors as well, so putting it there would not put unnecessary strain on UDD. My proposal is to display the list of RFA/O packages installed on the system by default as part of the after-installation summary, because we currently have no way of reaching users of at-risk packages before the package is removed, and I would like to change that. Simon
Re: lpr/lpd
Hi, On 9/23/23 12:10, Paul Wise wrote: You may be thinking of how-can-i-help, which is recommended to every new person who asks about how to contribute to Debian. There is also the older wnpp-alert in devscripts. What I'd like to see is something like Scanning processes... Scanning processor microcode... Scanning linux images... Running kernel seems to be up-to-date. The processor microcode seems to be up-to-date. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. No installed packages are looking for a new maintainer. And ideally, that check would be quick and use information already on the system at this point, maybe a supplemental file under indices/ that can be downloaded during an update. Simon
Re: lpr/lpd
Hi, On 9/22/23 16:41, Christoph Biedl wrote: That's not surprising: lpr is an old technology, it may be simple but it has quirks. People moved on, and if they cared a little, they let go. Erm. We're talking about printers here. lpr is the protocol with the fewest quirks. I agree that the desktop users have moved on, and they will move on to whatever we present them as default next. But I also think there are quite a few installations that still use these packages, and where people don't have much of an issue with the packages being effectively NMU-maintained, because their system works, and change only brings uncertainty -- especially if it is change towards something that is "actively" maintained and constantly updated. My suspicion is that a lot of these installations will have a heavy amount of homebrew scripting added to them, so supporting them from a newer printing system would mean creating a bug-compatible emulation, which we can all agree is a forever maintenance burden unless you also have a plan to migrate these users to a legacy-free environment. Do we provide our users a good service if we keep such zombies alive for such a long time? Yes and no. We're providing a better service than pulling the rug under the users, but we could do better by communicating that these packages are in need of new maintainers instead of waiting for them to bit-rot to a point where we have an excuse to remove them -- because all we're doing that way is justify the rug pull, but the impact for the users isn't minimized. Plus, most of that code is in C, and I take the liberty to state they all have security issues. They are just unknown, because no one bothered to take a closer look. But these bugs exist simply because twenty and more years ago, secure programming wasn't that common. It is simple, straightforward code with a service record of over twenty years, and I remember that we had security researchers back then and there were several advisories on bugtraq for lprng. Without an external limitation (mostly specific hardware) it's hard to draw a line when it's obviously the time to remove near-dead packages, at least from the stable releases. I don't have a good idea either what to do here. Same. I think that it cannot be solved on a per-package basis, instead we might need infrastructure. One thing I'd think might help would be a tag in the package database that is derived from WNPP status, which would allow the summary output at the end of installs also list packages that are installed that are currently in RFA or O state. I believe we used to have a tool for that in Debian, but almost no one had it installed because it had additional overhead, making it pointless. Simon
Re: Do not plan to support /usr/lib/pam.d for Debian pam
Hi, On 18.09.23 05:16, Julian Andres Klode wrote: If you follow the argument for /usr to its logical conclusion of being the complete image, you end up moving state of the image (as opposed to the system) from /var/lib to /usr as well, for example /var/lib/dpkg and /var/lib/apt/extended_states. The way I understood it, images cannot be updated. Instead, users will build or receive a new image, and quickly "reboot" into that by having running services transfer their state over to new instances. Simon
Re: lpr/lpd
Hi, On 18.09.23 04:29, Russ Allbery wrote: It at least used to be that you could print directly to a remote printer with lpr and a pretty simple /etc/printcap entry that you could write manually. I used to use that mechanism to print to an office printer until I discovered rlpr, which is even better for that use case. It's possible some of those installations are people doing that, rather than via dependencies or other things (in which case they probably should move to rlpr). Yes, that's basically how I use it. Pretty much all existing printers with network interfaces support the BSD lpr protocol still, and accept PostScript or PDF. People in Germany are likely to have a FritzBox DSL router, which conveniently allows you to export USB printers that way too. And yes, it is quicker for me to copy another printcap entry and swap out the host name than it is to find out how to authenticate to CUPS, set up the printer, print a test page then remove and recreate it because the generated "short" name I need to pipe data into lpr isn't short. I will definitely be looking into rlpr. I think those packages are probably still useful to someone, I guess several universities will be running these because they are small and robust, and can just leave a job in the queue if there is a temporary problem rather than mark the printer as offline and any jobs queued to it as paused. Oddly specific rant, I know, but it's "small" things like that that can break a use case completely, and that's why we usually ship alternative implementations for everything, and why a lot of seemingly small changes are controversial: they break non-standard use cases. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: /usr/-only image
Hi, On 9/11/23 23:08, Simon McVittie wrote: Some packages rely on their own configuration existing in /etc. For these packages, ideally they'd try loading from /etc as highest priority, but fall back to /usr as a lower-priority location. This is a package-by-package change, and probably best achieved upstream. The problem isn't so much the location of the configuration file, but the method used to merge default, distro-provided and system-specific configuration, and how much deviation from the default configuration is expected. I'd argue that udev and systemd are kind of special here in that they mostly provide a registry for other components to hook into, and that the majority of users stick with the default configuration. This would not work for a package like postfix, which absolutely requires system-specific configuration, and we'd have to be careful with packages like postgresql where there is a default configuration that works fine for hobbyists that we do not make life too difficult for professional users. Simon
Re: Second take at DEP17 - consensus call on /usr-merge matters
Hi, On 7/11/23 00:55, Sam Hartman wrote: * The more I look at this, I think the real complexity is not in bootstrapping, but is in the rest of the proposal for canonicalizing paths. I am very uncomfortable overall; it seems complicated enough that we probably will break something somewhere. I do not see anry better options though. I think this affects things in a couple ways: * I hope we can put the bootstrapping decision behind us soon and focus on the harder problem, because I think bootstrapping is a bit of a bikeshed at this point. There is only one important decision to be made about bootstrapping: do we want to extend the protocol, or not? If yes, then the rest of the bootstrapping process can be decided after we have a solution for upgrades, and especially should not put additional constraints there, so I'd explicitly avoid making a decision here that will then reappear with a prefix of "but we already decided that" during the complex part. If no, then bootstrapping becomes part of the constraint set, same as "upgrades need to be facilitated through packages whose installation order is defined through package relationships as interpreted by the current stable release of apt" we also get "packages need to be installable by unpacking their data member through tar, then subsequently installing the package over that with the just-unpacked version of dpkg." And because the bootstrap scenario uses a as-of-yet unreleased version of the dpkg package, we have way more freedom there than with the upgrade process, so optimizing this first is the best way to sink a lot of cost into a solution. Simon
Re: Second take at DEP17 - consensus call on /usr-merge matters
Hi, On 7/7/23 16:55, Helmut Grohne wrote: If we have a consensus we're unwilling to wait for a patch, it doesn't matter whether that's because: While you try to remove the reasoning for this point of view from the consensus, the "willing to wait" language implies a reason of "too slow" already. The main problem I see with that language is that it is entirely passive, as if a patch would appear by waiting for it to do so. Right now, I seem to be the only person working on one, which is clearly ridiculous, as I can invest maybe a total of four hours per week into it, so obviously I need to be realistic here and say that it is a bad idea for the project to wait for me to implement this, and I don't want to be in the position of having the project wait for me either because I already have that in 60 hours of my life between Monday and Friday. We also have quite some disagreement on what "the patch" is in terms of what semantics would help here. Yes, that's underspecified. I've tried to get started on this in [1], that is by no means complete but it has not generated any comments either. If there is interest in dpkg changes as a solution, then I could add this to the document. I carefully avoided adding reasoning to the proposed consensus as I was seeing our disagreement on reasoning. I now see how that second sentence could be read as precluding a dpkg patch in general. Would it help to add "+For the purpose of finalizing the transition,+ dpkg is not augmented ..." to leave open the possibility to add such a change but still implying that we are not waiting for it to happen? Yes, that was why I objected to that wording as well. I do agree that we need to move forward without waiting for a patch, if only out of the self-serving interest that I don't want to sit on the critical path. I think we need consensus on decisions and confirmation that everyone feels heard. Heh. I see how you get there. I agree with that latter part and tried to use the agreement on problems and mitigations as a vehicle for ensuring that everyone feels heard. Evidently, that does not work out either. It's the best way I can see to reach a consensus, but consensus, like voting, can only allocate resources and modify procedures. If we have a viable technical procedure that finishes the transition, then we can get consensus to implement it. If we have a technical procedure that requires changing a procedure to be viable (such as lock-step upgrade of packages in the archive so the installer is never presented with an inconsistent set), then we can get consensus to modify the procedure and implement the solution. However, if there are open questions about the technical viability of a solution, then the answer to those should be technical as well. Hence my suggestion to say we have consensus to focus our efforts on the solution space that doesn't require a dpkg change -- that is purely an allocation of resources, and the remainder of the document is a very useful collection of the knowledge we have gathered so far, the avenues that have been investigated and possible blockers for those. We can also try to get consensus or votes on expanding the solution space: - mitigations that require archive-side changes, like an upload check that a package doesn't trigger a known file loss condition - mitigations that require users to upgrade using a special procedure - mitigations that (further) violate policy by e.g. manipulating the dpkg database from the usrmerge package, or wrapping dpkg in a diversion - mitigations that require updating a stable release, such as shipping an updated debootstrap package I'm not sure all of these would be unanimous, but if there is a technical solution that becomes viable through something we can vote on, that is completely valid. In any case, the rough consensus on moving forward without major changes to dpkg (for reasons we disagree about) paves the road for selecting a subset of mitigations and proposing that as decision. The major missing piece to perform this selection is an agreement on how we protect aliasing symlinks from deletion (aka P9), because we still have quite strong disagreement on the mitigations selected there. The questions I see are - Can we get a full column of checkmarks by any combination of mitigations? - Are there procedural mitigations that can be useful? - Are these mitigations mutually compatible? Simon [1] https://lists.debian.org/debian-dpkg/2023/06/msg00050.html
Re: Second take at DEP17 - consensus call on /usr-merge matters
Hi Helmut, On 6/29/23 04:37, Helmut Grohne wrote: Consensus proposal #1: This updated DEP17 is a complete representation of the known and relevant problems and known mitigations under discussion at the time of this writing. Do you miss a related problem important to you? Do you miss your preferred mitigation? Please speak up, so we can record it. I think "backports" are missing as a user story. Most packages should be harmless, and the Contents file for bullseye-backports doesn't have too much in any of the affected directories. For /bin and /sbin, the list is short: admin/brltty admin/btrfs-progs admin/e2fsck-static admin/e2fsprogs admin/glusterfs-client admin/logsave admin/mdadm admin/moosefs-client admin/systemd admin/systemd,admin/systemd-standalone-sysusers admin/systemd,admin/systemd-standalone-tmpfiles admin/systemd-container admin/systemd-resolved admin/systemd-sysv admin/udev net/iproute2 net/wpasupplicant but the list of packages installing files into /lib is longer and includes all the kernel backports, so I guess that is another potential source of problems. There might be an easy solution here, I have not investigated this very deeply because it is a workday and 11 hours out of every day are already spoken for. Stating a goal has been quite difficult, but I think that most of the participants agree that we want to eliminate the file move moratorium without getting problems in return. I'd even widen that to "no more special handling needed in any place for the transition", with the moratorium being an example of that. When we get into mitigations, consensus is hard to come by. My impression is that we have roughly two camps. One is in favour of major changes to dpkg and the other is not. It's difficult to summarize the situation in one sentence, because neither group is really objecting to dpkg changes, so I'd put the fault line at whether the transition should be facilitated through dpkg or not. * Even if dpkg were fixed in trixie, trixie's libc6 would still be unpacked using bookworm's dpkg. At least for this package, we cannot rely on dpkg changes in trixie. Therefore we need workarounds or spread the transition to forky. For other packages, even a Pre-Depends on dpkg is not a guarantee that a changed dpkg will perform the unpack. * Changes to dpkg will not fix bootstrapping. The dpkg changes will fix bootstrapping, but we can't finish the transition until forky this way, because we need to be able to bootstrap with a libc6 package that can be installed from bookworm. The subset of affected packages is rather small though. There also is a minority arguing in favour of doing both. I've kinda ruled out that option already as we get the downsides of both without any further benefit in return. I think that was me being pessimistic and expecting that we're going to end up doing both anyway. The downsides of the combination approach are more severe than those of either solution in isolation, because dpkg incurs a permanent and significant performance penalty to verify its database against reality, and gains additional error states. So unless there is someone else, I'd call it a stretch to say that someone is "in favour" of this solution. Consensus proposal #2: The primary method for finalizing the /usr-merge transition is moving files to their canonical locations (i.e. from / to /usr) according to the dpkg fsys database (i.e. in data.tar of binary packages). dpkg is not augmented with a general mechanism supporting aliasing nor do we encode specific aliases into dpkg. I recognize that this is not unanimous, but I think we still have sufficient consensus on this. I suspect that maybe Simon Richter and a few more would disagree here. If consensus fails, we may have to put this to a vote. We need to be careful here to not conflate the goal of the transition with the method for reaching it. We have consensus on the goal (basically, data.tar and filesystem matching is the definition of "done" I use for this transition). We do not have consensus on the technical implementation because there are people who believe the technical implementation proposed is not actually feasible. In my opinion, it is a 95% solution, which is very tempting but we need the remaining 5% as well. To a large extent, we are having this discussion because the usrmerge package left us with a 90% solution. All a vote can achieve in that situation is assign responsibility, and authorize more drastic measures such as bypassing regular procedures (e.g. the "flag day" transition proposal for bootstrapping). Unless the vote clarifies which of those it is, we will have disagreement on the scope, which would not be much of an improvement over the status quo where there is disagreement on the implications of the TC decision. The minimal
Re: Second take at DEP17 - consensus call on /usr-merge matters
Hi, On 6/30/23 02:32, Helmut Grohne wrote: If the dpkg maintainer were to merge aliasing support, I haven't seen anyone objecting strong enough to try and override that maintainer action for example. Correct. The proponents of the "work around dpkg" approach quite often insinuate that it would be impossible to get changes merged for "social reasons", but there is no opposition from either camp to changing dpkg. I think this is a misrepresentation. There is no readily mergable patch for aliasing support. The most complete one is the tree from Simon Richter and he considers that work incomplete still. At this time, it's no a question of merging or not really. Indeed. I'd place progress at 10%, with 80% being "call for testers", 90% "mergeable", 95% "translations done" and 100% "Policy updated." I feel confident that this approach works, but there is a lot of boxes to tick, like diversions and alternatives, it will require a massive test suite, and it will require documentation as there is a new admin tool. Simon
Re: Replaces without Breaks or Conflicts harmful? [was: Re: Policy consensus on transition when removing initscripts.]
Hi, On 6/29/23 04:49, Helmut Grohne wrote: * Package A 1.0-1 is installed providing file F. * File F is moved to package B as of package A 1.0-3. * User installs package B, which replaces the file in package A. * User uninstalls package B. F is now gone, even though it's supposed to be still shipped by A 1.0-1. I am convinced by this. I think this is a sufficiently bad footgun to simply forbid Replaces that are not covered by a suitable Breaks or Conflicts relation. That is already in Policy 7.6.1, with a footnote that gives exactly this explanation. Simon
Re: Policy consensus on transition when removing initscripts.
Hi, On 6/29/23 01:56, Sam Hartman wrote: Russ> This feels like exactly the kind of problem that normally Russ> Debian goes to some lengths to avoid creating, but it sounds Russ> like several commenters here don't think the effort is worth Russ> it? Normally, Debian spends a fair bit of effort to avoid these kind of breakages. But I think init systems other than systemd are in kind of a special case. I agree, but it doesn't make a difference anyway. The init scripts are conffiles. Dpkg should not remove them on upgrade, so no such window exists, unless maintainers explicitly remove these files, which would be "deleting user data" and therefore wrong, except in very specific cases. Longer term, we might either want to also develop a strategy to remove these files, or decide to not bother because the impact is minimal, especially without the compatibility layer and with immutable images. It also seems a bit strange to require more from the maintainer when they are dropping an init script than we would if a maintainer started depending on a feature like socket activation such that their packkage simply would not work without systemd. This would be a case where the init script and the systemd unit deviate in functionality. I don't see a problem with that, and my expectation is generally that the people running sysvinit and the people running systemd have different expectations here anyway. Very few services depend on systemd and are completely unable to manage their own listener sockets, and those that do never had an init script in the first place and are not in use on any machine not running systemd, so there is no regression. In general, the sysvinit crowd prefers "traditional" Unix daemons, many of which also exist on BSD, so these are unlikely to lose the ability to work outside the systemd environment. Simon
Re: Developer Workload and Sustainability
Hi Russ, On 6/29/23 01:58, Russ Allbery wrote: According to Policy as currently published, systemd units are encouraged, and init scripts are mandatory. I thought I found and fixed all of the places init scripts were said to be mandatory, but apparently I missed one. Could you file a bug and point it out? That obviously need to be fixed. It appears I have misread that paragraph. Neither are mandatory, just encouraged, which makes sense to me. Policy certainy needs work. What is definitely missing is the feature set systemd units can expect. Since we're effectively delegating part of Policy to systemd upstream, this reference should be made explicit, and versioned. We certainly do not want to maintain our own copy of that documentation. What is unclear to me is whether we want to include smaller policy items imported from systemd verbatim or by reference (i.e. is usrmerge a Debian policy, or a systemd policy?). Simon
Developer Workload and Sustainability
Hi, On 6/28/23 22:42, Holger Levsen wrote: I'm not sure Debian Policy is the best place to document this, because Debian Policy documents what packages *must* comply with, while legacy initscripts are a thing of the past which still are permitted (and liked & prefered by some), so *maybe* src:dev-ref would be a better place for documenting those best practices? According to Policy as currently published, systemd units are encouraged, and init scripts are mandatory. We have GR and TC decisions that say otherwise, but the proponents of the systemd migration have, to this day, not done their homework, to a large degree because there are rather few people doing the actual work[1], and, as Luca indicated, that happens largely in their spare time after a full workday working on the same software. This is not sustainable, and it cannot be made sustainable by incurring more technical debt[2], and we will need to address this at a project level soon. A large part of the opposition to systemd is that this situation was predict(able|ed), yet people kept forging ahead anyway. That's the thing: few people want init scripts. I don't want init scripts either. What I want is an init system that can be maintained inside a community project like Debian without burning out people and endangering the long term viability of the project. That is where systemd fails us as a community project, because the environment in which most of development is happening is not hospitable to community building efforts, and the complexity of the project constitutes a high barrier to entry, which acts as a further selection filter for contributors. This is why we are stretched so thin here: anyone we wanted to onboard into systemd package management we'd have to properly train for this role, we have no process for that because we didn't need one before, and everyone is aware that it will be a thankless job because a large part of it will be taking decisions made upstream into the project. I don't yet see a clear path out of this. The only thing that is obvious to me is that this is not a technical problem[3]. Simon [1] a distinct group from "proponents" [2] and pointing out that we are incurring technical debt is additional stress and makes the people trying to come up with solutions feel unappreciated. [3] apart from the existing technical debt
Re: Policy consensus on transition when removing initscripts.
Hi, On 6/28/23 15:19, Russ Allbery wrote: Yeah, I knew that part, but for some reason I thought we normally always combine Replaces with Breaks or Conflicts even for other cases. Maybe I just made that up and confused myself. No, we just have very few use cases for Replaces alone these days, since typically files moving from one package to another will take place within the same source package, and they will just either have a strong dependency on the exact same version, or a conflict with all other versions. Simon
Re: Policy consensus on transition when removing initscripts.
Hi, On 6/28/23 13:05, Russ Allbery wrote: In that case, I don't actually know why we usually use Conflicts with Replaces. Maybe we don't really need to? Replaces+Conflicts together has a special meaning, that is used for replacing a package completely in an atomic operations, such as when a package is being renamed. Replaces alone defines how file conflicts are to be resolved. Conflicts alone stops installation if the conflicting package is still installed, and the frontend is responsible for uninstalling the other package first. With both Replaces and Conflicts, the new package can be installed over the old one, causing the old package to be deinstalled at the same time, but only if unpacking the new package succeeds. This is very useful for replacing a package that other packages depend on without going through an invalid state that needs --force to get to. Policy 7.4 specifies this behaviour. Simon
Re: Policy consensus on transition when removing initscripts.
Hi, On 6/28/23 02:31, Russ Allbery wrote: Normally Conflicts is always added with Replaces because otherwise you can have the following situation: * Package A version 1.0-1 is installed providing file F. * File F is moved to package B as of package A 1.0-3. * User installs package B, which replaces the file in package A. * User upgrades package A to version 1.0-2 (*not* 1.0-3). Or, rather, tries, because this will fail with an error due to the file conflict. No, that is fine. "Replaces: A (<< 1.0-3)" is sufficient here that the file is not unpacked from A 1.0-2. (Reading database ... 3 files and directories currently installed.) Preparing to unpack A-2.deb ... Unpacking a (2) over (1) ... Replaced by files in installed package b (1) ... Setting up a (2) ... Debian packaging is already hard enough; we should automate as much as we possibly can. Nothing against automating it, but this doesn't need to be done centrally, so I'd expect the orphan-sysvinit-scripts maintainers to set up a cronjob or systemd timer[1] that checks for removed init scripts if they felt that was necessary. I don't see the need for mandating that automation for what is essentially a very small transition that is low-impact and well-supported by our existing tools, and doing so does not decrease the workload on individual maintainers because they still need to look up the appropriate action to take on removal of an init script even if that action is "do nothing." A comprehensive checklist of everything you're supposed to think about when packaging a Debian package doesn't exist (Policy is certainly shy of that), and if it did, would form a hardcover book large enough to use as a boat anchor. We have an upgrade checklist for moving to a newer Standards-Version, and we have the Developers' Handbook. Simon [1] probably not
Re: Policy consensus on transition when removing initscripts.
Hi Ansgar, On 6/27/23 01:45, Ansgar wrote: [systemd service wrapper provided by init] I think sysvinit maintainers looked at such ideas in the past, but weren't capable to get it to work. That might be a blocker for such approaches. There was also a GSoC project in 2012 and some other work. I'm fairly sure they are capable of doing that, but there is no point in doing so. There is already an implementation of the systemd service API, and it's called systemd. This interface is not useful as an abstraction layer because it is tightly coupled with the existing implementation, to the extent that writing a conforming implementation requires largely making the same design choices. This work would be redundant, and if we had volunteers for that, they could be working on systemd instead, that would make more sense. The init.d interface, too, is directly tied to an implementation. It can be implemented by a compatibility layer, which doesn't work the other way around, but you are losing most of the benefits of the systemd interface, so the desire to drop this layer is understandable, and has been predicted (and those predictions rejected as paranoid) back when systemd was introduced. There is a fundamental disagreement between the systemd and sysvinit camps, and it is not at an implementation level, but at conceptual and community management levels. We cannot implement our way out of this disagreement. Fortunately, we don't have to. Proposed policy: - Each package that contains something that is useful as a systemd unit should also ship the unit definitions. There may be exceptions, so I'm writing "should", not "must". - Each package that contains something that can also be a daemon can, but does not need to ship an init script. The maintainer of a service package should know best how to invoke the program, so there is a technical benefit in having them maintain the init script, but that only extends as far as their motivation to do a good job. - If an init script exists, a service unit with the same name needs to exist as well so systemd before the removal of the compatibility layer ignores the init script. We will have a full release cycle with an old systemd and backports. - Unit files and init scripts do not need to provide the same functionality. The sysvinit users have different requirements than the systemd users. There are services that have their own implementation of a graceful restart that cannot be used from systemd, and there are services you would not find on a machine running sysvinit anyway, so any effort writing an init script for those would be wasted. - Maintainers dropping init scripts are asked to file a bug on orphan-sysvinit-scripts so they can be added there. Orphaning an init script is just like orphaning a package: you're handing it over to a team that is not necessarily that invested in your software as you are, but if you feel you're not doing a good job at it, stepping down gracefully is expected of a maintainer. Extending this notion to a part of a package is not a big stretch. Neither orphan-sysvinit-scripts now! nor the packages it ships legacy startup scripts for seem to have Replaces or Conflicts though? The only thing we actually need is a versioned Replaces that allows orphan-sysvinit-scripts to take over ownership of the conffile. Conflicts is unneeded here, and the daemon package does not need to declare any relationship. They can use Depends: systemd-sysv | orphan-sysvinit-scripts but really that doesn't do much because orphan-sysvinit-scripts is going to be pulled in anyway, so I'd rather avoid the clutter. Less prone to errors than a manual process might be to watch automatically where legacy startup scripts disappear anyway; it's not that complicated to do. People tend to forget things. No, we shouldn't set the bar this low. We're talking about DDs here, they should all have passed P and T tests. Simon
Re: Policy consensus on transition when removing initscripts.
Hi, On 6/25/23 23:15, Mark Hindley wrote: The most recent proposal[6] for updating the Policy with a requirement to use tmpfiles.d(5) states "Init systems other than ``systemd`` should allow providing the same functionality as appropriate for each system, for example managing the directories from the init script shipped by the package." The way I see it, we are getting a split between "daemons" and "services", and it simply does not make sense to start a "service" directly from an init script, because it requires the service orchestrator as its runtime environment. Init scripts are useful for starting daemons. If you need a service started, you need an appropriate service wrapper, and that is outside the scope of an init system. The lack of these interfaces is not a deficiency, but a deliberate design decision. My expectation is that some daemons will gain some systemd integration, but keep the non-systemd code because it is still useful during debugging and in container environments, and init scripts would use the non-systemd invocation interface. This creates an inconsistency whereby non-systemd inits are required to provide functionality in their initscript, but that initscript is not required to be present. "If it is present, it also needs to work." sounds like a reasonable statement though. Simon
Re: [Pkg-opencl-devel] opencl-icd virtual package(s)?
Hi, On 6/19/23 01:37, Vincent Danjean wrote: All we can do with the dependency system (not yet done) would be to force a dependency on a new virtual package to ensure that at least one ICD with the correct minimal version is available. I'm not sure if this is really possible (is it possible to install several packages providing the same virtual package ?) Yes, that is fine. The typical pattern for most packages that provide virtual packages is to also conflict with the same virtual package, because that doesn't make sense, but if you don't declare that conflict, packages will happily co-install. Since these provide a shared library with a fixed name, the ICD loaders either need to conflict or provide alternatives. Simon
Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)
Hi, On 10.06.23 00:41, Steve McIntyre wrote: What exactly do you mean here? You know that even a statically linked executable needs an interpreter defined in the ELF header? /sbin/ldconfig has no PT_INTERP segment. If you use libdl, you need to be loaded through ld.so, and since PAM uses libdl to load plugins, accessing databases doesn't work when statically linked without an interpreter, so a lot of people use an interpreter even in a static link, but it's not required (and would be counterproductive, the interpreter chain needs to terminate somewhere). Simon OpenPGP_signature Description: OpenPGP digital signature
Re: [2016] client-side signature checking of Debian archives (Re: When should we https our mirrors?)
Hi, - when you use switches, the local network segment has no other nodes - if there were other nodes, they would likely miss some packets in the conversation, which means they cannot generate checksums - there is no software that can perform this inspection Yep, there are limitations to work within, and the result would be the groundwork for some kind of 'integrity appliance', either within the local network or deliberately outside of it. There are commercial security solutions that aim to implement this, and many of these also contain solutions to inspect the contents of TLS traffic, usually by creating their own CA and requiring clients to install their certificate as trusted. These are then rolled out on the edge of company networks. As you can imagine, that comes with its own set of downsides, and I'd argue it makes things a lot less secure, especially if you treat it as a product that can be installed once and then forgotten about. For the specific case of package downloads, I'd also argue that it is unnecessary, because apt already verifies integrity on the client, and such an appliance would have no additional information to act on that the client does not have. An alternative (that may already exist, I would be surprised if not) would be to run two-or-more instances of the same computing hardware with key I/O interfaces between all of them. This is being done already, but systems like these are generally not connected to the Internet. I think it's fine if we're making a change that is broadly agreed upon; I wasn't able to find a decision that would be suitable to link to from the documentation (or commit log) when searching the mailing lists earlier, and since it's debatably-meritable change with potential impact on users and infrastructure (albeit in a gradual way) I wanted to raise some awareness and also check where the reasoning for the change originates. I'd also have to search the archives, but I remember the last discussion as I was part of it. The reason for the change is that it reduces user confusion. Users are learning that unencrypted HTTP has neither integrity nor confidentiality, and that they should actively check that web sites use HTTPS, so we have gotten several inquiries why apt uses an "insecure" protocol. It takes time to explain that apt performs its own integrity checks, and we can not and should not explain away the missing confidentiality, because we're happy that they want to protect their information and are insisting on using TLS -- that is far from perfect, but already a vast improvement. Infrastructure-wise, Docker is the far bigger problem, because people keep downloading the same packages over and over again, and the traditional mirror network donated by universities already had trouble keeping up. Two large CDNs are also donating mirrors now, and we use these preferentially. The CDN mirrors also come with caching infrastructure and can terminate TLS, which the traditional network cannot because there is no key/certificate distribution infrastructure (and such a thing can only exist within a single organization), so the last strong technical reason for not using HTTPS went away. The remaining technical reasons are weak, so a non-technical reason prompted the change in default. Simon
Re: [2016] client-side signature checking of Debian archives (Re: When should we https our mirrors?)
Hi, On 5/31/23 05:42, James Addison wrote: * It allows other devices on the local network segment to inspect the content that other nodes are sending and receiving. That is very theoretical: - when you use switches, the local network segment has no other nodes - if there were other nodes, they would likely miss some packets in the conversation, which means they cannot generate checksums - there is no software that can perform this inspection A place where such an inspection would be possible is a local proxy, but this is also not being done because even a caching proxy wants to forward data as it arrives, and there is no gain in security as the proxy can only use the same verification method as the client. * As another thread participant mentioned, if you don't trust a global passive adversary, then it may be sensible to question whether you can trust their certificate issuers (I admit that your HPKP comments partially address this concern). If you don't trust either, you might choose to save some CPU cycles (both for yourself and those who may be gathering your data). The reason we're even having this debate is that the use of TLS is of little consequence from an integrity point of view, because we're doing our own verification with our own PKI that is independent from TLS certificate issuers, and that isn't going to change. Because the use of TLS is inconsequential, it is a trade-off between various weak reasons: 1. unencrypted HTTP can be cached with a caching proxy 2. unencrypted HTTP can be redirected to a local mirror 3. encrypted HTTPS does not let a listener determine quite as easily what software you are running on your machine 4. encrypted HTTPS requires CA certificates to be deployed to clients and marked as trusted, or apt will fail. Whether a caching proxy and/or redirection to an unofficial nearest mirror are desirable or undesirable also depends on your stance -- my Docker machines download the same packages sometimes thirty times a day, so being able to redirect deb.debian.org to a local mirror is a massive performance boost, but also means sometimes images are built with stale packages. To everyday users in developed countries, this change is completely irrelevant: they get a default configuration with all CAs trusted either way, they get packages from a content delivery network that has HTTPS acceleration so that is not a bottleneck, and modern CPUs have cryptographic primitives so this uses less CPU time than decompression. I've raised all of these points in a previous debate already, so I believe they have been taken into account in the current decision. No users in developing countries have chimed in, so it does not seem to be a priority there either, and the Docker users can help themselves with a sed invocation as part of the image build. So far, I'm not seeing a reason to restart this debate; for that to happen, we'd need to find at least someone for whom that change is more than a slight inconvenience. Simon
Re: Using i386 by mistake on 64-bit hardware [was Re: i386 in the future 32-bit archs: a proposal)]
Hi, On 20.05.23 16:15, Josh Triplett wrote: That might help reduce the number of actual installations of i386 by people who don't realize they could be and should be using amd64. Crossgrades are probably broken with systemd, but it might be possible to hack something that diverts /sbin/init and performs the crossgrade after a reboot, that might also help a few people leave i386 behind. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)
Hi, On 5/18/23 18:08, Luca Boccassi wrote: Without it, leaving them in place makes no difference for usrmerged systems, and allows derived distributions that don't need usrmerge to continue using our packages. Not quite. Having packages only ship files under /usr (and possibly /etc) is very much a goal in itself for a lot of us. My point is: that is not an end goal, because it provides no real tangible benefit on its own. It does make sense in the context of building immutable images, which is *also* a bootstrapping problem, and probably worth being supported with proper tooling. - there is no guarantee that usrmerge will be permanent or the last transition of this kind It is permanent, there are several upstream projects that will drop support for legacy layouts very soon, and it will not be re-added back. You are currently building a "legacy" system, it will just take a bit of time to reach that status. The less you anticipate future needs, the faster this will be. I understand that you are also a member of one of these upstream projects, and that you are taking the interests of this project to "pretty much the last relevant holdout" here. Has it occurred to you that you are also wearing a Debian hat, and you could be taking the interests of the Debian project to said upstream project? - it also solves the bootstrap problem It also is the least likely to succeed, and the most likely to cause significant "social" upheavals. The only social problem I see is that you are trying to create a situation in which other people are compelled to do your work for you if they want it to be done properly. So far, you have been throwing out "solutions", and left the analysis of the feasibility of those to other people, then, after three iterations, you demanded a full write-up of all existing use cases and blanket permission to ignore anything not brought up in this list. The thing is: I see more enthusiasm and self-directed problem solving skills from the interns at the company where I work, and at the same time you are one of the top contributors of the upstream project whose ideas of a "supported" configuration we are supposed to follow. Simon
Re: booststrapping /usr-merged systems (was: Re: DEP 17: Improve support for directory aliasing in dpkg)
Hi, On 5/18/23 02:15, Sam Hartman wrote: Helmut> I think at this point, we have quite universal consensus Helmut> about the goal of moving files to their canonical location Helmut> (i.e. from / to /usr) as a solution to the aliasing problems Helmut> while we do not have consensus on precisely how to do this Helmut> (i.e. with changing dpkg or without). If you believe that Helmut> this is not consensus, please speak up. I agree we have strong consensus that we want to move files to their canonical locations. I'm not entirely sure I'd agree that we have consensus that's our solution to the aliasing problem. It's the other way around: moving the files as a solution to the aliasing problem is the strongest argument in favour of moving the files inside the packages. Without it, leaving them in place makes no difference for usrmerged systems, and allows derived distributions that don't need usrmerge to continue using our packages. If for example we accomplish the move to canonical locations by changing dpkg, we might well get some form of aliasing support in dpkg. IMO, that is still the preferred solution: - it is actually safe, because dpkg knows what is going on and can reject conflicting changes - there is no guarantee that usrmerge will be permanent or the last transition of this kind - it also solves the bootstrap problem Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 5/8/23 20:38, Luca Boccassi wrote: [local diversions] Sure, they are supported in the sense that they can be enabled, and then you get to keep the pieces. They are supported in the sense that someone actually added an explicit flag for dpkg-divert for specifically this feature and documented it in the manual page as an interface. Maintainer scripts don't need to work around the admin installing arbitrary incompatible tools, because we generally expect admins to know what they are doing -- however requiring admins to perform multiple diversion registrations to have them count is a change of the interface. The /bin/sh diversion is a bit ...special. This should have been an alternative, but cannot be because /bin/sh needs to be functional after unpacking Essential packages, so maintainer scripts work during bootstrap. There is an interesting use case here as well, bootstrapping a foreign architecture, where we only unpack the Essential set, divert /sbin/init to /sbin/init.real and place a shell script as /sbin/init that completes the installation once the resulting tree is mounted as a root filesystem on the target platform. It has been a while since I used that, in principle this should even still work with systemd. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 5/7/23 18:14, Ansgar wrote: Is there any specific reason why specifically diversions are a problem where "it might work" is not sufficient? That is, why should we divert from the usual standard for dealing with packages outside the Debian ecosystem here? Locally created diversions are a supported feature, and the only way for admins to replace single files in a way that is safe for installing updates. Even within Debian, it is not sufficient to just coordinate uploads of packages that divert each others' files, because the new diversion needs to be in place before a newly-canonicalized package is unpacked, a Breaks relationship does not enforce that ordering, and while a Conflicts without a Replaces does, this adds a lot of constraints to the solver. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 06.05.23 21:28, Luca Boccassi wrote: [shipping usrmerge symlinks in packages] In the far future I'd like for these details to be owned by image builders/launchers/setup processes rather than a package, but this can be discussed separately and independently, no need to be tied to this effort. Ideally I'd like to have this information in a single package rather than distributed over ten different tools, especially as this is also release and platform dependent. If possible, I'd like to go back to the gold standard of - download Essential: yes packages and their dependencies - unpack them using dpkg --fsys-tarfile | tar -x - install over this directory with dpkg --root=... -i *.deb to get something that works as a container. Right now, that only works if I remove "init" from the package list, which is fair since a container doesn't need an init system anyway. The less an image builder needs to deviate from this approach, the better for our users. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 06.05.23 07:11, Luca Boccassi wrote: - every package is forcefully canonicalized as soon as trixie is open for business You will also need to ship at least - /lib -> usr/lib (on 32 bit) - /lib64 -> usr/lib64 (on 64 bit) as a symlink either in the libc-bin package or any other Essential package, to fulfill the requirement that unpacked Essential packages are operational. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 05.05.23 18:36, Timo Röhling wrote: - it is not an error to register a diversion for an alias of an existing diversion, provided the package and target matches, this is a no-op - it is not an error to unregister a diversion for an alias of a path that has been unregistered previously, that is a no-op as well How do you distinguish between aliased diversions and "real" ones? Because if you allow the registration of duplicate diversions, the following can happen: - Package A is installed, preinst creates a diversion - Package B is installed, preinst creates the same diversion - Package A is uninstalled, postrm removes the diversion That's why the package and target need to match -- so by extension package B may not divert a path that package A already diverts. With dpkg-divert as it is, this works because no aliasing support exists. A diverted dpkg-divert that generates additional diversions would need to catch this case, reject it and restore the previous state, same as an aliasing aware dpkg-divert. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 04.05.23 20:26, Helmut Grohne wrote: From my point of view, the ultimate goal here should be moving all files to their canonical location and thereby make aliasing effects irrelevant. Do you confirm? Yes, that would solve the problem for the current transition without any changes in dpkg. There will be some small aliasing effects still, e.g. ldd will probably output the linker path from the ELF ABI instead of the canonical linker path, but nothing major. What I'm concerned about is that, whether we like it or not, we're still (d|r)efining interfaces here, and that affects future development, including future transitions. As such, I do not see aliasing support in dpkg as complementing the forced file move approach with lots of workarounds such as diverting dpkg-divert. Rather, I see them as exclusive strategies. Yes, absolutely. No one can coordinate two groups that don't even agree on the scope of the work or whether a definition of the scope is even necessary before beginning. Each of these strategies has significant downsides. In combining the different strategies, we combine their downsides, but since their benefit is shared, we do not gain anything in return but increase the price to pay. For this transition, yes, but we need to think beyond that. Any workaround we introduce here will need one release to be rolled out, one release to be stable, and one release to disappear. We can, in many but not all cases, introduce dependencies to enforce a particular ordering and reduce that by one release cycle. We're too late to roll out workarounds in the bookworm cycle, so we can deploy them with trixie at the earliest. Essential packages cannot contain diversions, so we need support from all installers (debootstrap, cdebootstrap, mmdebootstrap) or leave file paths for Essential packages as they are, delaying the end of the transition until forky. Making the workarounds obsolete before that will allow us to finish the transition earlier and thus reduce the effort of future transitions. For the users, nothing changes, they have had the files moved forcefully years ago. At the same time, there is little overlap in the groups that would implement the different strategies, so there is no person who would need to decide what they are working on to the detriment of the other option. On the flip side, if dpkg (and thus dpkg-divert) is to gain aliasing support, I see no reason (and benefit) to diverting dpkg-divert. Me neither. The problem is, we have no commitment from anyone to implement aliasing support. Volunteering Guillem hasn't worked, I don't really have time to do more than "best effort", and I don't know many other people even looking into that, so we cannot decide on that strategy. Can you explain why you see combining these strategies as something worth exploring? It will get something done on the way to finishing the transition until forky, even if dpkg changes don't manifest, but can provide a solution in trixie if they do. then a package containing /bin/foo and a package containing /usr/bin/foo now have a file conflict in dpkg. Not sure if that is a problem, or exactly the This case already is prohibited by policy section 10.1. It can only happen as a temporary state during a file move (from / to /usr and from one package to another). Yes, but from a technical point of view we cannot rely on Policy, it only allows us to assign blame, but that will not fix the error message a user is experiencing. With the status quo, dpkg will not detect the file conflict, which may or may not be the desired result, but it will not flag an error during execution. With two file names being diverted to the same name, it is likely (again, I haven't tested) that installation of the conflicting package will be aborted unless there is a Replaces: in effect, which is something that whatever tool is calling dpkg will need to deal with. behaviour we want. Probably the latter, which would allow us to define a policy "if aliased paths are diverted, the diversion needs to match", which in turn would allow the conflict checker during alias registration to verify that the aliased diversions are not in conflict. If we do not modify dpkg to improve aliasing support, then yes, such a scenario will require a Conflicts declaration or a different measure averting this problem. No, that would require the package declaring the diversion to conflict with itself, that makes no sense. We need to clarify the interface for dpkg-divert here (i.e. create Policy), and that interface will need to be supported by both the diverted dpkg-divert and the one shipped with dpkg. As of now, the one in dpkg will happily register diversions for aliased paths, but to my knowledge, none have been registered, and the diverted dpkg-divert does not exist yet, so we have full freedom to define what the policy should be. My proposal would be to put the onus on
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 03.05.23 19:19, Helmut Grohne wrote: What still applies here is that we can have usr-is-merged divert /usr/bin/dpkg-divert and have it automatically duplicate any possibly aliased diversion and then the diverter may Pre-Depends: usr-is-merged (>=...) to have its diversions duplicated. Of course, doing so will make usr-is-merged very hard to remove, but we have experience here from multiarch-support. For aliasing support in dpkg, that means we need a safe policy of dealing with diversions that conflict through aliasing that isn't "reject with error", because the magic dpkg-divert would always generate conflicts. One thing we need to check is whether diversions to the same target cause file conflicts -- I think they should. So if you divert /bin/foo -> /usr/bin/foo.dontdelete /usr/bin/foo -> /usr/bin/foo.dontdelete then a package containing /bin/foo and a package containing /usr/bin/foo now have a file conflict in dpkg. Not sure if that is a problem, or exactly the behaviour we want. Probably the latter, which would allow us to define a policy "if aliased paths are diverted, the diversion needs to match", which in turn would allow the conflict checker during alias registration to verify that the aliased diversions are not in conflict. The diverted dpkg-divert would probably generate extra register/unregister calls as soon as dpkg-divert itself is aliasing aware, but all that does is generate warning messages about existing diversions being added again, or nonexistent diversions being deleted -- these are harmless anyway, because maintainer scripts are supposed to be idempotent, and dpkg-divert supports that by not requiring scripts to check before they register/unregister. And of course, we can always draw the diversion card and have usr-is-merged divert /usr/bin/update-alternatives to have it canonicalize paths as required to be able to migrate alternatives in a sane way (from a consumer point of view). We get to draw this card exactly once, and any package that would need the same diversion would need to conflict with usr-is-merged, which would make it basically uninstallable. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 30.04.23 03:08, Marvin Renich wrote: My understanding from following this thread (and others) is that dpkg has a bug that can easily be triggered by a sysadmin replacing a directory with a symlink (and some other necessary conditions that don't happen very often), which is explicitly allowed by policy. No, it becomes "easily" triggered because usrmerge aims to move the files in the packages, which would cause /bin to become empty at some point in the future, at which point dpkg correctly needs to remove it. As long as /bin/ls exists in coreutils, the /bin -> usr/bin symlink will be safe from deletion. The --add-alias solution that has been suggested in this thread seems like it would fix the general problem iff policy was changed to require sysadmins to use it if they replaced a directory with a symlink. Yes, we want something that is basically --add-alias that returns us to a state where dpkg's model is an accurate representation of reality. I do not understand why the dpkg maintainer has rejected this solution; I've understood this to be mostly an issue with the quality of the patches submitted, not the general idea. Dpkg has a really extensive test suite, I have not so far seen any patch being submitted that even adds test cases there. My branch adds six, that is still ridiculously low. it would still be a fix for the general bug after the usrmerge transition has completed. And it would be at least one order of magnitude more performant than scanning the filesystem for directory symlinks. That, and as far as I can see, fixing the general problem will also be a prerequisite to finishing the transition in the first place because there are lots of subtle little things caused by the model mismatch, and the workarounds solve the problem at hand but further entrench the mismatch. Simon PS: Current state of my branch: Selecting previously unselected package pkg-usrmerge. (Reading database ... 6 files and directories currently installed.) Preparing to unpack pkg-usrmerge.deb ... D01: process_archive oldversionstatus=not installed /home/geier/dpkg/tests/t-file-conflicts-usrmerge/../dpkgdb/tmp.ci/preinst install D01: ensure_canonical: new, (re)loading D01: ensure_canonical: found mapping /bin -> /usr/bin D01: ensure_canonical: mapping /bin -> /usr/bin seems to be new D01: create_alias: alias /bin/test-conflict -> /usr/bin/test-conflict conflicts with pkg-b(0) OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Thu, Apr 27, 2023 at 12:34:06AM +0200, Helmut Grohne wrote: > Ok, let's move on. I've proposed diversions as a cure, but in reality > diversions are a problem themselves. Consider that > cryptsetup-nuke-password diverts /lib/cryptsetup/askpass, which is > usually owned by cryptsetup. If cryptsetup were to move that file to > /usr, the diversion would not cover it anymore and the actual content of > askpass would depend on the unpack order. That's very bad and none of > what I proposed earlier is going to fix this. Another question: how would this interact with a patch that teaches dpkg to do aliases properly, because such a patch would affect diversion handling as well -- specifically, aliases mean that diversions are also aliased (I believe my patch implicitly does this right, but I think I just got 60 more testcases to write), and that diversion targets are implicitly redirected to a resolved form (I don't do that yet, but it's simple to add to my patch). I think the "divert, but do not rename" approach itself should be fairly safe, because all it does is make a deletion fail. Registering the diversion to the new package should be sufficient to make sure the newly unpacked file is not diverted. This probably needs some additional undo operations for failed installs/upgrades, so the diversion is properly removed in these cases (there is no guarantee that postinst will be called after preinst, we could also end up in postrm). > So how do we fix diversions? Let's have a look into the dpkg toolbox > again. I've got an idea. Diversions. What you say? How do you fix > diversions with diversions? Quite obviously, you divert > /usr/bin/dpkg-divert! And whenever dpkg-divert is instructed to add a > diversion for a non-canonical path, you forward that call to the real > dpkg-divert, but also call it with a canonicalized version such that > both locations are covered. When initially deploying the diversion of > /usr/bin/dpkg-divert, we also need to transform existing diversions. Ouch, if you deploy that, I will definitely need to add diversion merging code to alias registration. That's another 60 testcases, but we need defined behaviour for that anyway. Transforming existing diversions: yes, if you can find out about them without looking at dpkg internal files. It may very well be necessary to update the file format on one of these, and if that would cause your script to create nonsense diversions, that'd be another thing we'd have to work around while building real aliasing support. My current mood is "I'd rather focus on a proper solution, not another hack that needs to be supported by the proper solution." Anything we build here that is not aliasing support for dpkg, but another "shortcut" will delay aliasing support for dpkg because it adds more possible code paths that all need to be tested. Keep in mind that we're here because someone took a shortcut, after all. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Thu, Apr 27, 2023 at 11:57:58AM +, Holger Levsen wrote: > Constitution 2.1.1 is great, however we don't really have a mechanism how to > deal with people flat out ignoring Constitution 6 aka the tech-ctte and > doubting > and activly working against it's decisions. We have: we can find a new maintainer and transfer the package. For that to happen, someone would have to step up and volunteer to either develop the necessary functionality, or accept a patch that does so, and then continue as maintainer. This has not happened. I also doubt it will happen, because anyone capable of maintaining a core system component such as dpkg is aware that nothing implemented so far is of sufficient quality that they want to be responsible for its continued maintenance. The tech-ctte decision mainly adds an additional constraint on proposed solutions: they may not require a temporary rollback (through dpkg-usrunmess or similar), but must accept transitioned and half- transitioned systems as they are, and bring them to a fully-transitioned and consistent state. It's unclear if it also means that dpkg need not provide a way to remove aliases. This adds a moderate amount of additional complexity as we need to add more checks to hot paths in dpkg, and we need to verify that these code paths work. In my opinion, having these paths will add robustness to dpkg, so we want them anyway, so this doesn't add further delay. The core problem remains however: the tech-ctte decision has not made code appear, and the Constitution is also powerless to do so. Simon
Re: Upgrade package from init script to Systemd, move config folder
Hi, On Wed, Apr 26, 2023 at 04:25:19PM +0200, Marc Haber wrote: > I am not sure whether it is doing non-systemd users a favor to keep a > probably outdated, bitrotting and untested init script in the > canonical place. My gut feeling is that it might be better to ship the > old init script in /usr/share/doc/package/examples unless the package > maintainer is reasonably sure that the init script will actually work. No, that is worse, because if an updated init script is shipped as an example only, I will not even get a notification that I might want to change my installed init script. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Thu, Apr 27, 2023 at 12:34:06AM +0200, Helmut Grohne wrote: > At some point the question becomes: Do we want that complexity inside > dpkg (aka DEP 17 or some variant of it) or outside of dpkg (i.e. what > we're talking about here). It seems clear at this time, that complexity > is unavoidable. My gut feeling is that returning to "dpkg's model is an accurate representation of the file system" will be less complex to manage long-term. For this to work, the model needs to be able to express reality, so I guess we can't avoid updating dpkg. I'm also not convinced that the current filesystem layout will remain as it is, for example I can see a use case for installing kernel modules outside of /usr. It would be great to have a generic mechanism here, and be able to do transitions like these without inventing new tools every time. Also, the more we can do in a descriptive framework, the better we can do static checks. The main reason we can argue about what packages are affected is that we have a database of what files are installed where, and that still accurately reflects reality, so we can apply a transformation onto this data and check for conflicts -- but we cannot see diversions in this database as these are created from imperative code. So my fear is that if we create a workaround here that is implemented as imperative code in pre/postinst, this will be invisible to whoever plans the next transition, so this would create immense technical debt. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Tue, Apr 25, 2023 at 09:07:28PM +0200, Helmut Grohne wrote: > This and /bin/sh is the kind of files I'd consider important. And then > upon thinking further it became more and more difficult for me to make > sense of the objection. On a merged system, we can just move that file > to its canonical location without having any trouble even with an > unmodified dpkg. So from my pov, the question about important files can > be disregarded. I hope Simon Richter agrees. Yes, the relevant code at https://github.com/guillemj/dpkg/blob/main/src/main/unpack.c#L749 already handles moving a file inside the same package, and that has existed for some time, that's why I use two packages for the PoC. I have not looked for more issues beyond that, so there might be others lurking in the depths of this code. What I'm mostly concerned about (read: have not verified either way) with /lib/ld.so and /bin/sh is what happens when dpkg learns of /bin and /lib as symlinks -- because right now, the symlinks created by usrmerge are protected by the rule that if dpkg expects a directory and finds a symlink, that is fine because that is obviously an action taken by the admin. But if dpkg sees a package containing these as symlinks, then this is entered into the dpkg database, and subject to conflict resolution, and for that, a separate rule exists that directory-symlink conflicts are resolved in favour of the directory, so the interaction between a newer base-files packages shipping /lib as a symlink and an older or third-party package containing /lib as a directory (e.g. a kernel package from a hosting provider) could overwrite the /lib symlink. It might be possible to avoid that by never shipping /lib as a symlink and always creating it externally, but I still think that's kind of wobbly. > If we look deeper into the dpkg toolbox, we pause at diversions. What if > the new package were to add a (--no-rename) diversion for files that are > at risk of being accidentally deleted in newpkg.preinst and then remove > that diversion in newpkg.postinst? Any such diversion will cause package > removal of the oldpkg to skip removal of the diverted file (and instead > deleted the non-existent path that we diverted to). Thus we retain the > files we were interested in. O_O Yes. Hahahahaha yes. Brittle, but it could work. What is a bit annoying is that we'd have to keep this for an entire cycle. Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On 22.04.23 11:36, Helmut Grohne wrote: For clarity let me propose the following requirements for the definition of done: * All files shipped in Debian packages are shipped in their canonical location rather than an aliased path. With proper support in dpkg, that is even somewhat optional. * The symbolic links are part of some essential package. Yes, but simple symbolic links in packages would lose against directories in other packages due to the symlink-vs-directory rule. Treating these as a conflict in the future instead of silently resolving them is not sufficient for transition handling, and making symlinks win over directories and cause aliases to be created automatically will cause curious and interesting effects if someone makes a mistake in a .links file while packaging. I'd ship the aliases as part of base-files, because that is Essential and has very few reverse dependencies, so giving it a Pre-Depends on a recent dpkg version does not cause ordering issues on upgrade. The Pre-Depends is ignored during bootstrap, but that is fine as the dpkg that is about to be installed is already new enough if it is from the same distribution. In addition, derived distributions that don't want usrmerge will likely ship their own base-files package. * Therefore no special handling code is needed in bootstrapping tools for future releases. For the requirement that unpacking Essential packages gives enough of a running system that installing packages is possible, this unpack phase needs to leave us with a working ld.so in the place the ABI standard requires it, so we probably need to ship the symlinks as symlinks in data.tar.gz, and register them as aliases from metadata so they are not overwritten by a directory. The alternative would be to keep shipping ld.so in /lib, and let dpkg do usrmerge during the phase where the unpacked Essential packages are overwritten using dpkg. We won't really be able to avoid at least some ugliness there, given the interaction between usrmerge and the ABI standard. The alternative would be a consensus that dpkg is simply not expected to always leave the system in a useful state if it encounters certain invalid situations My main reason for disliking it is that it shifts the effort from the proponents to everyone else. The aliasing effects consume so much mental capacity of regular package maintainers Exactly that's what I'm feeling as well, and why I want dpkg to be able to reject invalid packages safely so if a maintainer makes a mistake, this can simply be fixed by an updated package. Testing alone will be an absolute nightmare because we can enter invalid states through multiple avenues, for example, if I have a conflict a.deb: /bin/test b.deb: /usr/bin/test c.deb: /bin -> /usr/bin Arguably, we can rule out a lot of test cases by policy. We already have policy that forbids the combination of a.deb and b.deb in Debian. This property can relatively simply be checked on a distribution level and therefore I think we can entirely skip this case. No, dpkg needs to be able to handle situations forbidden in Policy. File conflicts without declaring "Conflicts:" are also forbidden in Policy, but detecting these is one of the main functions of dpkg. # move file to /usr, install symlink, then remove symlink, move back dpkg -i a.deb c.deb dpkg --remove c.deb Again, I think c.deb would likely be essential and since removal of essential packages is undefined, we can remove such cases from our test matrix. Essential packages can become non-Essential in a later release, can change or be replaced, including piecewise. If we ship the symlinks inside the data.tar, then at least the latter is very unlikely, but I'd think that there needs to be a way to remove aliases. The only thing we could argue that this can be delayed for a release or two, but this would introduce additional error paths into package removal, and possibly create situations that the package resolver does not understand, so I'd rather not. Simon OpenPGP_signature Description: OpenPGP digital signature
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi,
On 21.04.23 15:03, Raphael Hertzog wrote:
Here you are considering all files, but for the purpose of our issue,
we can restrict ourselves to the directories known by dpkg. We really
only care about directories that have been turned into symlinks (or
packaged symlinks that are pointing to directories). That's a a much lower
number of paths that we would have to check.
Having all paths in the database is cheaper, because doubling the number
of paths multiplies the (average) cost by log_{262144} 2 only, and we do
significantly more lookups than inserts.
The other problem is that we do not know all of these paths, because the
file system has been modified externally without informing dpkg. The
closest thing we can do is scan everything that is supposed to be a
directory.
As an additional complication, dpkg silently resolves
symlink-vs-directory conflicts in favour of the directory (which happens
seldom, but third-party tools sometimes generate broken packages like
that, so it is useful to keep it that way).
Thus this time-consuming operation would be done once, the first
time that the updated dpkg starts and when /var/lib/dpkg/aliases
does not yet exist.
That is still a public interface. :/
In any case, now that you have a database of aliases, you can do the other
modifications to detect conflicting files and avoid file losses.
How does that sound?
Alas, that is the easy part. My branch already implements most of that,
including the logic to trigger a reload after a maintainer script if the
stat information changed (like for diversions).
The proposal I made above is not a real database in the sense that we
don't record what was shipped by the .deb when we installed the files...
it's rather the opposite, it analyzes the system to detect possible
conflicts with dpkg's view of the system.
That is going to be slow, and it changes dpkg's public interface to a
more complex one where our tight loop that handles unpacking files gains
additional error states.
It can be seen as complimentary to it. In any case, I don't see how
implementing metadata tracking would help to solve the problem that we
have today. dpkg would know that all .deb have /bin as a directory and
not as a symlink, and it would be able to conclude that the directory
has been replaced by a symlink by something external, but that's it.
It should still accept that replacement and do its best to work with it.
That means there are two sources of truth: packages and the file system.
We then need a (lowercase) policy how to resolve conflicts between
these, which becomes our public interface, and thus part of (uppercase)
Policy.
I'd also single out the usrmerge transition here. This package operates
in a grey area of Policy where technically a grave bug is warranted
because it manipulates files belonging to other packages without going
through one of the approved interfaces, but since we accidentally
shipped that, we need to deal with it now. That does not mean this is
acceptable, it just wasn't enforced.
To me it would also be acceptable to just hardcode "if usrmerge or
usr-is-merged is installed, take over the known aliases and silently
discard that package", then salt the earth in dak that no package of
this name can ever be shipped again until bookworm+3.
That would be significantly easier than finding a generic solution that
covers all existing use cases.
Simon
Re: DEP 17: Improve support for directory aliasing in dpkg
Hi, On Fri, Apr 21, 2023 at 02:15:54PM +0200, Raphael Hertzog wrote: I'd like to express some disappointment that nobody replied publicly sofar. There were a few replies on the dpkg mailing list. Last year's developer survey concluded that "Debian should complete the merged-/usr transition" was the most important project for Debian [1] (among those proposed in the survey). That's what we are trying to do here and it would be nice to build some sort of consensus on what it means in terms of changes for dpkg. The first thing we need consensus on, IMO, is the definition of "complete". The maintainers of the usrmerge package consider the status quo an acceptable technical solution, so their definition of "complete" is to roll out the change to the remaining users. Other people view the transition as complete when the transition handling code can be removed, which, with the current implementation, is bookworm+2 because that is the first time we could ever have ld.so in /usr/lib. The handling code for bookworm can never be removed from the various bootstrapping tools, because users will rightfully expect to be able to install older versions. I believe cross debootstrap is also currently broken. The problem for dpkg is that there is already a considerable amount of cases it handles. Paths are owned by specific packages, ownership can be transferred, they can be marked as conffiles and diverted. That alone gives us a massive testsuite already, including such elaborate cases as a package providing two files that are subsequently replaced by files in two other packages so no files remain, which automatically makes the first package disappear. Extending these cases to include aliases is a monumental task, and limiting the allowed combinations does not reduce that, it only inverts the expected return code from that test case. We need defined behaviour for what happens if someone tries to register an alias for /etc (not allowed because there are conffiles inside), and we need error handling paths for cases like /lib -> /usr/lib, where creation of the /lib symlink fails and the original /lib needs to be restored before exiting with an error. There are several possible designs here, including a variant that is similar to dpkg-divert, which is called from a maintainer script, safely performs the filesystem transformation and updates dpkg's database, which is then reloaded after the maintainer script finishes. This variant has the drawback that even in future releases we can never move ld.so to /usr in the packages, because this would be safe only if we could guarantee that the alias is registered. We can approximate that by giving ownership of the alias to a package, which allows declaring a dependency, but then the maintainer scripts need to be able to handle transfer of ownership at some point or we're again creating technical debt. Another variant would be a declarative approach, which would, conveniently, work for initial installation for foreign architectures, and permit moving files to /usr eventually (one or two releases later). The drawback there is that we need a defined point when that declaration is read, and at this point the main dpkg binary (which for cross installation is the host binary) is responsible for moving the files over. This also gives strong ownership of the alias to a package, but now it is dpkg that is responsible for ownership transfer and removal of the alias when the last package providing it vanishes. That solution again spans up a massive space of possible edge cases. Either we handle the alias through the normal file handling code, which gives us ownership transfer for almost free, but introduces lots of special cases into that code, or we add separate handling, which introduces the special case where all files in a package providing an alias are gone and the package should be deinstalled -- do we deinstall when the alias is gone as well, or when all regular files are? This is the problem with doing it right: there are so many corner cases that need to be handled, even if the handling consists of an error message about an unsupported configuration and a rollback to the last valid state. The alternative would be a consensus that dpkg is simply not expected to always leave the system in a useful state if it encounters certain invalid situations, and hoping that we will also be able to point to a few million installations where that has not exploded and call it a success, but that would need to be communicated. I know that Guillem (dpkg's maintainer) is generally opposed to the approach that Debian has followed to implement merged-/usr but I have yet to read his concerns on the changes proposed here (besides the fact that they ought to not be needed because we should redo the transition in another way, but that's a ship that has sailed a long time ago...). A sensible solution needs to be able to perform the
Re: Bug#1030279: ITP: hud -- Backend for the Unity/Lomiri HUD
Hi Mike, On 2/2/23 00:14, Mike Gabriel wrote: * Package name: hud Unity HUD is a heads-up-display interface for controlling the behavior of applications as well as Unity via typed-in commands. It provides access to all applications menu via a single central interface, in order to simplify application usage and make menus more accessible. Would it make sense to call the package "ubuntu-unity-hud" then, since it's part of a larger ecosystem, and also not the ecosystem that most people would associate with the name "Unity" these days? Or is that a user-visible commandline tool, which needs a short name for that reason? Simon OpenPGP_0xEBF67A846AABE354.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
