Re: booststrapping /usr-merged systems
Luca Boccassi writes: > On Sun, 11 Jun 2023 at 18:06, Russ Allbery wrote: >> On the other, related topic, I've also been somewhat confused in this >> discussion why it seems like there's a long-term goal to not have any >> Debian package ship the /bin and /lib symlinks. I would assume we >> would keep those symlinks forever, and thus will always be shipping >> them in a package from now on (once we get to a place where it's safe >> to ship them in a package). In the long term, that seems like the most >> efficient way to prevent them from being removed, and also just seems >> obviously correct semantically. > In the long term, there are two camps: those who would like to ship > everything as package content (ie: top level symlinks in data.tar of > some package, probably base-files) and those who would like packages > to exclusively ship files under the vendor trees (/usr and optionally > /etc) and let image builders set up required mount points, symlinks, > and so on. Ah. Well, let me register my (preliminary, rebuttable) opposition to that second strategy in advance to hopefully make it clear well before it becomes an issue that there is no current project consensus to go that route and we need an actual design discussion (hopefully this time with considerably more attention paid to details) before starting down that path, if we do. (Now is not the time for that discussion; please don't try to convince me that I'm wrong at the moment. I'm only asking that people remember that this is not something we have all agreed upon.) > To be clear, despite being in the latter group, I am absolutely fine > with going with the first option for now to finish the transition, if > it's the best and easiest way to do so, and eventually revisit it later. At least at first glance, and while adding the additional rule to not delete /bin, /lib, etc. seems helpful, it looks like the correct approach to me. It also works well with the strategy that *is* a current project goal and that we *have* been working (slowly) towards, namely making dpkg aware of every file on the file system so that it has a complete picture of the resources that it's managing. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: booststrapping /usr-merged systems
Helmut Grohne writes: > Indeed. However, if you actually manage to trigger this, it can be very > surprising. Your hard coded list would also contain /lib32, /libx32 and > /libo32. Then you install some mipsen packages, remove them and wonder > why /libo32 does not go away. piuparts is definitely unhappy at this > point. I am quite sure that this behaviour would break something. What I > am not sure about is whether accepting such breakage is a reasonable > trade-off. Compared to most of the problems we've discussed on these threads, this seems like a very unimportant problem and a very acceptable trade-off. We can just change puiparts to not care. It's not *idea* to have a dangling symlink that points to a nonexistent directory, to be sure, but given that normal UNIX symlink semantics treat that symlink as identical to a nonexistent file/directory unless you go out of your way to treat it as a symlink, I find it hard to imagine what specifically would break, and am therefore much less sure than you are that this would break something. (Other than QA tools like piuparts, which IMO don't count. A failing test case doesn't necessarily mean a real problem; it can mean that the assumptions the test case were written under have changed. One has to look at the specifics to see whether there is a real problem or whether the test case should be changed.) On the other, related topic, I've also been somewhat confused in this discussion why it seems like there's a long-term goal to not have any Debian package ship the /bin and /lib symlinks. I would assume we would keep those symlinks forever, and thus will always be shipping them in a package from now on (once we get to a place where it's safe to ship them in a package). In the long term, that seems like the most efficient way to prevent them from being removed, and also just seems obviously correct semantically. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: 64-bit time_t transition for 32-bit archs: a proposal
Christoph Biedl writes: > My first idea was to check for "Using 32bit time_t in a packed > structure", something a C compiler could warn about - assuming > on-disk/wire format data is handled in C structs, and omitting the > "packed" attribute should have created trouble years ago. I'm not sure that last assumption is a good one. INN writes C structs directly to disk without the packed attribute and has never had problems due to that. (To be clear, this is definitely a bad approach; I didn't know better when I wrote my contribution to the problem, but do now. But it means that looking for packed won't find one case of this that I know exists.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: 64-bit time_t transition for 32-bit archs: a proposal
Russ Allbery writes: > I'm afraid the problem with INN is worse than the history and overview > databases. The on-disk CNFS header contains time_t, so if you're using > CNFS as your spool storage, the whole article spool becomes unreadable. Oh, wait! No, I'm wrong, CNFS actually does something smart and encodes the header in ASCII when writing it to disk. Okay, phew, this isn't going to be nearly as bad as I had thought. I still have to check if some of the more obscure storage formats are affected, but most INN installations will just need to rebuild history and overview. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: 64-bit time_t transition for 32-bit archs: a proposal
Marco d'Itri writes: > I do not think that inn2 is going to be a problem, because it has been > built with --enable-largefiles for quite a few releases (and yes, > switching was a bit traumatic for the 32 bit systems involved). > OTOH, inn will be affected. But inn systems are small enough nowadays > that users can just rebuild the database. I'm afraid the problem with INN is worse than the history and overview databases. The on-disk CNFS header contains time_t, so if you're using CNFS as your spool storage, the whole article spool becomes unreadable. It's essentially exactly the same problem that we had with LFS, and I'm really regretting not switching everything on-disk to uint64_t or the like a long time ago. (But there was never a point after we introduced CNFS when that was going to be easy.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Luca Boccassi writes: > On Wed, 17 May 2023 at 01:05, Russ Allbery wrote: >> I do think the industry is moving away (well, has already moved away) >> from Linux Standards Base pre-compiled C binaries without wrappers like >> snap or flatpak, although there are some very notable exceptions, such >> as Steam's special use cases explained elsewhere on the thread. I >> don't believe the industry is moving away from Go and Rust binaries, so >> this is, I think, a more complicated picture than your description >> indicates. > I know absolutely nothing about Go, and preciously little about Rust, > but my understanding from news article and so on was that both of those > ecosystems pretty openly avoid "traditional" distributions, in favour of > their own self-contained ecosystems. I think apt-cache search will show that's a significant overstatement and quite a lot of Go and Rust code is packaged for Debian. Lots of Debian maintainers put a lot of hard work into that. > So would that use case be affected, even by an hypothetical and at this > point extremely unlikely archive-wide change? If we only modified specific core packages to use a different PT_INTERP, obviously only those packages would be affected and execution of general Go and Rust binaries would not be affected. I suspect none of those core packages today are Go or Rust. (I do think it's likely that we'll have critical bootstrapping packages written in Rust within the next decade, though.) If all Debian packages were built with a different PT_INTERP but the toolchain itself was not modified, then Go and Rust binaries packaged for Debian could not be copied to and run on non-Debian Linux systems (unless it happened to work by accident because those Linux systems were also /usr-merged and thus the non-canonical PT_INTERP also was present on those systems). This would be unfortunate and surprising and I'm sure that it would break use cases that currently work, but it probably wouldn't be catastrophic. Still, it would obviously be a regression. If the toolchain shipped with Debian were modified to build binaries with a different PT_INTERP, then Go and Rust binaries built with that toolchain would not work on other Linux distributions. If every other major Linux distribution were /usr-merged and thus the modified PT_INTERP worked somewhat by accident on most other distributions, this likewise would still break some things for some people, but similar to the previous point it would be a regression but probably wouldn't be catastrophic. It would be more serious than the previous problem, though, since now we're affecting user-built binaries and not just packaged binaries. Many more of those will be built with the intent to copy them to other systems. If changes caused Go and Rust binaries built on Debian to not work on numerous major non-Debian Linux distributions, I would consider that a serious release-critical bug. But that doesn't seem all that likely from this specific change assuming that the modified PT_INTERP would also be present on other major Linux distributions such as Red Hat, Fedora, SuSE, Arch, Gentoo, etc. I am *assuming* those are all /usr-merging, and thus it might be, but I have not tracked this and do not know if that's true. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Jeremy Stanley writes: > Throwing another common one on the heap, similar to the previous Steam > example, Python wheels with compiled extensions are often distributed on > PyPI for a fictional "manylinux" platform which indicates they're > intended to be usable on most GNU/Linux distributions (though in their > case, they compile on ~ancient RHEL versions to achieve that). One way > that manulinux wheels are less like Rust or Go is that they typically > don't use static compilation, but rather vendor in additional > dynamically-linked libs which are unlikely to be present on the target > installations. Yeah, I've not been digging too deeply into cases like this since they wouldn't be affected by the original proposal that started this off (it wouldn't change binaries compiled with normal Debian tools or change what ABI Debian can execute), but if we move into the more general question of "does anyone care about building binaries on one Linux system and running them on another," I think there are quite a lot of examples. Another one just from the Python world is conda, which provides a parallel set of pre-compiled libraries in addition to the Python modules and expects to be able to install and run them on arbitrary systems outside of a chroot or container, just by manipulating the dynamic loader search path. Conda is very widely used in scientific computing, where a lot of use cases require Python libraries with substantial underlying C library components that often do not have compatible or sufficiently optimized versions packaged for the underlying distribution. There are also, obviously, non-free software use cases galore, including I suspect some in the Debian non-free archive, where binaries built on other Linux distributions are being run on Debian systems. I remember, in a previous job, changing a few files in /etc to convince the Oracle installer that a Debian system was actually a Red Hat system so that it would run, and the resulting binaries then worked fine (and it would have been a huge problem for us if they hadn't). I think Oracle subsequently added more support for Debian, or at least Ubuntu, but I'm sure there are other cases like that still around today. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: 64-bit time_t transition for 32-bit archs: a proposal
Steve Langasek writes: > * Largely via NMU, add a “t64” suffix to the name of runtime library > packages whose ABI changes on rebuild with the above flags. If an > affected library already has a different suffix (c102, c2, ldbl, g…), drop > it at this time. This is possibly me being too fiddly (and also I'm not 100% sure that I'll get to this in time), but ideally I'd like to do an upstream SONAME transition for one of my shared libraries (and probably will go ahead and change it for i386 as well, since I'm dubious of the need to run old binaries with new libraries in this specific case). What's the best way for me to do that such that I won't interfere with the more automated general transition? Will you somehow automatically detect packages that have already been transitioned? Or should I wait until the package has been automatically transitioned and *then* do an upstream transition? > Your thoughts? The one additional wrinkle is that there are packages that, due to historical error or unfortunate design choices, have on-disk data files that also encode the width of time_t. (I know of inn2, which is partly my fault, but presumably there are more.) Rebuilding that package with the 64-bit time_t flags would essentially introduce an RC bug (data loss) because it will treat its existing data files as corrupt. Do you have any idea how to deal with this case? (The LFS transition was kind of a mess and essentially required users to do manual data migration. This time around, maybe we'll manage to write a conversion program in time.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Luca Boccassi writes: > It does say something interesting. When we started, the assertion was > that packages not relying on the symlink being present was fundamental > for portability and cross-compatibility. Then, it shrinked to > portability and cross-compatibility of a subset of packages. Now it > further shrank to portability and cross-compatibility of a subset of > packages of a subset of vintages. I think it's pretty clear that I'm totally failing to communicate what I was getting at with this example and the process is becoming incredibly frustrating, so I'm going to stop trying here. > But does it really work, or do we just hope it does? I personally see a > great deal of signals that say it doesn't. For one, in 2023, "build one > binary and run it everywhere" doesn't really seem to me to be the > principal choice in terms of portability. Well, believe what you believe, but I literally do that daily, as does anyone else who regularly uses software from a Rust or Go ecosystem. Not a single work day goes by without me running, on some random Ubuntu or Red Hat or Debian system, binaries that were compiled on some random other Linux distribution (often I have no idea which one). > It might have been once, and there might be a number of relevant use > cases still, but it seems to me the industry has largely gone in a very > different direction nowadays. I do think the industry is moving away (well, has already moved away) from Linux Standards Base pre-compiled C binaries without wrappers like snap or flatpak, although there are some very notable exceptions, such as Steam's special use cases explained elsewhere on the thread. I don't believe the industry is moving away from Go and Rust binaries, so this is, I think, a more complicated picture than your description indicates. > Not everything of course, but wouldn't it be worth it to specify core > items such as cross-compatibility requirements? Not in terms of > implementation details, but in terms of what the goals and the minimum > expectations are. There are obviously some, and they are obviously of > great importance to many, and yet to me it feels like they shift every > time I ask a different question. I am certainly never going to say no to more maintained documentation. That would be great. If you're feeling motivated and you're willing to really listen to what people are saying and accept and understand a lot of ambiguity and a lot of assumptions that don't match your assumptions, go for it! I personally am not even doing justice to Policy in its existing scope, so this isn't something I will personally be tackling. Honestly, I should have spent all of the time I spent on this thread working on Policy instead. :) > But if you prefer to focus on first principles, that's great! I've been > asking this a lot: is cross-distribution harmonization a core > requirement? Is it important enough to trump other goals and > distro-local benefits? If so, isn't it worth to discuss it and then > pencil it down? I think some parts of it are a core requirement. It would be very surprising, and very bad, if we couldn't run Go and Rust binaries built on another distribution, for example, or if Go or Rust binaries built on Debian couldn't be used on other distributions, both subject to the normal constraints of glibc cross-compatibility that everyone building binaries already knows about. I think other parts of it are not a core requirement, but still something that is nice to have and that we shouldn't break without a really good reason. I think the specific details of the Linux Standards Base process mostly didn't turn into something the Linux world wanted to support going forward, and thus LSB harmonization, while an interesting idea, is no longer a requirement in general. However, we still follow some pieces of it that were properly implemented (like the FHS), and while we shouldn't do that blindly forever (if for no other reason than the FHS is no longer maintained), it's also valuable to not change that too fast and to only break compatibility with now-widely-expected file system layout properties when we have a really good reason. Ideally, we would pick some smaller subset of the LSB that still matters and agree with other major distributions on some points of compatibility to, at the very least, help ease the common problem of needing to administer multiple systems running different Linux distributions. There is no one answer to whether this trumps other goals and distro-local benefits. It depends on what those benefits are and what those goals are and how important they are. For Guix, obviously their immutable tree and hard dependency pinning are more important to them than cross-distro compatibility, and given their goals, that seems like an entirely reasonable decision. I would disagree vehemently with that decision for Debian because D
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Didier 'OdyX' Raboud writes: > This has existed in a (now distant) past as the "Linux Distribution > Checker", in the context of the Linux Standard Base, that Debian and > Ubuntu stopped caring about in late 2015. Ah, yes, thank you, that makes sense. > I'm not aware of more recent efforts in that direction; but it's an > understatement to say the landscape has changed quite a bit since: > containers, sandbox environments (and others) have forever changed the > way we think about distributing binary executable. LSB had that > ambition, and failed. While that is certainly true, I feel like the pendulum may be swinging back in a slightly different way with Go and Rust popularizing the idea that you should be able to copy around a binary and run it on any Linux system with a compatible architecture. This is a much smaller problem than LSB was trying to solve since LSB was trying to standardize things like the shared library ABI and SONAMEs, which Go and Rust intentionally avoid with static linking. But they do rely very deeply on every system being able to execute binaries built to the Linux ABI and glibc. (I realize that's a different question than the one discussed in this thread.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
James Addison writes: > We've almost certainly all encountered limitations in upstream > specifications and wondered when it's worth attempting a perceived > improvement despite potential friction. > If Debian did want/need to change the PT_INTERP path, is there a way to > achieve that in both a standards-compliant and also a distro-compatible > way? My recollection is that we considered this when starting with multiarch but none of the other distributions outside of Ubuntu were very enthused, so we dropped it. I saw those discussions on the glibc development lists, which is not really the place for detailed x86_64 ABI discussion but which would certainly be an interested party and has roughly the right people present. If I saw a problem that would need to be addressed with an ABI change and didn't have anyone else to ask, that's where I personally would start, but the Debian gcc and glibc maintainers would probably know more. The bar for justifying a change will be fairly high, based on past discussions. I would expect it to require some sort of significant bug that can't be worked around in another way. > Although tests may not exist publicly in this case, the idea of using > tests where possible to catch regressions seems relevant to me. Tests > can help people to identify when a compatibility boundary has been > reached or overstepped, and, socially, they can provide a clear place to > gather discussion if & when they become outdated (particularly if the > tests are themselves are provided free and open source). Copying > binaries and running them seems like a form of testing, but perhaps we > could find better ways. I don't know if anyone has written an ABI compliance test for binaries. That sounds like something that would be in scope for the Linux Test Project, though, and it's possible their existing tests do some of this. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: What *is* the amd64 ABI
Sam Hartman writes: > We've all been talking about the x86_64 ABI, and I'm realizing I'm never > read it. > Are we talking about > https://refspecs.linuxbase.org/elf/x86_64-abi-0.99.pdf That's the one that I was looking at, at least. Specifically (for the purposes of the current discussion) page 84. I'm not sure if there's a newer version, since that one is labeled draft. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
I'm dropping the TC bug from this thread, since I don't think it has anything to do with that discussion at this point. I probably should also change the Subject line, but I'm keeping it to make it easier for the people who want to tune out this thread, since I very much doubt they want to tune back in at this point. Luca Boccassi writes: > On Mon, 15 May 2023 at 16:18, Russ Allbery wrote: >> Note that we're not talking about complicated packages with lots of >> runtime like, say, Emacs. As I understand it your proposal wouldn't >> change PT_INTERP for that binary anyway. We're presumably talking >> about the kind of binaries that you need to bootstrap a minimal system, >> so packages like coreutils or bash. And I would indeed expect those >> binaries to be generally portable, as long as the same critical shared >> libraries are available on other systems (in this case, PCRE2 and >> ncurses). > Is that really the case? Let's test that hypothesis: I think you may have not read my whole message before replying, and also please assume that I know really basic facts about glibc compatibility and am not referring to that. I said "of a similar vintage" (farther down in my message) because of course we all know that binaries built against newer versions of glibc don't run on systems with older versions of glibc (and likewise for shared libraries in general and thus libselinux), and you tested a Debian unstable package on an Ubuntu system from 2020. This says nothing interesting and has nothing to do with my point. > Whoops. Does this make coreutils rc-buggy now? ;-) You are the only person who is talking about RC bugs. The bar here is not "prove to me that this is RC buggy," the bar is "you have to prove to a bunch of Debian core maintainers that they should break the ABI in their packages" (not to mention the additional small but permanent build complexity). Demanding they prove to you that it's a bad idea is not how this works. The point of standards like an ABI is that a bunch of entirely unrelated people who never talk to each other and never look at each other's software are allowed to rely on them and assume no one else will break them. This is how free software scales; without invariants that everyone can rely on without having to explain how they're relying on them, it is much more difficult to get an ecosystem to work together. We don't just break things because they don't seem important; the space of people who may be relying on this standard is unknowable, which is the entire point. Opening those boxes is really expensive (in time, planning, communication, debugging, and yes, compatibility) and we should only do it when it really, really matters. > It did look like a veto to me. More importantly, isn't relying on > passersby to spot alleged harmful changes dangerous, especially for > undocumented, uncodified and untested use cases, like unspecified and > vague cross-compatibility requirements? I'm honestly boggled. This is a thread on debian-devel, which is literally how we do architecture vetting in this project. I absolutely do not think that we can write down everything of importance in designing a distribution so that we don't have to have conversations with other people in the project who have deep expertise when considering a significant architectural change like changing the PT_INTERP path of core binaries. >> I mostly jumped in because it felt like you and Steve were just yelling >> at each other and I thought I might be able to explain some of where he >> was coming from in a way that may make more sense. > I don't believe I've done any yelling here. I'm using yelling pretty broadly and probably rather inaccurately here. Maybe a better way of putting it is that Steve was yelling and you didn't appear to be listening or understanding why he was yelling and were responding in ways that were guaranteed to make him yell more. You *are* coming across as kind of contemptuous of other people's arguments, but then it's entirely possible that I am as well, so I'm trying to ignore it. > What if "setting the parking brake" is not enough, as the wheels are > already off and rolling downhill, as shown above, because while > everybody was looking at the parking brakes lever somebody ran off with > the bolts that kept the wheels attached? Why is it worth worrying about > compatibility with something that is already not compatible, and it's > not expected to be compatible for almost all other aspects? You can certainly decide to try to make the argument that ABI compatibility between Linux distributions is a lost cause and we should give up and not care about it any more. That is a valid architectural argument. (To be clear, I don't think this is the argument you've been making up to this point, which is about a very limited AB
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Luca Boccassi writes: > On Mon, 15 May 2023 at 02:26, Russ Allbery wrote: >> (Also, no slight on the GUIX folks, but GUIX is not exactly an, uh, >> major player in Linux distributions, and I'm not sure how much they >> care about compatibility with anyone else.) > This is a counter-example to confute the assertion that *everybody* does > the same thing, which has been made multiple times. I'm not sure whether > it's an experiment or not, I mean if you ask their users/developers I > think they'd tell you it's very much production ready, but it is largely > irrelevant: it exists, and that's the only reason why it was mentioned, > as it shows that it is _possible_ to do that and be a working > distribution. Doesn't imply it's automatically desirable, but that was > not the intention. Ah, okay, I'm happy to agree with that point: you can violate the ABI and continue to be a working distribution. (There are a lot of parts of the ABI that if you violated them you would not have a working distribution, but this is not one of them so far as I can tell.) > Not quite: my argument is that binaries from these packages are not > intended and not required to be ran on non-Debian systems, so there's no > incompatibility introduced in the first place - everything still works > where it is supposed to, exactly as it was before. I think we're saying the same thing but quibbling over phrasing. I'd put that as saying that it's fine for the binaries of certain core Debian packages to be incompatible with the ABI because they're not intended to be used outside of Debian. (In other words, I'm talking about incompatibility as a concrete, testable property of a binary, and I think you're talking about incompatibility as a more abstract concept of a distribution.) > No aggression intended whatsoever, sorry if it appeared that way. I am > trying to understand what the rules are. Well, the rule that I'd ideally set is don't break the ABI, even if it's not obvious why breaking the ABI is a bad idea or you can't see any bad consequences that could come from it, unless the reason for breaking the ABI is absolutely central to the mission and purpose of Debian. That said, it's not like we've never shipped a binary in Debian with a different PT_INTERP. (I vaguely remember that some programming language uses PT_INTERP tricks for some sort of private binary scheme? Objective CAML or something? I ran across it once years ago and can't remember the details. Also, IIRC klibc does some sort of PT_INTERP trick in some situations that I don't remember the details of, although I don't think it does that with general binaries.) So I do see your point that you would prefer the rule to be more pragmatic than that. My counterargument is that this proposal seems to mostly be about avoiding having to create a symlink at a critical point in the bootstrap process, and while it's tricky to get the timing right (and therefore kind of annoying), the resulting usable system has to have that symlink anyway (I think there's no disagreement about that). Not following the ABI for core binaries seems like a scary change with unknown consquences to a bunch of core packages to solve what looks like a relatively minor (if admittedly annoying!) problem. Note that the target of PT_INTERP on Debian is *already* a symlink, to /lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 which was the multiarch path that we actually want to use. We're already ensuring compatibility with a symlink and I think we should just keep doing that and not venture into these waters. >> The world does not divide neatly into supported and unsupported use >> cases. There are a lot of things I do to computers that I expect to >> work in some situations but not in others. That includes, say, having >> a Debian chroot on some other OS and running binaries from that chroot >> without going into the chroot. Often that will absolutely not work. >> Sometimes it will work, and it's convenient that it will work for some >> obscure recovery situations or other weird one-off use cases. I've >> also copied files from working systems to broken systems running a >> different distribution before, and there's a list of caveats as long as >> my arm, but sometimes it's a quick fix for something. > Vast, vast majority of binaries from existing packages will already > not work out of the box in that use case though. I'm not sure why you think this is true and it makes me wonder if maybe my intuition about cross-distribution compatibility is wrong. I would expect to be able to copy, say, most (all?) binaries from coreutils from a Debian system to some other distribution and run it (and that's exactly the sort of binary that is useful in this kind of cross-distribution rescue case). Is this not true today? What breaks? Note that we're not talking about complicated p
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
make bootstrapping Debian somewhat more convenient in this case, I am unconvinced that this is a good enough reason to undermine one of the foundations of what makes Linux a collective and fairly mutually compatible ecosystem. I realize it's not necessarily obvious that changing PT_INTERP for some binaries is a big deal, in part because it's not even obvious that it's part of the ABI. That's why people who are familiar with the ABI process are jumping in to say "please don't touch that, this is a big deal to us." -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1035904: dpkg currently warning about merged-usr systems (revisited)
Luca Boccassi writes: > Why would "software compiled on Debian" fail to work in other > environments? Well, there are many reasons actually, people invented > containers/flatpaks/snaps exactly for that reason. But nothing do with > anything discussed here though, as far as I can tell? My understanding is that this specific thread is about a mention that we may want to change PT_INTERP to /usr/lib64/ld-linux-x86-64.so.2 or some similar path. If PT_INTERP points to a file that doesn't exist, the program is obviously not going to run. The Linux x86_64 ABI says it must point to /lib64/ld-linux-x86-64.so.2. If we build binaries that use some other value, then we are not building ABI-compliant binaries and they may not run on other systems. This is the whole point of an ABI. An obvious specific example of such a system would be one that didn't merge /usr and thus only had /lib64/ld-linux-x86-64.so.2 and not any other path, but that's just one obvious example. There may be others; the whole point of an ABI is that you do not change things like this, not even if you can't personally imagine why your change wouldn't be harmful. There's a whole process for changing an ABI that involves everyone else agreeing as well, and unless one goes through that process, the ABI is what it is. Debian not building ABI-compliant binaries would be highly surprising. Incidentally, that remains true even if we only do that in distribution packages. I certainly have copied binaries from a Debian package to other Linux systems before for various reasons and expected them to run. Sure, this might not work for other reasons outside of our control, but that's no reason to be gratuitously incompatible by breaking the ABI, particularly for what seem to be annoyances of our own creation with known workarounds. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: DEP 17: Improve support for directory aliasing in dpkg
Ansgar writes: > On Wed, 2023-05-10 at 13:50 -0700, Russ Allbery wrote: >> Caring about them isn't the same thing as doing everything they want. >> We can both try to make things as smooth for them as possible and still >> make design decisions about Debian that they may disagree with or that >> may make some property they want to maintain difficult or impossible. >> It's the sort of decision we have to make on a case-by-case basis. > Debian going out of its way to tell derivative users to switch back from > merged-/usr to split-/usr is the *opposite* of trying to make things as > smooth for them as possible. Yes, I agree with that part and I think I objected to that at the time. Nonetheless, one bad decision doesn't mean that it is Debian policy that we don't care about derivatives or their users. I think we made a mistake there which is not in alignment with our ideals or our goals. We should try to reverse that mistake, not double down on it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: DEP 17: Improve support for directory aliasing in dpkg
Ansgar writes: > So why do we allow changes that require listing all reverse dependencies > in Breaks then? This is known to be wrong for all non- listed packages, > e.g., all local/vendor/derivative-specific packages. Because it's a balance; we don't want to stop making changes, and never making a backward-compatible change doesn't work, so we do the best we can with the tools we have. However, if someone with an out-of-Debian package tells us that a change breaks it, historically we did add them to Breaks. We just don't have a good way of discovering this. > As far as I understand, we do explicitly *not* care about our > derivatives with regard to merged-/usr as some packages in Debian > recommend users to move *away* from merged-/usr to split-/usr on > derivatives, i.e., to an unsupported fs layout. Caring about them isn't the same thing as doing everything they want. We can both try to make things as smooth for them as possible and still make design decisions about Debian that they may disagree with or that may make some property they want to maintain difficult or impossible. It's the sort of decision we have to make on a case-by-case basis. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: DEP 17: Improve support for directory aliasing in dpkg
Sam Hartman writes: > As someone who has been following this, I support the work Helmut and > Simon Richter have been doing. > I have more confidence in that view than the one Luca is proposing. > I also support Shawn's interpretation that being conservative here is > good. > I think even with my support we have no consensus. However hopefully we > can get a few more people who have been reading the whole thread to > chime in and a consensus will appear. I've also been following this. I appreciate Luca's questioning of the necessity of parts of the approach and looking for simpler solutions; I think that's valuable feedback, and we should avoid assuming that every conceivable edge case is supported in Debian. There are unsupported edge cases in Debian already and likely always will be because distributions are complex. That said, I find Helmut and Simon's analysis to be more persuasive so far. I do think we should try to find a fairly robust solution, because the feature we're trying to support here (smooth upgrades) is a core and defining feature of what makes Debian Debian. That doesn't mean we need to support literally anything someone might have done; that won't be possible. But I think there are going to be enough unanticipated problems that we should try to cover the anticipated problems, and that includes at least the relatively obvious or known outside-of-Debian uses of things like diversions. I would like to stay open to addressing some of those problems via documentation or explicit upgrade instructions where that makes sense. If we have places where there's a choice between writing extremely tricky and complicated code versus providing people with simple instructions for how to locate problematic diversions on their system, remove them before the upgrade, and then put them back afterwards (or accomplish their goal in some other way), we should consider taking the documentation approach instead. But that still requires being able to enumerate at least the most likely problems and understand them. For example, if local system administrators have been deactivating systemd units by diverting them, at first glance I think it would be better to clearly tell them that they should stop doing this and instead use masking rather than writing code to try to ensure this continues working. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: DEP 17: Improve support for directory aliasing in dpkg
Helmut Grohne writes: > On Thu, Apr 27, 2023 at 10:58:46AM +0200, Marc Haber wrote: >> My gut feeling is that we are wasting prescious time of numerous >> skilled Debian Developers to find ugly workarounds to something that >> should be done in dpkg, but isnt being done because one dpkg maintainer >> has decided to not go the way the project has decided to go. > I find this mail of yours very disappointing and possibly even failing > our Code of Conduct on multiple accounts. I am unhappy to see the Code of Conduct used in this way. Marc's message was not a personal attack. It did not assume bad faith, or indeed make any statements about motives at all. He expressed his opinion about project priorities and put it in the context of his personal judgment of the facts of the situation as he sees them. You may disagree with his summary of facts, or his opinion about or evaluation of the current situation, or even the usefulness of him raising this point due to lack of resources. It is certainly appropriate to raise those disagreements in response, or even to ignore the message if you don't think it's a constructive line of discussion. (In particular, I think Marc assumes that a solution in dpkg would be more straightforward, something that I think is debatable on technical grounds.) But to say that this is possibly a violation of the Code of Conduct is to say that this message doesn't meet the bar for civil discussion on our lists, and I think it is unreasonable to expect anyone to be more civil or even-handed than Marc was in his summary of behavior that he strongly disagrees with. (And, to state the obvious, I don't believe that message was a violation of our Code of Conduct.) Trying to set the bar higher than this would have the effect of forbidding particular types of hard conversations, which is not healthy for the project. We have to be able to talk about interpersonal disagreements and problems of alignment of motives and goals among the people working on the project. Sometimes those discussions are going to be uncomfortable, but we can't ignore them and never discuss them because they're uncomfortable. We are a collection of humans working together collaboratively, which means there will be tension and conflict and we have to deal with that, constructively but honestly and forthrightly. Part of working collaboratively with other people is that those people get to criticize how you are doing your work, as long as they do so respectfully and assuming good faith. Sometimes that includes saying that one believes the actions of another developer are causing a misallocation of project resources or time. Whether or not we end up agreeing that is true, this is a valid topic for discussion, and sometimes it is feedback that other developers need to hear so that they can do some introspection and evaluate whether that may indeed be the case. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Dependency for pre-configure script
Antonio Russo writes: > Indeed, trying to install a package with a pre-depends on certbot that > has a debconf debian/config file does not have certbot installed by the > time the debconf script starts. This is on testing. See section 8 of the debconf specification: The config-file contains a new element, which I call the configmodule. This is a program that will determine the configuration before the package is unpacked. This means it is usually run before the preinst, and before the package is unpacked! This could definitely be clearer (and easier to find), but yes, this is expected behavior. > My question: is there a way to ensure a package is installed *before* > getting debconf input from the user? Alternatively, is there a way to > run debconf questions during a preinst script? The tripwire package prompts in preinst, so that worked at some point, but it's for an old upgrade so I don't know if it's still working. I think it's more common to prompt in postinst during configure. I know of several packages that do that, although it's not considered ideal if you can avoid it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Unlock LUKS with login/password
Jeremy Stanley writes: > Disk encryption is great (when properly implemented) to protect > sensitive information on your machine from prying eyes if it gets > stolen, but unless you're putting sensitive data in /boot why go to the > added trouble of encrypting it? I think this is the key point: you should not be putting sensitive data in /boot, and this is generally always avoidable (and architecturally better to put it elsewhere). I have put sensitive data in /boot in the past because reasons, so it's not strictly true there is *never* a benefit, but I agree that this wasn't a great architecture and there were better ways to do it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Reducing allowed Vcs for packaging?
Diederik de Haas writes: > Question as I don't know: is that only the package change that gets > uploaded to the Debian archive, or is there also a place where the (git) > history of the changes leading up to a new upload gets stored? dgit maintains a history of every package in Debian. If you use dgit to make your upload, that will include the full Git history of the changes, regardless of where you store your Git repository. If you don't, then it only has the uploads to work with, so each package upload is a new commit and the detailed breakdown of the work between those uploads is not available via dgit. But it does provide at least upload-level granularity as a Git history for every package, even those maintained with no version control system at all, with the option for each package maintainer of opting into providing more. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: DEB_BUILD_OPTIONS=nowerror
Simon McVittie writes: > In a typical build system like Autotools, CMake or Meson, it's going to > be much, much easier for the answer to be yes, because the obvious way > to make linters easy to run is to implement them as a (slightly > specialized) test. I agree for separate linters, but I'm not sure this is true for -Werror. It's an *interesting* way of handling compiler linter flags that would have a lot of advantages, but it's a bit annoying to set up. I think it's more common for projects to decide what compiler flags to use in Autoconf or to have a separate target to build with -Werror enabled (or just enable it unconditionally, alas). > The way I've generally set up lint checks in my recent projects is to > make them a test that usually always passes (with non-fatal warnings > when a problem is detected, like "not ok # TODO" in TAP syntax), and > then have a non-default way to turn those warnings into a test failure, > which I use in upstream CI (but usually not in Debian packaging). A bit of an aside, but I use the Lancaster consensus environment variables for this in all of my projects, which came out of the Perl community but aren't Perl-specific. https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md#environment-variables-for-testing-contexts RELEASE_TESTING and AUTHOR_TESTING should not be run during normal builds, only during the QA process during development and release preparation. (although sometimes I turn on RELEASE_TESTING in the Debian build if I know the chances of it failing are very low). AUTOMATED_TESTING would generally be appropriate for running during a Debian package build. (Usually that is for tests that require a bunch of prerequisites that aren't needed to build the software, only to test it.) Now that we have autopkgtests, I probably should disable RELEASE_TESTING in all cases during the regular package build and enaable it only for autopkgtests, but I haven't done that. It would be great if more upstreams would pick up these conventions. The Perl community already has some tools to deal with them, and I have additional Perl modules in my gnulib-like collection of helpers that I copy into my packages. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Yearless copyrights: what do people think?
Wouter Verhelst writes: > On Wed, Feb 22, 2023 at 08:20:27AM -0800, Russ Allbery wrote: >> Unfortunately, it's often against the upstream license. >> Redistribution and use in source and binary forms, with or without >> modification, are permitted provided that the following conditions >> are met: >> 1. Redistributions of source code must retain the above copyright >>notice, this list of conditions and the following disclaimer. >> and: >> The above copyright notice and this permission notice shall be >> included in all copies or substantial portions of the Software. > It says you need to do that, yes. It does not say *where* that copyrigh > statement must be. While this is true, we don't put the copyright statements anywhere else in binary packages. I think arguing that having them in the source package is a stretch for those licenses, since they don't give any special significance to source distributions and the normal way of using the archive is to download the binary package without the source. The Expat license specifically says "all copies"; it doesn't say that if you distribute a few different forms of the software, you can leave the copyright notice out of some of them. I agree that we would satisfy the license if we had a separate file in the binary package that collected all the copyright notices, but we don't; that's the copyright file. All that said, I think the chances that anyone would care enough about this to sue is fairly low. But not zero -- there do exist bad-faith copyright holders who are looking for excuses to sue (although they're thankfully quite rare in free software). -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Yearless copyrights: what do people think?
Paul Wise writes: > On Wed, 2023-02-22 at 09:47 -0800, Russ Allbery wrote: >> People doing this should be aware that you're probably waiving certain >> damage provisions in US copyright law should you ever sue someone in >> the US for violating the license on the Debian packaging, at least so >> far as I understand US copyright law. (I am not a lawyer and this is >> not legal advice for your specific situation.) > Could you give some more details about this? I hadn't heard of it yet. https://www.law.cornell.edu/uscode/text/17/401: (d) Evidentiary Weight of Notice.— If a notice of copyright in the form and position specified by this section appears on the published copy or copies to which a defendant in a copyright infringement suit had access, then no weight shall be given to such a defendant’s interposition of a defense based on innocent infringement in mitigation of actual or statutory damages, except as provided in the last sentence of section 504(c)(2). Also see https://www.copyright.gov/circs/circ03.pdf, specifically "Advantages to Using a Copyright Notice" starting on page 3. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Yearless copyrights: what do people think?
Not an ftp team member or anything, so this is just my personal opinion. Peter Pentchev writes: > 1. Does the Debian Project consider it okay for an upstream package to >include one or more (or all) files with clear license grants (either >included or referred to in the files), and with clear copyright >notices (again, either in the files or in a central file) that >contain the authors' names, but no years? Does such a package >comply with the DFSG? I believe the answer ought to be "yes", but >I thought it wouldn't hurt to ask. Yes, I can't see any reason why this would be a problem. Copyright notices are optional. I suppose it's conceivable someone could put wording in a license that requires the years, but I've never seen something like that unless one takes an extremely aggressive interpretation of "copyright notice" that I wouldn't take. > 2. If an upstream project does that, the debian/copyright file should >reflect that and have a `Files: *` (or whatever pattern) notice that >has a copyright notice without any years... right? And if an upstream >project does that between releases, the debian/copyright file should >change (drop the years) in the next "new upstream release" upload... >right? Yes, that seems to logically follow. For licenses like BSD and Expat where including the copyright notices is a license condition, we should just copy the license notices that upstream uses (I believe it's fine to consolidate years), so if there are no years, we shouldn't put in years. > 3. Now, what about the `Files: debian/*` section of the debian/copyright >file? The common wisdom seems to be that, if only to make it easier to >submit patches to the upstream project, the debian/* files ought to be >licensed under the same terms as the upstream source. Now I know that >licensing and copyright are different things :) So would the Debian >Project consider it okay for a Debian package to have >a `Files: debian/*` section in its copyright file that does not >mention any years? This question is both from a DFSG point of view and >from a "what would be best for our users" one. And does the answer >depend on whether the upstream project's copyright notices include >years or not? (as in, should we follow upstream's lead in that, too) I think it's fine to omit the year from copyright notices in debian/*. It certainly seems clear to me that it's fine from a DFSG perspective; a lot of packages don't even have any separate stanza or copyright notices for debian/* at all and copyright notices are optional. (Not saying this is ideal, just that it's not a DFSG violation.) Sam made the point that including the year communicates when the Debian packaging would enter the public domain. This is true, but I can't bring myself to care that much about it since (sadly in my opinion) that point is so far into the future that I'm dubious of the effort to reward ratio of curating years for all those decades. Not to mention that the debian/* packaging is continuously updated so additional copyright may attach and that gets into a murky mess in terms of copyright expiration, which decreases the value of that information. People doing this should be aware that you're probably waiving certain damage provisions in US copyright law should you ever sue someone in the US for violating the license on the Debian packaging, at least so far as I understand US copyright law. (I am not a lawyer and this is not legal advice for your specific situation.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Yearless copyrights: what do people think?
Daniel Baumann writes: > On 2/22/23 14:26, Peter Pentchev wrote: >> Wait, I may have been unclear. I did not mean that I want to omit the >> upstream copyright years *when they are there*. > I know you didn't mean that, nevertheless, it's imho good idea. Unfortunately, it's often against the upstream license. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. and: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. The Apache 2.0 license only requires the copyright notices be preserved in "Source form," so debian/copyright probably doesn't count. (It instead requires inclusion of the NOTICE file, but allows you to distribute it "within the Source form or documentation, if provided along with the Derivative Works," which we probably qualify for.) The GPL doesn't seem to care about the notice in non-source forms. In practice, I doubt anyone will care, and it's of course fine to omit the year from your own copyright notices as long as you realize that means you cannot take advantage of the damage provisions of US copyright law that require you to publish a valid copyright notice (which I suspect no one cares about). But dropping the copyright dates from the upstream notices I think would often technically violate the upstream license depending on its wording. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Configuration files, local changes, and "managed section" markers
Marc Haber writes: > The issue is, however, a lot more complicated than one would might > think, imagine a structured configuration file like a systemd unit or an > icinga or bind or ISC DHCP config file which would need multiple > "managed sections", and the special case of a setting moving from > managed to non-managed in the package or at the local admin's > discretion. I had that sort of configuration file in mind when I was talking about include, but I guess it depends on what sort of settings one expects to be managed. I can think of a few possibilities, all of which include handles okay (complicated configuration that people can selectively override but where we don't want to treat the whole thing as a configuration file, a small Debian-generated group of settings that should be included in a larger conffile, or a bunch of completely static configuration that no one should ever change although that last case is weird). Yes, nested structure is a bit complicated but in the examples you list surely there isn't very much configuration that should be inaccessible to the local admin? Since that's what's implied by having it *not* be in a managed section. > <---rant---> > On the other hand, putting non-changing configuration in /usr reminds > me of the systemd way to handle things, with a complicated combination > of "if an identically named file is in /etc, it overrides the > package-delivered configuration entirely without the local admin > noticing that there was an upstream change" and "put configuration > snippets in a magic place in /etc and it will augment the > package-delivered configuration without the local admin notiting that > the upstram changed in a way that is incompatible with the local > augumentation". This way to handle things was of course invented in > the Red Hat world because rpm's conffile handling is so vastly > inferior to what we have available for 25 years now, and sadly we have > taken this over 1:1 into Debian, not making use of your vastly > superior methods here in favor of being compatible with Red Hat's > worse solution. > Well, I adore this way of configuring things and think it's way better than how Debian has been doing it and I haven't used Red Hat since the late 1990s, so *shrug*. :) But the point wasn't to advocate for that approach specifically, only that if you *do* have configuration that the user is not allowed to change because the package is going to override it, it needs to not be in /etc, because if it's in /etc it's going to be really confusing. I don't care where you put it, but /usr is the logical spot? I think your argument is that maybe you shouldn't have a bunch of configuration the user can't change, and I agree! -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Configuration files, local changes, and "managed section" markers
Christoph Biedl writes: > these days, I found a package in Debian (four-digit popcon count) that > in an upgrade happily removed some some changes I had made to a > configuration file of that package, in /etc/. > My immediate reaction was to consider this a gross violation of the > Debian Policy (10.7.3 "Behaviour"). Upon closer inspection however I > found there are markers in that file that define a "managed section", an > area where the upgrade process wants to do things - local modification > ought to be placed outside of that, they will not be harmed, then. FWIW, > this functionality was implemented upstream. Policy is fairly explicit that you can't do that, but I suspect we do this in other places as well because some very natural patterns for debconf prompting (load default answers from a file, ask some questions, write out a file containing the answers) tend to cause the same thing if someone puts other unexpected settings in the same file. I think the right answer (which as is often the case involves a lot more work) is to break the configuration file into separate parts, one of which is a true configuration file in the Policy definition and the other of which is the settings that are needed by the upstream software but that aren't a configuration file in the Debian sense (and thus aren't user-modifiable), put the latter in /usr, and convince the program to load them both in some way. I'm guessing that we're working around upstream limitations here, but if upstream's configuration file syntax supports some sort of include directive, that's ideal, and if it doesn't, maybe there's some way to add that? If the right fix isn't available, I would be tempted to convert the whole configuration handling to ucf or something similar that has built-in functionality to try to handle cases like this. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Consensus on closing old bugs
Adrian Bunk writes: > I am sometimes getting an email from the BTS that this "typo in the > manpage" bug I reported 20 years ago has just been fixed in the "New > maintainer" upload of a package. > When working on orphaned packages or doing NMUs, it is also often useful > for me to see the amount/age/contents of bugs in a package as an > indication in what state it is. Yes, I completely agree with this. When adopting an old, unloved package, closing a bunch of old bugs (by fixing them, usually) is one of the most satisfying and enjoyable parts of the work. Don't rob new maintainers of that joy by gratuitously closing bugs unless you're analyzing them and making a content-based decision! Also, it's useful to have a bunch of open ancient bugs as an indicator that the package probably could use more love (although this is a bit overloaded with "the package gets lots of love but is so popular that there are a ton of bugs of dubious quality that no one is looking at," which is less useful as an indicator). -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Consensus on closing old bugs
Adrian Bunk writes: > I remember being pretty pissed when in a different open source project > some abuser asked me every 6-12 months whether I can still reproduce the > problem with the latest upstream version, each time I spent several > hours for confirming it, but this abuser never bothered to follow up on > that after I did the work that was requested from me. Somewhere along the line, the free software world, particularly around larger projects, seems to have become convinced that bug triage is a good way to become involved in a project as a volunteer (maybe! true in some circumstances!), and separately has defined bug triage as closing old bugs (sort of! true if they no longer apply!). Neither of these are entirely unreasonable, or put another way both of these are true in specific situations. But they interact very poorly: people who don't know the project or the community very well are taught to tackle "bug triage" as their first project without good instructions, and naturally gravitate towards closing as many bugs as possible because this seems useful. (And, of course, as pointed out earlier in the thread, a lot of projects have now automated this and just autoclose all of their bugs, which is certainly a choice. I personally have given up reporting bugs against anything in the Kubernetes ecosystem since it's pointless; I'm just talking to an autoclose bot, and it's an unsatisfying conversation.) That said, I also agree with something Ian has said in the past about bugs: the function of bugs for volunteer free software projects is primarily as an aid to development and secondarily as type of user-facing documentation (here are things known to not work and possibly some workarounds). If they don't serve either of those purposes, and particularly if they get in the way of improving the software rather than provide a guide for how the software should be improved, they're not worth keeping around solely because a user did encounter that problem at some point. So in short I agree with Holger: it really depends. It's rude to ask someone to do a bunch of work when you have no intention of following through on that work, which happens a lot when new volunteers do bug triage without the skills required to follow up on the responses to that triage. But also if you're never going to work on a bug and you don't think it serves any documentation purpose, it's okay to close it as wontfix and arguably better communication to the bug reporter than leaving it open and creating the illusion that you might fix it some day. There are a lot of really low-quality bugs out there. Not as many in Debian because our users tend to be a bit more sophisticated, but if you've ever watched a reasonably widely-used package in Launchpad for a while, you'll see an endless series of bugs that are clearly some unrelated system problem, a corrupt installation, or just the user completely not understanding what they're doing. The more prominent the package and the larger the unsophisticated user base, the more aggressive you have to be about closing bugs if you want the open bug list to be a useful artifact for guiding development. A bug is a communication channel between the user, the maintainer, and other users. Like most other forms of communication, it's neither inherently good nor inherently bad. It doesn't make sense to keep all bugs any more than it makes sense to respond to every email message. It's a nuanced social judgment. Or one can just use an autoclose bot, I guess, which is basically the equivalent of one of those email autoreplies that says "this mailbox is unattended and no one will ever read your email." :) And, to be honest, if that's the reality of the situation, maybe better to know than not! -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Please, minimize your build chroots
Sebastian Ramacher writes: > As with all mass bug filings, discuss the issue first on debian-devel. > See developer's reference 7.1.1. This discussion could have both > answered the questions whther RC severity is appropriate for this type > of bug and whether it's a bug at all. Historically, we have not treated FTBFS bugs as falling into the category of mass bug filing or requiring this pre-discussion. Various folks have been mass-filing FTBFS bugs near the release freeze for many years now and they generally don't get a debian-devel discussion first. Now that we've identified that there is a significant project disagreement over the definition of build-essential, bugs specifically related to that disagreement may be worth some discussion (which is happening now). But I think that's new information for Santiago, not something that was obvious retroactively. I think it's pretty clear from this discussion that the current Policy wording is not sufficient, and we shouldn't have ambiguous language over something as central as what build dependencies packages need to declare. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: depends-on-obsolete-package lsb-base
Adam Borowski writes: > On Wed, Jan 18, 2023 at 08:19:15AM -0800, Otto Kekäläinen wrote: >> Does somebody know what is going on? >> >> Example: >> E: mariadb-server: depends-on-obsolete-package Depends: lsb-base (>= 3.0-10) > The severity of this warning is grossly overstated. It shouldn't be an > "error" but a "mild warning to fix if you're bored". Unfortunately, Lintian doesn't support different severities for different iterations of the same tag. We currently have to pick one severity for all depends-on-obsolete-package tags. Sometimes it's more important (because we're trying to retire a package for the next release); other times, it's not, such as here. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Does removal of global variables from a library break C ABI?
Scott Talbert writes: > In one of the library packages I maintain (hidapi), upstream removed a > couple of global variables (my .symbols file noticed this). See > abipkgdiff below. > Does this break ABI? My assessment is that it does NOT, but I would > like to confirm. These variables were not declared in a header file, so > I can't see how external user code would have referenced them. It does technically, but if the variables were never declared in a header file, it's equivalent to hiding private functions that were previously exposed by mistake but never prototyped for users. Traditionally, we don't consider that an ABI break worth bumping the soname unless we have some reason to believe that software is using those symbols. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile
Jonas Smedegaard writes: > Quoting Russ Allbery (2022-12-15 17:41:15) >> This is why I don't agree with Jonas: yes, there *are* other ways to >> achieve the same goal, but they're more complicated and harder to >> explain. The user experience of this build profile looks a lot simpler >> and cleaner. > I don't have a firm opinion here, so wonder what you disagree with, > Russ. > For the record, prior to this thread Ian reached out to me privately > where we discussed my work on forking dh-cargo to behave more > "Debian-ish" than what is developed by the Rust team. > Perhaps this is a language thing: When I "question" Ians proposed > approach, I do not imply that I reject it or am against it, only that I > want to better understand it. Oh, I'm sorry, this is probably just a sequence of misunderstandings. I think I misunderstood your message as being somewhat opposed, and then I think you read my "disagree" as stronger than I meant it. All I was trying to say is that I think (based on my very sketchy understanding of build profiles) that this would have a lower bar of entry for building non-Debian packages as a stepping stone to proper Debian packages than pre-downloading the dependencies. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Proposed `cargo-upstream` dpkg-buildpackage etc. build profile
Ian Jackson writes: > I would like to add the following entry to the table of build > profiles in BuildProfileSpec: > Profile anme: `cargo-upstream` > Can cause changes to built binaries, including functional chnges. (Y) > Should not cause changes to set of build debs. (N) > Description: >Obtain Rust dependencies from crates.io, not from Debian; use a >`Cargo.lock` from the source package if there is one. Will usually >cause cargo to make network accesses. Ideally, behaviour of the >resulting programs will be the same, but this is not guaranteed. > This is useful when developing packages which are to be uploaded to > Debian, but whose build dependencies are not yet available in the > targeted suite. I think this is a great idea. In addition to the benefits you list (and less importantly, because there's less direct affect on Debian), having this feature available would make it easier to create a first pass of a package of a Rust tool intended for local use. I often build semi-broken short-cut packages locally of things I need on some system but haven't yet found the time to make them properly ready for Debian, and then gradually finish the work to get them into Debian. It would be very useful to have this support to get over the initial barrier to making a Debian package. I think it would increase the likelihood that I'd eventually do the work to get all the way to an uploadable Debian package, whereas if it's difficult to bootstrap from Cargo, I'm much more likely to let laziness win, just build a static binary, and copy it around. This is why I don't agree with Jonas: yes, there *are* other ways to achieve the same goal, but they're more complicated and harder to explain. The user experience of this build profile looks a lot simpler and cleaner. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Is an autogenerated configure shell script non-editable source
Andreas Tille writes: > For the general case I somehow understand the consensus here on the list > that a missing configure.ac can be considered a bug but the severity > serious is not really rectified. If I understood this correctly I would > reset the severity to important at the beginning of next week. I think that seems fine baesd on my reading of the thread, although since we have now found the configure.ac, I would also stuff that in the Debian package somewhere so that all the pieces are available to someone who has the Debian source package. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Is an autogenerated configure shell script non-editable source
Bart Martens writes: > That file may be available online for this particular software. The > debate is about whether such configure.ac file must be included in the > distributed package for making the package dfsg. And more in general, > about where to draw the line on how easily editable (think: time well > spent) the included source code must be for making the package dfsg. In > my opinion there is no sharp line, and ftpmaster is well positioned for > making fair choices in a +/- uniform way for all packages. And there is > always be room for questioning those choices and allowing the meaning of > dfsg evolve over time. Back to configure.ac, I'd support a choice of > making a missing configure.ac an 'important' bug, and not enough for > rejecting the package as non-dfsg. The general rule of thumb that I've followed in similar situations in the past (PDF files where the original source has been completely lost, for example) is that the underlying purpose of the DFSG source provision and of free software in general is to ensure that the recipient of the software is not at a disadvantage. In other words, the distributor isn't allowed to hold back pieces that make it harder for the recipient to modify the software (within the general definition of "source," of course). Therefore, it matters what the distributor has. If they have the true source used to generate the file but are keeping it secret, then to me that's a violation of the DFSG and we shouldn't participate (even if their actions aren't technically a violation of the license since if they own the copyright they don't need a license). This is creating that disadvantage for the recipient that free software is designed to prevent. But if the original source has been lost, then everyone is on an equal footing, and I don't think there's a DFSG violation. We may not have the true source in the intended sense, but *no one* has the source, and therefore everyone is on an equal footing and has equal ability to modify the software. There is a different wrinkle in this specific case: we may have the source but not the software that was used to generate the file from the source. In this case, it sounds like an old version of Autoconf was used, and we don't package that version of Autoconf any more, so while the source is present in one sense, one can't regenerate the file. I'm not sure the DFSG is the most useful framework for analyzing that situation, since to me it feels more practical than freedom-based. Everyone is still in basically the same situation: the source is available to everyone, but some work may be required to go hunt up the old tool and regenerate the file. (This assumes a case like Autoconf where the old releases of the tool are readily available and not kept secret, but may be hard to get working.) The real problem in this case is less about the DFSG and more about the practical problems of maintaining Debian as a software distribution: if we can't regenerate configure using software in Debian, there are a lot of porting tasks and some bug-fixing tasks that we can't do, and that's a problem for us. But I'm dubious that it's a *software freedom* problem; it's more of a *software maintenance* problem, and thus the bug severity should be based on how much of a problem that is in practice. (I think this is mostly a long-winded way of saying the same thing Marco said.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Fwd: Put a package back
Luna Jernberg writes: > Our system is relying on an older version of this package > libnet-ssleay-perl. > Its left us in a bad position with clients as we are blocked from > releasing a hotfix. Can this package get put back temporarily? > https://deb.debian.org/debian/pool/main/libn/libnet-ssleay-perl/libnet-ssleay-perl_1.85-2+b1_amd64.deb You can download older versions of any Debian package from snapshot.debian.org. See, for example: https://snapshot.debian.org/binary/libnet-ssleay-perl/ You can then manually install that package with dpkg -i or the equivalent, and then pin it with apt-mark hold libnet-ssleay-perl until you're ready to upgrade it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1023778: TMPDIR behaviour in maintainer scripts [was: Re: Bug#1023778: mysql-server-8.0: fails to restart on upgrade with libpam-tmpdir]
Robie Basak writes: > This seems inconsistent to me. Where is the expectation that TMPDIR must > be unset if dropping privileges coming from? Obviously for users of > libpam-tmpdir that's a problem. But in the default case, it's something > that would be entirely reasonable to inherit through a drop of > privileges, for the same reason that I think you find that setting > TMPDIR for maintainer scripts to use would be useful. You were asking what changed from the historical behavior of UNIX. This, right here, is what changed. As you note, historically UNIX uses a world-writable /tmp and assumed basically everything on the system would use it. Once UNIX started caring more seriously about security, we discovered this came with a whole host of security problems. Most of those have been tediously fixed over the years, but the fixes can be quite complex and to this day we continue to see new software that gets it wrong, because it's very easy (particularly when writing standard C) to not use the right combination of file name randomization, O_EXCL flags, permissions, and so forth to mostly make this safe. Accordingly, private /tmp directories have become increasingly popular. There are several different mechanisms to use them; the mechanism isn't very important. But they often use $TMPDIR since that variable is widely supported. Because the security problems with /tmp happen whenever two users with different privileges use the same directory, private /tmp makes just as much sense for root (if not even more sense, since root is at inherently higher risk) as it does for any other user. So it's not surprising to start seeing root users with private /tmp, which will inherently require *something* to change when dropping privileges from root and needing to use a temporary directory. The whole point of private /tmp is that every separate permission domain uses a separate directory for temporary files that's restricted to just that domain. Obviously, private /tmp is optional, but it has very real defense in depth security properties because shared /tmp is a surprisingly tricky security challenge. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Firmware GR result - what happens next?
Michael Biebl writes: > The main difference is, that the renaming caused an error message by > apt, so you knew something needed to be fixed. One could argue that having non-free but not non-free-firmware is sufficiently strange that it would be worth a suppressable apt warning (that you could turn off in apt.conf). I have no idea how easy that would be to implement, though. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Firmware GR result - what happens next?
Shengjing Zhu writes: > On Sun, Oct 2, 2022 at 10:53 PM Steve McIntyre wrote: >> So this is the one bit that I don't think we currently have a good >> answer for. We've never had a specific script to run on upgrades (like >> Ubuntu do), so this kind of potentially breaking change doesn't really >> have an obvious place to be fixed. >> Obviously we'll need to mention this in the release notes for >> bookworm. Should we maybe talk about adding an upgrade helper tool? > For upgrading, people already need to edit their source list to change > the suite name, why would it hurt to add one more manual step to change > the section name? I think the difference is that if you don't update your sources.list to point to the new suite, your system won't upgrade and so the problem will be very obvious. But if you currently have non-free configured but don't add the new firmware section, everything will appear to work but you won't get new firmware, so the problem may go unnoticed. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug email is not getting to me
Steven Robbins writes: > On Sunday, September 25, 2022 4:57:19 P.M. CDT Russ Allbery wrote: >> If someone sends mail from a domain that says all mail from that domain >> will always have good DKIM signatures, and if the signature isn't >> present or doesn't validate the mail should be rejected, and that >> message is forwarded through bugs.debian.org to someone whose mail >> server honors DMARC settings, the mail will be rejected. That's >> because the process of modifying the message in the way that >> bugs.debian.org needs to do (adding the bug number to the Subject >> header, for instance) usually breaks the signature. > So are you effectively confirming this is indeed the DMARC bug [1] filed > in 2014? > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=754809 Yeah, that's my guess. It's a very common problem. The right solution is probably for bugs.debian.org to rewrite all incoming mail to change the From header to a debian.org address (probably the bug address) and add a Reply-To header pointing to the original sender. This is the fix that Mailman mostly uses, and it seems to work (I've had this problem with multiple mailing lists I run and turning on this message mangling has fixed it). But of course someone has to find time to implement this. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug email is not getting to me
Steven Robbins writes: > To re-iterate: mail sent today to my debian address from outside debian > came through OK. Mail sent from bugs.debian.org apparently does not. I > think if there were any issue with the incoming email server (e.g. the > DMARC thing that Andrea Pappacoda referenced) that would affect all > email to me at debian, wouldn't it? No, the annoying thing about the DMARC problem is that its effect depends on the configuration settings of the person sending the email. If someone sends mail from a domain that says all mail from that domain will always have good DKIM signatures, and if the signature isn't present or doesn't validate the mail should be rejected, and that message is forwarded through bugs.debian.org to someone whose mail server honors DMARC settings, the mail will be rejected. That's because the process of modifying the message in the way that bugs.debian.org needs to do (adding the bug number to the Subject header, for instance) usually breaks the signature. If you send mail from a domain that has a more relaxed (or no) DMARC configuration, then your mail will go through fine and you'll not see the problem. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Presenting DPKG_ROOT
Johannes Schauer Marin Rodrigues writes: > in 2016 we filed our first DPKG_ROOT patch #824594 against > base-files. The dpkg version at the time just had included support for > the DPKG_ROOT variable being set for maintainer scripts and we were > excited to try out this new feature for creating foreign architecture > chroots. At the time we thought that no discussion on d-devel was > necessary before filing the bug because we knew only 10 source packages > had to add DPKG_ROOT to their maintainer scripts and because doing so > would not affect any normal installation. [...] Thank you for this excellent write-up! This is exactly the type of fairly obscure Debian lore that, although it only affects a small number of packages, is worth documenting because it can be very difficult to understand otherwise why it's present or to debug problems caused by accidentally breaking it. I would therefore love to see this documented in Policy. The documentation doesn't have to be long, but even though this only affects a small handful of packages used during early bootstrapping, I think we should write it down somewhere official so that we have a record of what we're doing and how it's supposed to work (and what packages need to care about it). If possible, could you write up a brief description along those lines and open a bug against debian-policy with that description? We can then figure out where to put it in the document. Thanks! -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Comments on proposing NEW queue improvement (Re: Current NEW review process saps developer motivation
Scott Kitterman writes: > If I look at a package and determine it's only in New due to a new > binary package name and that means the project has prohibited me from > looking for other issues in the package until some time later when it's > not in New, then I feel pretty precisely like I'm prohibited from doing > something. I think the part that will help this make sense and make it more obvious that such a GR would not do this is to realize that ftpmaster has a special ability not permitted to other members of the project, namely to delay or block the upload of a package in its entirety. I know I'm repeating myself, but to me it's just so central to this debate to remember that preventing packages from entering the archive is *not* how the project normally deals with RC bugs. There are a lot of good reasons for why we don't normally handle RC bugs that way. The debate is precisely over when we want ftpmaster to exercise their special blocking superpowers not available to any other Developer, powers that exist only because of a delegation and are not normal Developer abilities. Saying you cannot exercise that special ability to delay or block the package from entering the archive, something that, for example, I have never done or been able to do in the time that I've worked on Debian, doesn't prohibit you from doing all the things that Debian Developers normally do. You can see the bug and file the bug right away; you don't need to wait! The point isn't to restrict what you look at as a human being doing work; the point is to discuss when you can take a special blocking action rather than take a normal RC bug reporting action. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Comments on proposing NEW queue improvement (Re: Current NEW review process saps developer motivation
Scott Kitterman writes: > Sean Whitton wrote: >> I think we still want the binary package namespace checking? >> I.e., a GR just saying "ftpteam should not do a full >> licensing/copyright check for packages in binNEW". >> Then no software changes are required. > I think that a GR to prohibit developers from looking for bugs is at > least in principle inconsistent with not hiding problems. Saying that a project delegate, acting as a delegate, should not block binNEW uploads for a specific sort of check that's currently mandatory is not at *all* the same thing as prohibiting developers from looking for bugs. It doesn't do that at all. Anyone who does ftpmaster work would still be able to (and encouraged to!) look for and file bugs just like any other developer. If those bugs are RC, they would be treated like any other RC bug. But the project is entitled to override the decisions of a project delegate by GR if it so chooses (constitution 4.1.3), and one of the reasons why the project may decide to do so is if we collectively believe the project delegates have misjudged the trade-offs of making a particular process mandatory on the grounds that it catches some number of RC bugs. The project may, for example, decide that yes, this process catches some RC bugs, but the number of bugs caught are not worth the other impacts of that process, and the RC bugs can be dealt with via other means. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Current NEW review process saps developer motivation
Paul Wise writes: > On Fri, 2022-08-26 at 11:58 +0100, Simon McVittie wrote: >> For instance, take rust-glib-sys, a package of Rust bindings for the >> GLib library, which is one of the libraries whose handling prompted >> this thread. > I feel like the right way to organise this upstream is for GLib itself > to be building the bindings for itself (as Bastian Roucariès suggests). Speaking as an upstream maintainer who historically has maintained language bindings for a package together with that package (remctl), I can tell you that this is a bad idea, I would tell my past self to never go down this path, and that maintaining each language binding as a separate upstream release is a superior technical solution. This is unintuitive! I believed the opposite for many years! But it turns out that maintaining language bindings as part of a single combined source package is an incredibly difficult thing to attempt to do in the upstream build system, and it is getting more difficult by the day. Each language ecosystem has its own build system, and many of them are undergoing rapid development. Each language now also has its own archive to which build artifacts should be uploaded, often including pre-built binary artifacts for major platforms, and doing this with a combined build system from a single source package is quite difficult. All of the tools are oriented around a single package building for a single language, with all of the build system configuration for that language at the top level of the package. It's *possible* to do this. I have for years supported bindings for four languages plus the C source in remctl. But it's hard; I have personally spent multiple hours keeping this working, hours that were not spent on the software itself or even on the build system but just on integration problems between the various build systems mushed together into the same release. There's very little payoff for this work and it's contributed to not integrating some language bindings or other implementations, such as Java. I haven't yet had the time to do this, but I plan on separating remctl into five separate upstream releases for different languages. This will also, ironically, help in Debian, since it will mean that the remctl source package doesn't tie together language transitions for four different languages. remctl doesn't have the specific problem that Simon is pointing out with the definition of what source code is, so it's unaffected by the point of this thread. But I think it's important to note that upstreams do things like this for good reasons, and undoing them may mean that someone spends a bunch of time and effort, on an ongoing basis, working around problems that upstream has wisely avoided and doesn't think should need to exist. It's very hard to find motivation or volunteers for that kind of work! -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1017716: ITP: muon-meson -- Meson-compatible build system
Andrea Pappacoda writes: > Would alternatives really be that bad? What if the current /usr/bin/muon > was moved to /usr/bin/muon-kde, muon-build was installed to > /usr/bin/muon-build and /usr/bin/muon was shared between the two > packages? What issues could it cause? Debian specifically disallows providing the same binary path from multiple packages in cases where the two binaries do not do the same thing with roughly the same arguments [1]. It would create a situation where /usr/bin/muon on different Debian systems can do wildly different things depending on what packages are installed or even on what order in which they're installed. That in turn creates user confusion, can cause weird problems, etc. It's just too surprising for users for the same binary to be entirely different things on different Debian systems with the same major release installed. (It's bad enough when it happens between releases, although in that case sometimes we live with it as the least bad option.) [1] https://www.debian.org/doc/debian-policy/ch-files.html#binaries -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1016130: ITP: asdf -- multiple language runtime version manager
Matt Barry writes: > I'm not really opposed to using asdf-vm if that prevents genuine > ambiguity. Are any of these packages a) packaged for Debian, b) provide > a binary called 'asdf'? python-asdf, cl-asdf, and asdftool are all packaged for Debian. None of them so far as I can tell produce a binary called asdf, but cl-asdf does have an info page named asdf, so there's some potential for confusion there. (cl-asdf and python-asdf narrowly avoid having a conflict over info pages already.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1014908: ITP: gender-guesser -- Guess the gender from first name
Vincent Bernat writes: > On 2022-07-14 17:14, Russ Allbery wrote: >> (Also, due to the limitations and history of naming conventions, the >> software is inherently trying to map into a gender binary, which if one >> is attempting to capture self-identification is likely to be unhelpful >> for many populations, such as ones with lots of people under 30, due to >> not having a way to represent nonbinary people.) > This one does not. It maps a first name to male, female, androgynous, > mostly make, mostly female, or unknown. Oh, is that what "andy" in the output meant? I thought that was some other quirk of the software, but in retrospect I should have figured that out. Thanks for the correction. Androgynous and nonbinary are not really the same thing, but at least the software is trying to incorporate that, to the extent that people's names reflect their gender at all. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1014908: ITP: gender-guesser -- Guess the gender from first name
Edward Betts writes: > I've been writing some code to work out the gender balance of speakers > at a conference. It parses the pentabarf XML of the schedule and feeds > the speaker names to this module. > Here's the results for Debconf 22. > 72 speakers > male 48 66.7% > unknown 16 22.2% > female 45.6% > mostly_male22.8% > andy 11.4% > mostly_female 11.4% I fear this may be an example of statistics that look meaningful but probably aren't because the error bar is much higher than the typical consumer of the statistic intuitively thinks it is. Although maybe that's not a worry in this case since the program itself says that it totally failed to make a guess about a quarter of the time. I don't really have any objections to the package being in the archive; this is certainly something that a lot of people seem to want to do and thus seem to find some utility in doing. But unless one has a higher-quality source of data than just names (preferred pronouns, direct self-identification, etc.), I personally would be worried about attaching the appearance of scientific accuracy (three significant figures!) to data that, depending on the nationalities involved and the strength of naming conventions and other factors, may be only rough guesswork. I know someone who keeps similar statistics as an aid to balancing the range of authors of books he chooses to review, and I see why someone would want to do that. But he tries to use higher-quality data sources than guessing based on names, and that feels like a best practice for that kind of thing to me. (Also, due to the limitations and history of naming conventions, the software is inherently trying to map into a gender binary, which if one is attempting to capture self-identification is likely to be unhelpful for many populations, such as ones with lots of people under 30, due to not having a way to represent nonbinary people.) Anyway, that's just all my personal opinion and I don't think any of that says that the package shouldn't be in the archive. We package all sorts of not-very-useful software and that's totally fine. But I've worked in identity management fields for long enough to have a professional knee-jerk reaction to anyone doing computer analysis of names or trying to record gender in any way other than simply asking people. :) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Bug#1014908: ITP: gender-guesser -- Guess the gender from first name
Steve McIntyre writes: > IMHO there are 2 points to an ITP: > * to save effort in case two people might be working on the same >package > * to invite discussion on debian-devel / elsewhere > If people post an ITP and upload iummediately, then I don't think that > helps on either count. > If the only reason for the ITP is to make lintian quiet then I think > that's a total waste of time - it's following a guideline blindly > without understanding the reason for it. > How do others feel? I feel the same way, although I have for a long time been dubious of the benefit of debian-devel review for ITPs (and, to be honest, the benefit of WNPP in general apart from orphaning, although sometimes ITP and RFP bugs are a convenient central place to document all the reasons why packaging some specific piece of software is really hard), so the whole system feels kind of creaky to me. ITPs do occasionally catch things that really shouldn't be packaged, and we don't have another good mechanism for doing it. But the whole process as we currently follow it feels oddly dated and manual and sometimes like a box-ticking exercise. (It also adds a lot of noise to debian-devel from the perspective of, I suspect, most participants. But we've talked about that aspect of it before, and there was some moderate desire to see the new packages flow by.) Given that new packages as uploaded (a) include nearly all of the information in an ITP in a more structured form, and (b) have to flow through NEW anyway, I do sort of wonder if it would make sense to notify some mailing list of every new source package, extracting similar fields and the top entry of the changelog (which hopefully has some explanation for why the package is being packaged for Debian, and we could encourage people to do that), and then use the time the package sits waiting for NEW review as the window for people to raise concerns. That doesn't address the locking purpose of ITP (avoiding duplicate work). I'm not sure how frequently ITPs are effective at doing that. It feels like the percentage of the total software ecosystem that Debian is packaging is smaller than it used to be (we've grown but free software has grown way faster) and most of the places where I'd expect contention to happen are handled by language packaging teams that probably have their own processes. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Package uploads silently discarded: how to investigate?
Ben Finney writes: > My guess is that this is something to do with an update to the signing > GnuPG key expiry date. I can get into that in a different thread if > needed. The trouble is, I can only guess, because there are no messages > from anything in the Debian archive telling me what went wrong. My recollection is that if the signature on the upload is invalid, we intentionally delete the upload with no notice (because we have no confirmed knowledge of who to notify). It's possible that my information is dated, though.... -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: popularity-contest: support for XB-Popcon-Reports: no
Philipp Kern writes: > I like the idea, especially for organizations who know what they are > doing[1]. I just fear that it won't actually solve your denylisting > problem at hand. People will keep not specifying it. Can't popcon go and > just accept reports for packages in the archive somehow? I don't know if it currently does this, but it would be useful for popcon to show counts for public third-party packages that aren't in the archive. Counts for things like google-chrome or non-archive Kubernetes packages provide possibly useful information about how our users use the distribution, and they've opted into sharing that information with us. That said, I'm not sure that anyone uses that information right now, which may make this a weak argument. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: popularity-contest: support for XB-Popcon-Reports: no
Bill Allombert writes: > I plan to add support for 'XB-Popcon-Reports: no' to popularity-contest. > This allows to build packages with private names that will not be > reported to popcon, by adding 'XB-Popcon-Reports: no' to debian/control. > This must not used by packages in the debian archive, however that can > be used by packages generators that create packages with randomly > generated names (package names that include 128bit uuid for examples) or > by organizations that generates packages for internal use whose name > include the organization name. > The rationale is that the only info really leaked is the package name, > so it only make sense to hide a package if every system that have it > installed are also hiding it, so it is better to make it a property of > the package than of the system. > Any comment ? This sounds like a good idea to me. Using an additional binary package control field felt weird to me, and I wanted to believe there was some better place to put this information rather than introducing yet another boolean control field, but after thinking about it for a bit, I couldn't think of any better place that made much sense. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Advice for package needing stuff installed in /srv (node-shiny-server)
Eric Brown writes: > The issue is that the documentation, samples and all of the upstream > online documentation, e.g. > https://support.rstudio.com/hc/en-us/articles/219002337-Shiny-Server-Quick-Start-Host-a-directory-of-applications > (or on third party sites like Stack Overflow), refer to > /srv/shiny-server as the default location for hosting. I doubt > upstream would want to change all of that without a strong reason. > From a user's perspective it would be much clearer if Debian could do > the same, in my opinion, so all of this existing and distributed > documentation and support still applies to new users of the Debian > version. Is there a technical solution to enable this, e.g. post > installation, that satisfies Debian's requirements? Not really. This is the same issue that we've had before with other things. Given current Policy, you have to choose between pointing at a file installed with the package or defaulting to the upstream /srv location without there being a file there. It's not a great situation, and there was a long discussion a while back about how to address the same issue with tftp services that I think didn't arrive at a clear conclusion. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Advice for package needing stuff installed in /srv (node-shiny-server)
Ryan Kavanagh writes: > From FHS §5.8.1, > This hierarchy holds state information pertaining to an application > or the system. State information is data that programs modify while > they run, and that pertains to one specific host. Users must never > need to modify files in /var/lib to configure a package's operation, > and the specific file hierarchy used to store the data must not be > exposed to regular users. > So, I think this suggestion of storing shiny-server (presumably > modifyable) configurations and served data in /var/lib/shiny-server > would violate policy. I would expect a workflow similar to that of the Apache packages. The default home page is a static file shipped with the package in /var, but one of the first things that any administrator does after installing the package is point it at a different docroot that suits their local file system layout. That avoids all of these problems, since there is no need to modify the static home page in order to make the package work. (Although the administrator can choose to do so if they want. I sometimes use /var to hold various configuration things on my systems. The actions of the local administrator are outside the scope of the FHS, which only constrains distributions. This is why we use /var rather than /usr, so that administrators can choose to just use /var if they feel like it without running into problems with, for example, a read-only /usr.) This issue comes up occasionally with different packages and we've never come up with a great solution beyond making it configurable at install (either via explicit documentation or via a debconf prompt). It's a bit of a gap in the FHS, in my opinion, but I'm not sure it's worth developing local policy to close the gap by, for instance, introducing a new top-level directory for default docroots for various services (I know of web, gopher, and tftp with this issue). We do try to hold the line on not installing files in /srv and not requiring any specific layout of /srv because the FHS is pretty explicit that the choice of directory layout in /srv is entirely up to the local administrator and distributions should not be messing with it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Advice for package needing stuff installed in /srv (node-shiny-server)
Richard Laager writes: > On 4/27/22 12:57, Nilesh Patra wrote: >> I am looking for more opinions. > Patch every instance of /srv/shiny-server to /var/lib/shiny-server. The > admin can customize from there. A possibly even better option would be for it to take the web root as a command-line or configuration argument. It's weird for that sort of configuration parameter, which administrators are going to want to customize, to be hard-coded into the package (and for me it raises questions about its general code quality). -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Firmware - what are we going to do about it?
Hakan Bayındır writes: > As everybody knows, Debian is also releasing the said firmware as > compressed archives and these are visible in the download page [0], > however usage and documentation is neither clearly documented, nor easy > for the beginners or casual users. > Considering most users are doing installs over USB, and official Debian > ISOs are hybrid by default, how the following plan feels? > 1) Document the use of ZIP files for firmware introduction during install > more clearly and prominently, > 2) Make these ZIP archives much more accessible via links and prominent > placement, > 3) Document creation of a USB drive which is both bootable and has > writable space for extra files > 4) Write another document detailing adding these ZIP files to the said > media in 3, > 5) Profit by allowing people to assemble their own installation media with > firmware blobs if they see the need. I've been a Debian Developer for quite some time and can usually manage to figure out most tasks like this, and providing separate firmware to the installer has completely defeated me every time I've tried it. I've spent frustrated hours trying to follow the documentation and doing things that look right only to have the installer say that there's no firmware visible without any clue as to how to debug the errors. Every time I have tried this, I have eventually given up and found the non-free images, which just worked. If this is going to be the solution, it has to be WAY easier to do. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Firmware - what are we going to do about it?
Steve McIntyre writes: > Thanks! That's a really good question, and one that we should also > include on the GR. I'd split my option 5 into two: > 5. Include non-free firmware but do not enable it by default > 6. Include non-free firmware and enable it by default > In either case, we'd make the opposite choice available via a d-i kernel > command line option and a boot menu option. I think that makes sense? I agree with this option split, but that reminds me of a different procedural note. While I recognize and respect the desire to create a comprehensive ballot, I'm still going to advocate for proposing a GR only with the options that the person proposing the GR would vote above NOTA, and possibly only your preferred options. There are a couple of reasons for this. One is that the people who prefer your disfavored options may have their own way of wording them that's slightly different than what you might believe they would want to say, and it's awkward to update other people's ballot options. The other, somewhat more technical reason is that I expect this GR to require a 3:1 majority for some options, and mixing 3:1 and 1:1 majority options on the same GR can be rather confusing. We may end up needing to do that anyway for this vote, but I think it would be a good idea to avoid creating the problem unless someone steps forward and proposes a 1:1 option that they really want to win. (That said, I think there's a big exception, which is that if you've canvassed a bunch of people who may not want to try to craft their own ballot options, and developed options to reflect their views, I think that's a fine approach and a good reason to propose options that aren't your personal preference.) As a side note, I don't remember exactly how this worked before, but under the current constitutional rules the original GR proposer doesn't have a special ability to put a bunch of options on the original ballot. You start with one option, and then you can add more options but they all need to be sponsored independently. So that also pushes in this direction of ballot construction. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Firmware - what are we going to do about it?
Jonas Smedegaard writes: > Quoting Russ Allbery (2022-04-19 19:29:09) >> We need some way to clearly label non-free firmware packages so that >> you can apply whatever installation or upgrade policy locally that you >> want to apply, but solution #5 provides that by keeping the non-free >> firmware in a separate archive area (which apt calls "components") to >> which you can apply different apt policy. > The issue I have with option 5 is that non-free blobs are then enabled > by default. I just re-read option 5 and I don't see where it says that. My understanding of the proposal is that the firmware would be on the image and thus available to the installer. That doesn't imply that it will be automatically enabled, either in the installer or on the installed system. That could still be gated by a prompt. In other words, rather than having to do what one does now and choose between the free installer and the non-free installer, my understanding of option #5 is that there would be one install image, but there could then be a prompt asking you whether you want to install non-free firmware. We could even offer a few different options (with the caveat that options tend to confuse users, so we may not want to add too many or gate them behind an advanced mode): 1. Purely free installation. 2. Enable non-free firmware in the installer but don't put it on the installed system. (Not sure how useful this is, but I could see needing non-free firmware to bootstrap from wifi but the running system may eventually not use the non-free firmware.) 3. Enable non-free firmware and install it on the system but pin it so that it's never upgraded by default. 4. Enable non-free firmware and enable normal upgrades, similar to adding the non-free archive area today but only adding the firmware archive area. I think 1 and 4 are the most useful options, and I'm not sure how many people really want 2 or 3, but if there are enough people who want them, I don't see any technical barriers to adding them. I feel professionally obligated to argue that Debian should, *by default*, upgrade anything that it installs, since from a security standpoint that is the least risky default configuration (with, as always, the caveat that there are special cases with different security models for which this default isn't appropriate). But that doesn't rule out a prompt or allowing a user to turn this off if they want to. > I agree that we should make it easier for our users to choose to trust > black magic "stuff" that they need to enable their devices. > I do not think that we should impose on our users to trust black magic > by default, though. I think this is a somewhat different question than whether we put the firmware on the default installation media so that it's *available* if users want it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Firmware - what are we going to do about it?
Jonas Smedegaard writes: > Now, some may argue that I am describing a case for package pinning > here, and that *might* be true but I don't know that yet - because the > proposed change to the system does not exist yet so I cannot really know > that yet. Possibly the implementation will be so that I continuously > need to check if some new non-free blobs was introduced and block those, > instead of the current situation of not needing to do anything actively > to keeping most possible risky "stuff" away from my systems. I feel like Debian already offers multiple mechanisms to prevent installation or updates of packages, both specific packages and packages by suite, archive, etc. I'm dubious that we need some additional extra-special mechanism just for firmware, as opposed to documenting the many and varied mechanisms we already support for pinning old packages, disabling automated upgrades, and so forth. We need some way to clearly label non-free firmware packages so that you can apply whatever installation or upgrade policy locally that you want to apply, but solution #5 provides that by keeping the non-free firmware in a separate archive area (which apt calls "components") to which you can apply different apt policy. Given your problem description above where you want to manually review each new non-free package, I suspect you will want to pin the non-free firmware archive area to something like priority 1 (similar to experimental) so that you can access those packages but they're not otherwise installed or upgraded. Another option would be priority 100, which would give automatic upgrades but not new installs. I'm assuming we'll continue to maintain the invariant that free packages won't depend on non-free packages, so unless you install a non-free metapackage, you presumably won't get new non-free firmware packages without seeking them out. We may want to install such a metapackage by default when the non-free-firmware-enabled installer is used (leaving open the question of whether that's the only installer or not), but you could remove it, of course. (I suspect you, like me and probably most other Debian contributors, make pretty extensive modifications to the installed package list after installing a new system, and have lists of packages that you always add or always remove.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: proposed MBF: packages still using source format 1.0
Richard Laager writes: > Could we only have "3.0 (quilt)" then, no "3.0 (native)"? Or, put > differently, if you had a "native" package that is using a Debian > revision and we allow that, what difference is left between "3.0 > (native)" and "3.0 (quilt)"? 3.0 (quilt) always has two tarballs, one for the upstream source and one for the Debian packaging. 3.0 (native) has a single source tarball, which I think is the desired goal here. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: proposed MBF: packages still using source format 1.0
Lucas Nussbaum writes: > The breakdown in terms of packages count is: > patch_system | direct_changes | vcs | count > --++-+--- > dpatch | no | no | 3 > quilt| no | no |26 > quilt| no | yes |96 > none | no | no | 185 > none | no | yes |78 > none | yes| no | 166 > none | yes| yes |74 > I propose to file bugs for packages in all categories above, except for > packages in the last category that are maintained in an active VCS > repository, because those are the most likely to be be using a git > workflow that makes it hard to use the 3.0 format (even if I don't fully > understand the arguments against using single-debian-patch in that > case). If you're going to omit the ones in the last category, I think you should also omit the ones in the none/no/yes category, since they may be packages that intermittantly have changes and are similarly using a VCS-based workflow that doesn't want to use the 3.0 format. A mass bug filing for the first three categories seems like the change with the biggest potential to benefit Debian, since it's a direct simplification of the number of ways packages are maintained in the archive. The packages without any patch system feel a bit less interesting. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: non-word "noone" in constitution
nick black writes: > i was reading our Constitution, and §5.1.4 states: > "Make any decision for whom noone else has responsibility." > "noone" is a non-word [0], and while i doubt it'll ever be the basis of > a court ruling or anything, "no one" was correctly used in the language > added to 6.3.6 by Constitution v1.8, and another instance of "noone" was > removed from §A.1.6. so unless it requires a vote or something, it > should probably be fixed? It does alas require a vote, at least according to my reading of the constitution. We don't have a provision for making minor changes apart from the normal amendment process. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: LESS copyright, not more!
Adam Borowski writes: > Guys, once again we had a complaint about forcing people to waste their > time on copyright matters then wait months or years for review of said > matters -- just for the discussion degenerate into a proposal to bring > even MORE copyright into our life! To be very clear, the reason why I am interested in SPDX and REUSE is so that I can do *less* copyright, specifically so that I can automate generation of debian/copyright for my packages from information that upstream already tracks and never have to think about it, touch it, or update it. This doesn't achieve less copyright law in the world (I fear nothing we do technically can do that since that's a political and legal issue), but it achieves the goal of me spending less total time dealing with copyright and licensing issues than I do now. So I think we may be more aligned than it may appear. Yes, I think everyone knows that SPDX and REUSE can specify a ton of extra stuff that we aren't tracking now. The goal (at least as I see it) is *not* to make everyone track all that stuff. It's to (a) use that work if it already exists instead of making folks waste time repeating that work, and (b) expose the information that we did have to gather in a standard format so that other people can similarly have less copyright in *their* life by automating their pain as well. But to be very clear, what I'm hoping to get out of this is less time spent on copyright and licensing, not more. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Automated copyright reviews using REUSE/SPDX as alternative to DEP-5
Russ Allbery writes: > I can tell you from prior experience with DEP-5 that 3 is wildly > controversial and will produce the most pushback. There are a lot of > packagers who flatly refuse to even use the DEP-5 format, and (pace > Jonas's aspirations) I don't expect that to change any time soon. That last parenthetical was in retrospect probably not very clear, particularly to people who aren't familiar with that specific English idiom. For the record, this is "pace" in the sense of: https://www.merriam-webster.com/word-of-the-day/pace-2017-09-28 What I meant to express is that I realize that Jonas is actively working on this and is hopeful, and I would love for him to be successful, but I am more pessimistic. But I wanted to acknowledge that he disagrees and I may well be wrong. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Automated copyright reviews using REUSE/SPDX as alternative to DEP-5
Stephan Lachnit writes: > It would require a minor change: putting the verbatim license texts in a > single file is not possible anymore. But I don't why just copying the > licenses to "/usr/share/doc/PACKAGE/licenses/LICENSE" in addition to the > SPDX formatted debian/copyright would be any worse than the current way. I recommend thinking about how to generate an existing debian/copyright file and putting the SPDX-formatted one in a different location. You're going to want to decouple the new work from any transition that requires other people change tools. There are a lot of tools that make assumptions about debian/copyright, and trying to track them all down will be counterproductive for trying to make forward progress on exposing information in a more interoperable format. The way I see this, there are three different things that have been discussed on this thread: 1. Consuming upstream data in SPDX or REUSE format and automating the generation of required Debian files using it. 2. Exposing SPDX data for our packages for downstream consumption. 3. Aligning the standard way to present Debian copyright information with other standards. I can tell you from prior experience with DEP-5 that 3 is wildly controversial and will produce the most pushback. There are a lot of packagers who flatly refuse to even use the DEP-5 format, and (pace Jonas's aspirations) I don't expect that to change any time soon. I think that's fine for what you're trying to accomplish, because I think 1 and 2 are where the biggest improvements can be found. As long as your system for managing copyright and license information can spit out a DEP-5 debian/copyright file (in its current or a minorly modified format, not with new files elsewhere that would have to be extracted from the package), you are then backward-compatible with the existing system and that takes 3 largely off the table (which is what you want). Then you can demonstrate the advantages of the new system and people can choose to be convinced by those advantages or not, similar to DEP-5, and we can reach an organic consensus without anyone feeling like they're forced to change how they do things. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Automated copyright reviews using REUSE/SPDX as alternative to DEP-5
Scott Kitterman writes: > Technically it would be the simplest, but there's a process for policy > changes that's more involved than writing emails to d-devel. I'm > suggesting you engage with it on this topic if you want the results of > your work to be usable in Debian. Speaking as a Policy Editor, the way that Stephan is engaging with this process is fine. We're in the early stages of discussion and understanding the shape of the problem. Writing emails ot debian-devel, and writing a DEP, are exactly the right way to engage with Policy on this topic right now. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Legal advice regarding the NEW queue
Timo Röhling writes: > The FTP team review should focus on these types of mistakes, and not > only with new packages: any "suspicious" upload should be rerouted to a > POLICY queue for additional verification. There is some prior art with > the auto-REJECT on Lintian errors, which could be extended to a > three-way decision (ACCEPT, POLICY REVIEW, REJECT). I feel like it would also be a lot easier to get volunteers to look at packages that had already been flagged, rather than do green-field reviews of every package everyone uploads (most of which are entirely fine or have at most minor issues). It certainly feels like a more interesting problem to me! "Here are a few things that looked weird, please manually look at this package to see if they're a problem." Or even "this package has maintainer scripts that aren't just debhelper-generated boilerplate, please look at them and make sure they're not going to run rm -r / or something crazy." > For instance, we could flag epoch bumps or major version numbers which > skip ahead significantly (think 020220101 instead of 0.20220101). We can > certainly continue to flag new binaries and potential copyright > violations (e.g., packages with incompatible licenses or files with > "(?i)(?:do|must) not distribute" in their headers), or anything else we > consider important. The automated checks need not be perfect as long as > we avoid false negatives on the critical issues. Exactly. We're not going to catch everything, to be sure, but we're not catching everything *now*, and improvements of automation scale in ways that trying to do more painstaking human review do not. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Legal advice regarding the NEW queue
Scott Kitterman writes: > Since we're doing strawman arguments in this thread: I disagree with the > notion that it's not a problem to put crap packages in the archive and > fix them later if anyone happens to notice. No, that's fine, that's not a strawman argument. That is, in fact, my argument: I think it should be okay to put crap packages in the unstable archive and fix them later, and I would rather put more effort into the "noticing" part than in the pre-review part. We may quibble about what "crap" means and we may disagree about how much of this potentially could be weeded by automated tools (and I want to be very clear that I'm not opposed to automated checks and indeed think we should make them stricter), but I think this is a blunt but fair characterization of my position. To be clear on the nuance I see here, I don't mean that this is "okay" in the sense that people should feel fine about doing it. I think we should all aspire to not do that, of course. But I think it should be "okay" in the sense that I don't think we should invest the level of resources we're currently investing in trying to avoid it, because I think that's causing other significant problems for the project. My argument in favor of this position is that while it's very obvious to see the harm from having crap packages in the archive, we're overlooking the very substantial cost we're paying with our current method of trying to reduce the frequency with which this happens. I think we're underestimating just how costly and demoralizing dealing with NEW delays is for project work, and also how much of a drain and competition for resources that is with other archive work that we could be doing. For example, in just the past two months I have seen two *extremely experienced* Debian developers who maintain extremely high-quality packages express qualms about package architectures that would fix other bugs in Debian *solely* because they would force more trips through NEW and the trip through NEW is so disruptive to their work that it was at least tempting to accept other bugs in order to avoid that disruption. To me, this indicates that we may have our priorities out of alignment. Now, all of that being said, I also want to say that I'm sketching out one end of the argument because I think that end has been underrepresented. I don't think this is an all-or-nothing binary choice. We could, for example, focus our review on only packages that are viewed as riskier (only packages with maintainer scripts or that start daemons, for instance, or stop doing NEW review for packages uploaded under the auspices of well-established Debian teams, or stop doing NEW review for shared libraries whose source packages are already in Debian), all of which would be improvements from my perspective. We could also do some parts of NEW review and not others and see if that makes it more attractive for other people to volunteer. (The manual review for d/copyright correctness is certainly the part of NEW review that I can't imagine volunteering to do, and I suspect I'm not alone.) To be clear, as long as the rules in Debian are what they are, I will of course follow them as I promised to do when I became a Debian Developer. If the project continues to believe that it is of primary importance for us to be the copyright notice and license catalog review system for the entire free software ecosystem (which is honestly what it feels like we've currently decided to volunteer to do on top of our goal of building a distribution), then I will do my part with the packages that I upload so that I don't put unnecessary load on the folks doing NEW review. But when we've collectively been doing something for so long, we can lose track of the fact that it's a choice, and other choices are possible. It's worth revisiting those choices consciously from time to time. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Legal advice regarding the NEW queue
The Wanderer writes: > What I read Scott as having been suggesting, by contrast, is that people > instead do copyright review for packages already in Debian, which may > well have had changes that did not have to pass through NEW and that > might not have been able to pass the NEW copyright review. > If a practice of doing that latter were established and sufficiently > widespread, then it would not be as important to do the review for every > package in NEW, and the FTP team might feel less of a need to insist > that the review take place at that stage of things. Various people have different reactions to and opinions about the necessity of this review, which I understand and which is great for broadening the discussion. But I feel like we're starting to lose track of my original point, namely that I don't see why we are prioritizing this particular category of bugs over every other type of bug in Debian. The justification has always been dire consequences if we don't stamp out all of these bugs, but to be honest I think this is wildly unlikely. In other words, this thread is once again drifting into a discussion of how to do copyright review *better*, when my original point is that we should seriously consider not doing the current type of incredibly tedious and nit-picky copyright review *at all*, and instead rely more on upstream's assertions, automated tools, and being reactive in solving the bugs that people actually care about (i.e., notice). In other words, what if, when upstream said "this whole package is covered by the MIT license," we just defaulted to believing them? And if there's some file buried in there that's actually covered by the GPL, we fixed that when someone brought it to our attention, or when we were able to detect it with automated tools, but we didn't ask people to spend hours reviewing the license headers on every source file? What, concretely, would go wrong? Scott correctly points out that there are a ton of copyright bugs in Debian *anyway*, despite NEW review. He sees this as a reason for not relaxing our review standards. I see it as the exact opposite: evidence that our current review standards are not achieving the 100% correctness we have claimed to be striving for, and the nearly complete lack of practical consequences for that failure. It really seems to me like evidence that this task is not as important as we think it is. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Legal advice regarding the NEW queue
Scott Kitterman writes: > On Tuesday, February 1, 2022 12:18:07 PM EST Russ Allbery wrote: >> Wookey writes: >>> For what it is worth I concur with everything that Russ has written, >>> and would like to have us look at this again (and that's honestly not >>> particularly because I currenly have the honour of the 6th-oldest >>> package in NEW (8 months) :-) In general I have found NEW valuable as >>> FTP-masters sometimes spot things that I missed, but the delay, and >>> perhaps worse, the highly uncertain length of the delay (anything from >>> a day to a year), is a significant cost and drag, and it seems >>> increasingly anachronistic as the rest of the software ecosystem seems >>> to accelerate around us (not entirely a good thing, of course). Who >>> needs quality when you can have updates, eh? >> I would hate to entirely lose the quality review that we get via NEW, >> but I wonder if we could regain many those benefits by setting up some >> sort of peer review system for new packages that is less formal and >> less bottlenecked on a single team than the current NEW processing >> setup. > It's my impression that review of copyright and license considerations > when not going through New is not a priority for most. I doubt making > New go away will make it more so. To be clear, that's not the part I was intending to reference. I think we spend too much time reviewing copyright and license considerations for the amount of benefit we get from it and would rather we rely more heavily on automated tools and upstream's assertions, and otherwise be more reactive and less proactive about issues that aren't fundamental to whether something is free software. (This is one of the reasons why I'm interested in REUSE. I would love for us to be able to leverage the work that some upstreams have already done and trust it unless we know it's wrong.) I was thinking about all the other things that NEW catches, where people have accidentally made more foundational mistakes in constructing a package. That comes up a lot in these discussions and I do think there's merit in NEW as another pair of eyes on each new package. That's the part that I think would be interesting to try to find a way to preserve if possible. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Legal advice regarding the NEW queue
Andrey Rahmatullin writes: > On Tue, Feb 01, 2022 at 09:18:07AM -0800, Russ Allbery wrote: >> I would hate to entirely lose the quality review that we get via NEW, >> but I wonder if we could regain many those benefits by setting up some >> sort of peer review system for new packages that is less formal and >> less bottlenecked on a single team than the current NEW processing >> setup. > What do you think, would it be more or less staffed than the current RFS > review process? Good point, it probably would be about the same. The fundamental problem we have is that we don't have enough packaging resources to keep up with demand. My intuition is that the resources we do have could be allocated with more impact than the comprehensive copyright NEW review (honestly, I'd rather rely on tools like licensecheck as much as possible and live with occasional bugs, since I'm not convinced the bugs are serious enough to warrant special attention above and beyond any other bug), but it's true that this won't change the basic resource constraint, just move around what we spend resources on. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Legal advice regarding the NEW queue
Wookey writes: > For what it is worth I concur with everything that Russ has written, and > would like to have us look at this again (and that's honestly not > particularly because I currenly have the honour of the 6th-oldest > package in NEW (8 months) :-) In general I have found NEW valuable as > FTP-masters sometimes spot things that I missed, but the delay, and > perhaps worse, the highly uncertain length of the delay (anything from a > day to a year), is a significant cost and drag, and it seems > increasingly anachronistic as the rest of the software ecosystem seems > to accelerate around us (not entirely a good thing, of course). Who > needs quality when you can have updates, eh? I would hate to entirely lose the quality review that we get via NEW, but I wonder if we could regain many those benefits by setting up some sort of peer review system for new packages that is less formal and less bottlenecked on a single team than the current NEW processing setup. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Do we need to hide packages in NEW queue
The Wanderer writes: > I am not on the inside of these things, certainly, but I have kept my > eyes open from the outside, and I am not aware of there being any > mechanism for removing something root-and-branch - across all affected > versions, however far back those may stretch - from these repositories > and archive locations once it's made it in. In order to avoid continuing > to distribute something which we once accepted but which has since been > deemed legally undistributable (and thus exposing ourselves to > copyright-infringement lawsuits), we would need to have such a > mechanism. The thing is, we need this anyway for something we would legally need to stop distributing, since otherwise we would be expecting ftp-master review to be perfect *and* to never introduce unredistributable content in a package update that doesn't go through NEW. I don't think either of those are realistic (or fair) expectations. Now, we could defer creating such a thing until we actually need it and then try to come up with something under emergency circumstances, and maybe we'd get lucky and never need it. But I think that's also true in either scenario. I'm not sure that the ftp-master review reduces the likelihood so much as to change the risk analysis that much. (But that could well be a point of disagreement.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Do we need to hide packages in NEW queue
Stephan Lachnit writes: > If I compare how other mediums handle copyright violations, most > services have a "file a claim infringed copyright here" button on their > site (e.g. YouTube). For example, we could write a DMCA policy like > e.g. Github [2], hyperlink in the footer of all our official websites, > make a small "debian-dmca" tool that is always available in our builds > to file claims and provide infrastructure to process such claims. Just as a side note, I believe the DMCA safe harbor provisions only apply to entities that allow unrelated third parties to upload material to their web sites (social media, GitHub, etc.). Debian only allows project affiliates to upload packages, so I suspect that we cannot use the DMCA safe harbor provision. (A lawyer would of course be able to answer that question more conclusively.) This is a bit murky given that Debian has only murky legal existence, but it doesn't feel like we're within the spirit of the DMCA safe harbor provision. (Also, I don't know what that looks like in non-US law, since the DMCA thing is specifically a US law.) > I highly doubt that anyone will ever directly start a lawsuit instead of > sending a cease-and-desist letter first, I'm not even sure if it is > legal to start a lawsuit without doing this first. Right, the other angle that it would be nice to have some more legal information about is what would happen if we did make a mistake. How likely is it that we wouldn't be able to remedy it without much cost if we acted promptly? This is less a question about the specific language of the law and more a practical question of how the law works in the real world, something that a lawyer would be better-qualified to weigh in on. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Do we need to hide packages in NEW queue
Marc Haber writes: > Even if a lawyer says A, it doesn't buy us anything if J Robert DD gets > sued and the judge says B, or "not A". Yes, a legal opinion cannot fully resolve the question, unfortunately, since it's a risk judgment. Copyright law is murky enough that it's unlikely that any lawyer will be willing to guarantee that we won't lose a lawsuit, and of course no one can guarantee that we won't be sued. What a lawyer can do is give us a better risk analysis. How *likely* is it that we would be sued over such a thing, and if we were, what would happen then? How much would it cost us to dispose of the resulting lawsuit? I think it's useful to view this as a price. We're paying a quite substantial price right now to implement pre-screening. If we increase the risk that we may temporarily distribute something that we shouldn't until we discover that and fix it, that comes with some corresponding increased risk of a legal cost. But in the meantime we'd be saving a substantial pre-screening cost. A lawyer cannot make that risk trade-off decision for us. We'll have to make it as a project. But my hope would be that they could help put a number on the likely legal cost in the worst-case scenario and provide some input into the likelihood of that scenario, and some context in terms of what other organizations do and what risks it's common to accept and mediate if it becomes a problem. My personal guess is that, given how completely casual or even openly contemptuous most companies are about copyright licensing and how insanely difficult it's been to get them to face any legal consequences whatsoever, it seems unlikely that dealing with some licensing issues more reactively would be a substantial legal risk. By dealing with them *at all*, and we would of course continue to hold ourselves to the same high standard that we always have, we're already doing far better than the industry norm. But a lawyer would have much more concrete experience and would be able to provide a far better analysis. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Do we need to hide packages in NEW queue
em to the degree possible, and if anyone reports a copyright or licensing bug, we will prioritize resolving it." In other words, pretty much exactly the policy we use right now for security issues, which I suspect are far more dangerous to Debian users on the whole than copyright and licensing issues (although both are important!). -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Automated copyright reviews using REUSE/SPDX as alternative to DEP-5
Phil Morrell writes: > The point about uscan is interesting, since if upstream does take on the > hard work of license verification such that packaging is just checking, > then they're unlikely to have any Files-Excluded, so that would have to > merged somehow. My intuition (I admit that I haven't done a survey) is that Files-Excluded is less frequently used for cases where upstream has not done license verification and is more frequently used for cases where upstream disagrees with Debian over what is and is not free software. (IETF RFCs are a sadly common example.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Lottery NEW queue (Re: Are libraries with bumped SONAME subject of inspection of ftpmaster or not
Jonas Smedegaard writes: > I just don't think the solution is to ignore copyright or licensing > statements. That's not the goal. The question, which keeps being raised in part because I don't think it's gotten a good answer, is what the basis is for treating copyright and licensing bugs differently than any other bug in Debian? The need for pre-screening was obvious when we had export control issues, but my understanding is that those have gone away. Are we working from legal advice telling us that this pre-screening is required for some legal purpose? If so, is it effective for the legal purpose at which it is aimed? Is this system left over from old advice? Have we checked our assumptions recently? NEW processing is a lot of friction for the project as a whole and a lot of work for the ftp team. If we were able to do less work at the cost of a minimal increase in bugs, or at the cost of handling bugs a bit differently, maybe that would be a good thing? In other words, it's unclear what requirements we're attempting to meet and what the basis of those requirements is, which makes it hard to have a conversation about whether the current design is the best design for the problem we're trying to solve. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: build-depend on autoconf-archive and rm convenience copy in orig.tar.gz?
Jonas Smedegaard writes: > Thanks for elaborating. > The concern is not licensing, but maintenance. It is covered in Debian > Policy § 4.13: > https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies Debian packages should not make use of these convenience copies unless the included package is explicitly intended to be used in this way. The part after the "unless" is the case for autoconf-archive upstream: shipping a copy with your package is how it is intended to be used. One can disagree with that statement in Policy, of course (I think it's debatable and there are some benefits to pulling everything from the original source even if it is intended for use as an embedded copy), but I didn't want to leave the thread with the impression that Policy supports a dependency here. It does not; since this meets the "unless" condition, Policy expresses no opinion on a dependency versus an embedded copy, either for or against. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Deps of -dev packages with pkg-config .pc file: Policy Change?
Simon McVittie writes: > For some libraries, the only maintainer-supported way to consume the > library is via pkg-config. If that's the case, then a dependency on > pkg-config can be appropriate - although we don't add a dependency on > cc or binutils, which is equally necessary. Well, cc and binutils are in build-essential, so this isn't entirely equivalent. > For other libraries, either there are other supported ways to consume > the library (CMake metadata or sdl2-config or similar), or the library > is in the compiler's default search paths (not parallel-installable) > like libjpeg - so you *can* use pkg-config, but you don't *have* to. Yeah, I think this is the key point: it's entirely possible to use most libraries without pkg-config because they're installed into the default search paths, so you can just... use them. If they *require* special flags (non-standard paths, non-standard compiler flags, etc.) such that pkg-config is the only supported interface, then I think one could make a good argument that pkg-config should be a dependency of the -dev package. > That's why I think it is best that the information about which libraries > GLib depends on is shipped with GLib and included by reference (via > either Requires.private: glib-2.0, or the Requires.internal: glib-2.0 > proposed on https://bugs.freedesktop.org/show_bug.cgi?id=105572), > instead of copying the information about GLib's dependencies into > libfoo-dev where it will become outdated when GLib changes. Yes. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Deps of -dev packages with pkg-config .pc file: Policy Change?
Simon McVittie writes: > If you're linking statically, you need to be able to satisfy the > recursive dependencies of libunbound (regardless of whether you are > using pkg-config or not), so, no, you will need nettle-dev and > libevent-dev either way. And, specifically, I think we should say as a matter of policy that Debian does not try to declare the dependencies required for static linking and therefore static linking may require tracking down what additional -dev packages you need and manually installing them. The dependencies for -dev packages should reflect the requirements for dynamic linking. Static linking of C libraries is rare enough at this point, and (possibly more relevantly) sufficiently untested by Debian that I don't think we should try to support it directly in the dependency structure. It would pull in a whole bunch of -dev packages that wouldn't be used 95% of the time. This is what I already do for libremctl-dev, for example. (Well, I list the libraries required for static linking in Suggests currently, but I'm not sure that list is complete any more and should probably just drop it completely.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: uscan roadmap
Yadd writes: > after few discussions with some devscripts maintainers, we decided to > build a new "version=5" format for debian/watch. > Principles: > * keep compatibility with versions 3 and 4, no need to change all >debian/watch files > * new version 5 format using the same syntax than other debian/* files >(rfc822 + "# comments") I have no detailed feedback on your proposal, but just wanted to say thank you so much for doing this. Being able to write a uscan file using a clear key/value syntax will make me very happy. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: merged-/usr transition: debconf or not?
Ansgar writes: > So I do not think there should be a command-line option for this; unless > it would behave like `--add-architecture` and register the setting > somewhere. Agreed. I also think it needs to be a two-phase thing, because dpkg has to convert its internal database to respect the new aliases. So there should be a desired configuration and then an applied configuration, and the latter is only updated once the database conversion is complete, with dpkg refusing to perform package operations if the desired configuration doesn't match the applied configuration. Even if the command that changes the configuration also does the database conversion, I think we need this, since I don't believe it's possible to do the database conversion atomically (or at least it would involve extra complexity that I don't think we want), so we have to handle the database conversion being interrupted. > I do not think this should be a configuration file (unless you suggest > supporting other values or removing individual lines; after all a > configuration file exists to be changed). Are you thinking of perhaps shipping this configuratino somewhere in /usr/lib instead so that it's not marked as a configuration file? -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: merged-/usr transition: debconf or not?
Johannes Schauer Marin Rodrigues writes: > Please don't. Since 2014 it is possible to create a Debian chroot > without taking care of the unpack order and without running any special > setup beforehand that cannot be put into apt or dpkg itself (like > creating /var/lib/dpkg/info which is not needed anymore since dpkg > 1.20.0). One of my biggest motivations in writing mmdebstrap was to get > rid of all the quirks that currently live in debootstrap, move them into > apt and dpkg so that we end up with a declarative bootstrap process that > is driven by the package contents, their metadata and maintainer scripts > but does not need magic from the outside other than standardized > interfaces. Requiring the merged-/usr setup to be done by the bootstrap > script would be a step backwards. I understand why you want this, but unfortunately it doesn't answer the question of how to safely perform the upgrade and reconfigure dpkg. Do you have any thoughts on the drawbacks of the approach of putting the symlinks in base-files that led me to lean away from doing that? I'd love to get all of the machinery into packages as well, but I think safely upgrading existing systems is even more important than that, and I'm worried about our ability to do so if we put the machinery in base-files. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: merged-/usr transition: debconf or not?
Gabor Gombas writes: > If you replace "rewrite /lib64 to /usr/lib64" with "rewrite /lib64/* to > /usr/lib64/*", then this can easily be avoided. True, although I think you still want to ensure that dpkg never messes with those links. I don't think you want base-files to be able to drop one of those links from the package and cause it to be removed by dpkg, for instance. Maybe I'm borrowing trouble for a scenario that won't happen (and I think dpkg may have some special-casing of symlinks to directories anyway). > Talking about "special casing" in dpkg is bothering me for a while. And > there is a relatively simple way to avoid any kind of special casing: > move the information out to a configuration file (which would _not_ be a > conffile) - and now the code has no special casing, just > configuration-driven logic. All that I mean by special-case is that I think it would be a mistake to try to shoehorn support for merged-/usr into some pre-existing dpkg concept. The merged-/usr links are special relative to other files that dpkg is managing and need to be treated with special care. We should add a new concept to dpkg for these sorts of links. I agree with making the exact set of links configurable. This also allows handling the architecture-specific links in a clean way. > This new configuration file could be shipped by base-files itself, to > ensure it does not get out of sync with the filesystem structure shipped > in base-files. Then base-files' postinst would more or less need to > include the current usrmerge package, plus whatever is needed to convert > dpkg's database. I don't think we want base-files to take over the logic of usrmerge. usrmerge is both complex (in part because it relies on Perl) and not 100% hardened against weird issues, being interrupted in the middle of running, etc. That's all fine for what it's doing as a one-time migration tool that's run with care during an upgrade when the administrator already knows Big Things are happening to the system. Putting the logic into base-files makes it more of a routine thing and I think it should be a lot more hardened if we do that (and I don't think that's a good use of anyone's time). We ideally want usrmerge to be a tool each system runs once, under controlled circumstances, and then never again (and isn't needed at all for bootstrapping). We have a migration strategy that's working. Let's stick with that migration strategy without messing with it and instead just add a dpkg database conversion step to the end of that migration. I do agree that something needs to handle installing the configuration file and laid out some options for that in another message. > This new configuration file would not be consumed by dpkg directly when > installing packages, but only when the database conversion is called, > and dpkg would keep an internal list of the rewriting rules which are > active. Doing so would allow enforcing rewriting rules can only be added > but never removed, which would avoid potential issues if base-files gets > downgraded. This is a very good idea. I was starting to think along those lines, but hadn't taken it quite far enough. That would make it akin to dpkg --add-architecture, which feels like a good model (although as you mention I don't think we want --remove-architecture). -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: merged-/usr transition: debconf or not?
Russ Allbery writes: > Well, bootstrapping a new Debian system involves running a tool that > bootstraps a new Debian system. I think you're constraining the problem > too much. > It's a nice property that everything on the system comes straight from a > Debian package, but it's not a strict requirement, nor is it currently > generally the case (maintainer scripts do things, for instance). Those > symlinks are very special from a dpkg perspective; dpkg needs to refuse > to mess with them even when upgrading a package that provides them, > which would mean some irritating special-casing I suspect. It's not > clear to me that shipping them as tar members of a package is the right > way to go, as opposed to creating them separately as part of early > system configuration. Having slept on this, let me make this problem concrete. I'm going to make the following assumptions: * We have some mechanism to put dpkg into what I've been calling merged-/usr mode. In this mode, it pre-filters all input paths from whatever source (including arguments to dpkg-divert, update-alternatives, etc.) and canonicalizes them in a very specific way: the directories that become symlinks in merged /usr are replaced in each path with their canonical paths. So /bin/ls becomes /usr/bin/ls, /lib64/ld-linux-x86-64.so.2 becomes /usr/lib64/ld-linux-x86-64.so.2, and so forth. * When bootstrapping a new Debian system, we want to put dpkg into merged-/usr mode as early as possible. * If dpkg is in merged-/usr mode, the first thing it does is checks the file system on which its operating and ensures that the expected symlinks already exist. If they do not, it aborts because operating on that file system is unsafe. (Leaving aside for the moment whether there should be some -force option, etc.) This produces a bootstrapping problem: all dynamic binaries on, say, an amd64 system refer to /lib64/ld-linux-x86-64.so.2 [1]. Therefore, before the first binary is run from within the context of the newly-installed system, either that path must exist as-is (which we don't want because we want to create a merged-/usr system where it belongs in /usr/lib64) or the symlink from /lib64 to /usr/lib64 must already exist. I think it's fairly obvious that we don't want a long-term design in which the libc6 package has to continue to ship /lib64/ld-linux-x86-64.so.2, we originally unpack that file in that path during bootstrap, and then something has to come along later and move it to /usr/lib64 and create a symlink. This has numerous drawbacks: weird special cases people have to remember, not being able to reconcile the contents of packages with their canonical paths in the long run, a window where we have to do file system surgery atomically, etc. Instead, we want to live in a world in which libc6 can ship /usr/lib64/ld-linux-x86-64.so.2, dpkg sees it as shipping that path, but everything works fine during bootstrap because the /lib64 to /usr/lib64 symlink is already established before we have to execute binaries in the new environment. (Obviously it may be some time before we actually change the contents of the libc6 package; that's fine, the point of adding a path filter to dpkg is that we can take our time on that.) So, assuming we have a libc6 package that contains either /usr/lib64/ld-linux-x86-64.so.2 or /lib64/ld-linux-x86-64.so.2, how do we bootstrap this system? I can see a few approaches: * Put the required symlinks into base-files. This preserves the nice property that every file on the system comes from and is registered as belonging to a Debian package (which we don't fully meet but which we always aspire to). I haven't studied all the various tools for bootstrapping a Debian system, but I presume that unpacking base-files is the first thing that any of them do. This will therefore lay down the symlinks from the start and it won't matter what path under which the libc6 package ships the files if it's extracted over the resulting file system with ar+tar. The drawback here is that dpkg is going to rewrite all paths like /lib64 to /usr/lib64, which would naively *also* apply to the base-files package when it looks at that package, but that can't be allowed because now we're back to the situation where dpkg's state database is inconsistent with the file system and dpkg thinks that base-files contains some nonsensical /usr/lib64 to /usr/lib64 symlink. I think in this approach there would need to be some special-case code directly in dpkg that recognizes the usrmerge symlinks (and only specifically those symlinks) as special and preserves them as-is in the installed package database. (I think it's probably better to special-case the specific symlink than to special-case base-files the package.) We will then need rules that base-files must always contain those symlinks and they can't move between packages, etc., (or the
Re: merged-/usr transition: debconf or not?
Simon Richter writes: > Bootstrapping a new Debian system works by unpacking Essential packages > with ar+tar, i.e. not using dpkg. These packages will always be unpacked > to the file names listed inside the packages. Well, bootstrapping a new Debian system involves running a tool that bootstraps a new Debian system. I think you're constraining the problem too much. It's a nice property that everything on the system comes straight from a Debian package, but it's not a strict requirement, nor is it currently generally the case (maintainer scripts do things, for instance). Those symlinks are very special from a dpkg perspective; dpkg needs to refuse to mess with them even when upgrading a package that provides them, which would mean some irritating special-casing I suspect. It's not clear to me that shipping them as tar members of a package is the right way to go, as opposed to creating them separately as part of early system configuration. > Dpkg has an elaborate per-file tracking to recover from errors, and this > change introduces several new states and transitions, Why? I am not convinced this is true, apart from the obviously-needed one-time conversion of the state database. It involves a new filter, applied whenever a package is installed, that changes all the paths in the package to their merged-/usr paths, something that seems like it should be done before any other package processing. Once that's done, why would any new states or transitions be required? You could convince me that writing the filter is complicated because there may not be one place in dpkg where you can currently put such path rewriting since dpkg probably wasn't designed with that operation in mind. But it's going to be harder to convince me that there are new states or transitions; that smells like an over-complicated design. > That is the goal, yes, but this is a massive undertaking. I'm still not convinced of this. > We already have a quick hack for usrmerge support in dpkg, which is the > check that allows moving a file from /bin to /usr/bin within the same > package without the file being deleted in the end (because deletions are > processed last for obvious reasons), and the obscure rule is precisely > the limitation of this hack. This already sounds conceptually more complicated than just solving the problem properly, with the caveat as mentioned above that writing the necessary filter may be difficult. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: merged-/usr transition: debconf or not?
Simon Richter writes: > I think the main blocker at the moment is that the dpkg change *also* > has the potential to break a lot of things if it isn't carefully > designed. I think you can simplify that problem space considerably because we know exactly what symlinks we care about and we don't intend to support going backwards (in other words, once you have merged /usr, it's not supported to turn /bin, etc., back into regular directories again). Therefore, dpkg can be configured with the list of the symlinks that *must* exist for it to be willing to operate on the system. Once it has been put into merged-/usr mode (which includes updating its state database to reflect the new canonical paths of all installed files), if those symlinks do not exist or do not have the correct targets, the system state is not safe and dpkg can (and probably should) abort and refuse to operate on that system. That modifies the problem of systems in various weird configurations from "weird things may happen" to "dpkg will (safely) refuse to run" and we can provide a tool to fix such a system by creating the correct symlinks. (This tool may well just be the existing usrmerge tool.) Getting into that state will be hopefully rare if we structure the upgrade process so that first one converts the system to merged-/usr if this has not already been done, then upgrades dpkg, then tells dpkg to switch into merged-/usr mode, and then safely upgrades the remaining packages. > From dpkg's perspective, we now (kind of) change the policy for > directory-vs-symlink conflicts, which dpkg currently resolves in favour > of the directory. I don't think we should be doing anything of the sort. I think we should introduce a new concept specifically for this case and have the remappings of files for merged-/usr, once dpkg is in merged-/usr mode, take precedence over any rules like that. We do not want dpkg to be applying some policy or heuristic about directory vs. symlink conflicts for symlinks related to merged-/usr. We know exactly what we want the rule and behavior to be. That sort of remapping should happen way before any other dpkg analysis of the package or system state. In other words, I argue that once dpkg is in merged-/usr mode, it should treat a package with files in /bin exactly as if the files were in /usr/bin, at an earlier stage than when it starts looking at the file system. We're not trying to create a generalized feature here. Merged-/usr has a very specific definition, with very specific and clear behavior that we need from the package manager. Trying to generalize this is going to make a mess and make it much harder to reason about the behavior or test it. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: merged-/usr transition: debconf or not?
Richard Laager writes: > Is this particularly hard to fix, though? > Can't we just do something like the following pseudo-code: [...] > (I've used /bin -> /usr/bin as an example, but the canonicalization must > deal with the other merged paths too.) > The second bit ensures that all new operations write canonicalized paths > (e.g. /usr/bin/foo) into the database regardless of whether the .deb > shipped with /bin/foo or /usr/bin/foo. This ensures the database stays in > sync with the filesystem moving forward. > The first bit performs a one-time canonicalization of paths in the > existing database. An on-disk flag ensures that this is run only once > (for performance reasons only, because it's already idempotent). This > corrects the existing database-filesystem mismatch. > The one-time canonicalization can be removed once non-usrmerged systems > are no longer supported. The second bit needs to live (probably a lot) > longer, until it's no longer reasonable to install a .deb containing > non-usrmerged paths (which might be old or from a third party). > Am I missing something here? I also don't understand why this isn't the correct solution. It seems obvious to me that we should simply teach dpkg about path aliases and have it update both its internal state and its processing of new packages to canonicalize paths so that it has an accurate representation of the world, and then ensure dpkg is upgraded first. Are we missing something? If not, what is blocking this solution? Is it simply a matter of someone digging into dpkg's code sufficiently to put these changes in the correct location? Or is there some other issue? It seems like a huge waste of resources to me to do archive-wide package inspection to try to find patterns that might cause problems, and to ask all maintainers to remember this rather obscure rule, when we could just fix dpkg to make the problem go away. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Crypto Libs: Linking to OpenSSL, GnuTLS, NSS, ..?
Stephan Verbücheln writes: > Then I also think that OpenSSL 0.9.x/1.x and the new OpenSSL 3.x have to > be treated like two completely different libraries. They have different > licenses and intentionally broke APIs to end the mess that OpenSSL > was. It is a situation like Python 2 and 3, we will have both around for > a long time, because upstream code has to be ported to new APIs. I don't think the changes in OpenSSL 3.x are anywhere near as significant as the changes between Python 2 and Python 3. They seem largely in line with the kinds of changes that are always required when there's a new major OpenSSL release (and are far, far less significant than, say, the differences between the OpenSSL and GnuTLS APIs). -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Crypto Libs: Linking to OpenSSL, GnuTLS, NSS, ..?
Michael Biebl writes: > What a coincidence. Just the other day I received > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=999375 in rsyslog. Yeah, this is exactly the sort of thing that I've seen with GnuTLS. While ideally one would work to debug and reproduce that problem and improve GnuTLS, switching to OpenSSL is a lot easier and makes most problems like this go away. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Crypto Libs: Linking to OpenSSL, GnuTLS, NSS, ..?
Alexander Traud writes: > Debian is very much OpenSSL. However, I see some packages default to > GnuTLS or even NSS without providing OpenSSL, although their source > project supports it. Historically, use of GnuTLS was mostly because of licensing restrictions because OpenSSL was incompatible with GPL-licensed code. Now, OpenSSL is compatible with GPL v3 and Debian has (with some controversy) adopted a policy of treating it like a system library even for GPL v2 code, so at least some of the GnuTLS usage has switched to OpenSSL. > Question(s): Is there a recommendation/guideline/policy that package > maintainers should prefer a specific crypto library (OpenSSL?) if they > cannot support all of them? If not, is there an argumentation aid to > convince package maintainers. I don't believe there is a policy. In practice, I believe OpenSSL tends to be more interoperable and better-tested upstream than GnuTLS. There have been long-standing problems with GnuTLS not handling weird corner cases or bugs in other libraries. Some of these do get fixed over time, but that's still my general impression. Also, if a software package was written to use OpenSSL, the OpenSSL compatibility layer in GnuTLS is very limited (I say this as someone who tried to use it for a package for several years) and tends to cause a lot of problems. NSS probably doesn't have the same interoperability problems. I personally have no opinions about using it. (Didn't Red Hat attempt to standardize on NSS a while back? I feel like that didn't work and they stopped that effort, but some quick searching didn't uncover any support for that belief.) -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Not running tests because tests miss source code is not useful
Jonas Smedegaard writes: > Right: It is ok to use upstream-provided pre-minified code, as long as > that code is DFSG-free, which requires the source of that code must > exist in Debian. > ...and because that is often complicated to ensure (not because it > violates DFSG in itself), it is easier to avoid upstream-provided > pre-minified code. Test suites are often a licensing mess. Another common case that's not in play here but that I've seen before is that long-standing projects that have been used commercially often have test snippets with unclear licensing that check for regressions that were previously seen in proprietary environments. Debian historically has erred on the side of maintaining clear source availability and licensing status for everything in Debian (which includes everything in any source package) at the cost of not availing ourselves of test suites that would otherwise be useful. That's unfortunately probably the easy path here as well, until someone has time to find non-minified versions of the test dependencies and either package them or include them in the package. It's frustrating to remove the tests, but the DFSG source requirements as currently applied do not distinguish between code shipped only in source packages and code also shipped in binary packages. I can see an argument that we should not worry about minified files in main that are (a) only in the source package and not in any binary package, and (b) only used to run tests, not to build the binary packages. (I'm not saying I agree or disagree, just that I can see the argument.) But given the apparent consensus on this in past discussions, I suspect that changing that rule would be GR material rather than debian-devel thread material. Making that sort of change without a GR to be sure the project is behind it feels like the kind of thing that's likely to spawn endless arguments that will sap everyone's will to live. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Not running tests because tests miss source code is not useful
Richard Laager writes: > I haven't looked into the specifics of this situation, but in general, > tests should be run against the same versions of dependencies that the > actual code will use, for what should be obvious reasons. If Debian has > the dependencies with different API versions, then it's all the more > important that the tests run against the Debian versions of the > dependencies. I believe the dependencies in question are test dependencies. In other words, they're dependencies required to drive the test suite machinery, not dependencies of the code under test. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
Re: Q. What is the best practice about +dfsg and +ds extension?
Kentaro Hayashi writes: > What do you think about it? > 1. We should use +dfsg-1 style > 2. We should use +dfsgN-1 style > 3. We should use +dfsg.N-1 style > 4. Other I would start with +dfsg-1 because it's fairly rare to have to iterate on the repackaging. You can then switch to +dfsgN-1 with the second and subsequent repackagings if needed. (Although if I knew in advance I would probably need to iterate, I'd start with +dfsgN-1.) There's an argument for consistency to always use +dfsgN-1, I guess, but I don't think it matters enough to bother. I would not use +dfsg.N-1. It's not consistent with the other places where we add suffixes, such as backporting and stable updates. -- Russ Allbery (r...@debian.org) <https://www.eyrie.org/~eagle/>
