Re: Huh?? sbuild fails and pbuilder succeeds in building a Fortran-containing Python package

[Johannes Schauer Marin Rodrigues]

Quoting Sebastian Ramacher (2021-02-11 10:00:42)
> > > A workaround for this problem is indeed to add
> > > 
> > > --chroot-setup-commands="chmod 777 /dev/shm"
> > > 
> > > to your sbuild invocation. A cursory glance into the pbuilder codebase 
> > > reveals
> > > that pbuilder will mount a tmpfs into /dev/shm:
> > > 
> > > https://sources.debian.org/src/pbuilder/0.231/pbuilder-modules/?hl=417#L417
> > > 
> > > For sbuild, whether this is done, this depends on your chroot backend. The
> > > package builds fine on the buildds though, so I guess they have a schroot 
> > > setup
> > > that sets /dev/shm up correctly?
> > 
> > OK, so I'll submit this as a bug report / feature request to sbuild.
> 
> schroot already does the same in the sbuild and buildd profiles:
> 
> https://sources.debian.org/src/schroot/1.6.10-11/etc/profile-templates/buildd/linux/fstab/
> https://sources.debian.org/src/schroot/1.6.10-11/etc/profile-templates/sbuild/linux/fstab/
> 
> Is your setup using any of the two profiles?

It seems it is. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982518#13

Lets continue figuring this problem out via the BTS.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Split Packages files based on new section "buildlibs"

[Johannes Schauer Marin Rodrigues]

Quoting Andrej Shadura (2020-11-10 10:27:44)
> On Tue, 10 Nov 2020, at 08:28, Paul Wise wrote:
> > > The current proposal is to reduce the main Packages.xz files size by
> > > splitting[4] out all of the packages that are not intended for users,
> > > writing those into an own file. Those packages would have a section of
> > > "buildlibs", independent of their other properties.
>  
> > Should (almost?) everything in the existing libdevel section move to
> > the new buildlibs section?
> 
> I wouldn’t say so.
> 
> If I install, say, libftdi-dev, I expect to be able to do actual development
> with it, for Debian or not. In fact, installing libftdi-dev would be the
> first thing I do if I were to develop with the library.
> 
> On the contrary, if I’m going to do some development with, say, clap (Rust
> command-line arguments parser), I wouldn’t install librust-clap-dev; more
> than that, if I actually did, I’d be very difficult for me to actually use it
> to develop an app.

Aha. Would it?

I have the following in my ~/.cargo/config.toml:

[source.deb]
directory = "/usr/share/cargo/registry"

[source.crates-io]
replace-with = "deb"

[net]
offline = true

Then I clone some upstream git repo that uses clap:

git clone https://github.com/dotenv-rs/dotenv.git

And then I run "cargo build". Every time I get a message like:

error: no matching package named `foo` found

I install librust-foo-dev until finally:

Parent pid 535147, child pid 535148
Child process initialized in 30.93 ms
   Compiling proc-macro2 v1.0.18
   Compiling unicode-xid v0.2.0
   Compiling syn v1.0.12
   Compiling dotenv v0.15.0 (/tmp/dotenv/dotenv)
   Compiling quote v1.0.7
   Compiling proc-macro-hack v0.5.9
   Compiling dotenv_codegen_implementation v0.15.0 
(/tmp/dotenv/dotenv_codegen_implementation)
   Compiling dotenv_codegen v0.15.0 (/tmp/dotenv/dotenv_codegen)
Finished dev [unoptimized + debuginfo] target(s) in 9.93s

Parent is shutting down, bye...

So how is this "very difficult"? The steps are the same as when I clone a
Python upstream git repo and I get the message "ModuleNotFoundError: No module
named 'foo'" -- I just install python3-foo and it will work. Same here with
rust and the librust-foo-dev packages.

I do not deny that many upstreams will tell developers to use cargo, pip, go
get... whatever to manage their software. Personally, I rather avoid using
these package managers and use the packages provided by Debian instead. There
are real advantages over using the packages from Debian. Instead of relying on
another 3rd party repository where anybody can upload anything, I benefit from
the additional work put in by DDs to make sure that no garbage ends up in the
archive. Rather than the "fast-paced" development style that seems to be modern
these days, I prefer the stability and security that I get by sourcing all my
code and libraries from the Debian archive.  With Debian packages, I cannot
really get typosquatted, I know that all software is DFSG-free and I know that
I will receive security updates.

This is not an argument against splitting "main" into several archives. I'm no
big fan of this solution but maybe it should be done. What I want to make an
argument for in this email though is, that I do not believe that our packages
have no value outside compiling other Debian packages even for languages where
upstream prefers its users to use their own package manager. There are many
reasons to prefer the trusted Debian sources for quality, security and software
freedom and if any split is done, it should not be made in a way that makes it
harder for our users to install those packages. They have real value.

At this point let me also give a big thank you to all the nodejs, python, rust
etc package maintainers for taking on the tedious task of making Debian
packages for software that is already present in another repo. It's really
great stuff -- thanks a lot!

cheers, josch

signature.asc
Description: signature

Re: base-passwd marked as Essential: yes but other packages depend on it being configured.

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Tim Woodall (2021-02-21 18:22:19)
> base-passwd is marked as Essential: yes
> 
> However, it actually creates the initial passwd and group files in the preinst
> script.

this reminds me of:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963788

Would this problem not also be solved by dpkg taking care of managing
/etc/passwd and /etc/group? If it could do that, we could solve the
reproducibility issue in the bug above *and* base-passwd would not anymore have
to do anything in its preinst script but could rely on dpkg doing the right
thing, maybe helped by some new declarative statements in the package?

Since dpkg 1.20.0, software like debootstrap or mmdebstrap do not anymore need
to create /var/lib/dpkg/* or /etc/dpkg/dpkg.cfg.d/ because dpkg will take care
of it. So maybe another approach would be to also let dpkg take care of the
initial bootstrapping stuff like making sure /etc/passwd has some sensible
content so that software like debootstrap and mmdebstrap can rely on it?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: base-passwd marked as Essential: yes but other packages depend on it being configured.

[Johannes Schauer Marin Rodrigues]

Quoting Tim Woodall (2021-02-21 20:07:17)
> What I'm trying to do is build in a fakechroot. My idea was to unpack the
> core packages, fakeroot fakechroot chroot image apt-get install apt and have
> a base system to install build-deps etc.
> 
> However, apt does not configure packages in the needed order due to this
> passwd/group issue.
> 
> (debootstrap does not support fakechroot minimal system and I thought
> debootstrap was perl so I didn't want to try to modify it)

Are you by any chance trying to do the same as:

mmdebstrap --variant=apt --mode=fakechroot unstable /path/to/chroot

Thanks!

cheers, josch

signature.asc
Description: signature

Re: New service: https://debuginfod.debian.net

[Johannes Schauer Marin Rodrigues]

Quoting Ian Campbell (2021-02-24 18:50:39)
> What are the security implications for users/clients of using this or more
> importantly enabling it by default?
> 
> Presumably clients have to trust that the server is not going to feed
> them malicious debug info. Are the tools which consume this information
> written to operate on completely untrusted inputs? It seems like many
> of them could have been written historically with the assumption that
> their inputs are mostly to be trusted. I suppose the use https helps
> mitigate this at least a bit when it comes to a debian.{org,net}
> service.
> 
> What about information leakage? apart from debugids does this leak
> anything else to the server? On a quick look it seems like it might
> potentially leak source code paths (at least the leaf bits) to things
> being debugged -- does this mean that if a user is debugging private
> software (perhaps unpublished or perhaps proprietary software for
> $work) on a Debian system they are at risk of leaking the source
> filenames if they run gdb on one of their binaries while debugging?  This
> might be a problem if it comes to enabling this transparently.

This sounds like it should be treated in a similar way as we treat submissions
to popcon.debian.org where we ask the user explicitly and which is not getting
enabled unless with explicit consent by the user.

Thanks!

cheers, josch

signature.asc
Description: signature

Unsolicited internet access in default installs (was: New service: https://debuginfod.debian.net)

[Johannes Schauer Marin Rodrigues]

Quoting Steinar H. Gunderson (2021-02-27 13:46:27)
> On Sat, Feb 27, 2021 at 12:29:34PM +, Thaddeus H. Black wrote:
> > I would prefer Kurt's option.  Network silence is important.  Network
> > noise would probably be a bug.  A sysadmin should not be made to take
> > special precautions to avoid the inadvertent disclosure of the user's
> > presence on the network.
> 
> It's 2021; machines are not silent on the network. That ship sailed long ago.

I was about to write a rant, stating that it's unacceptable for any requests to
remote servers to be made by a Debian default installation without my explicit
consent. So I ran:

qemu-system-x86_64 -enable-kvm -m 2G -netdev user,id=u1 -device e1000,netdev=u1 
-object filter-dump,id=f1,netdev=u1,file=install.pcap -cdrom 
debian-testing-amd64-DVD-1.iso -hdd disk.img

And found out that I was wrong and you are right. Even though I answered "No"
to all questions related to mirrors and popcon during the installation, the
above command still recorded a DNS query for debian.map.fastlydns.net and a
subsequent download of /debian-security/dists/bullseye-security/InRelease.

Later on, after the installation had finished and gnome started, I see requests
to cdn.fwupd.org where something got downloaded from and then some
communication with 0.debian.pool.ntp.org.

No idea why we are still asking whether popcon should be enabled or not because
apparently it's 2021 and it's okay to tell others out there that I just
installed Debian.

signature.asc
Description: signature

Re: sbuild and bind mounts

[Johannes Schauer Marin Rodrigues]

Hi Norbert,

Quoting Norbert Preining (2021-03-17 03:55:53)
> in the Qt/KDE Team we use a script for test-building our groups of packages.
> Those groups need to be build in tiers (levels) so that later packages have
> access to the build of earlier levels.  The script we currently use is based
> on cowbuilder with --bindmounts to get the package repo of hitherto built
> packages into the build process, and hook scripts to active the respective
> apt sources line.

with sbuild you can use --extra-package=package.deb or
--extra-package=/path/to/debs and then sbuild will use those debs or all debs
in the directory you specify, respectively, to satisfy dependencies and you
don't need to manage an apt repository yourself.

> Recently I started using btrfs snapshots with sbuild (and eatmydata)
> because it is sooo much faster than cowbuilder, so I would like to add
> sbuild support to our tier-build script.
> 
> My questions are how to achieve:
> - bind-mount directories into the sbuild chroot

This depends on the chroot backend you are using. The default backend is
schroot and you probably want a solution for that one and not for the
autopkgtest or unshare backend. With schroot you would add additional mount
points by editing /etc/schroot/sbuild/fstab

> - add lines to /etc/apt/sources.list
> before the build process starts?

You can either use this sbuild option:

--extra-repository="deb http://deb.debian.org/debian experimental main"

or you can use the option

--chroot-setup-commands="echo 'deb ...' >> /etc/apt/sources.list"

which runs arbitrary shell scripts inside your chroot.

All of the above is documented in the sbuild man page but there is lots of
stuff in there so I can understand why it's hard to find things without reading
the whole thing first. Anyways, if you see ways how to improve the man page,
I'd like to hear about them. Other than that there is
https://wiki.debian.org/sbuild which collects common recipes of how to use
sbuild in different scenarios, so feel free to add things you find useful to
it.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Tips for debugging/testing debian/control Depends/Breaks etc changes?

[Johannes Schauer Marin Rodrigues]

Hi Otto,

Quoting Otto Kekäläinen (2021-03-24 17:37:46)
> I've noticed I've spent quite a lot of time debugging various situations
> where the debian/control definitions for
> depends/breaks/replaces/conflicts/provides are not optimal.
> 
> The waste of time is two-fold:
> 
> 1) apt is not verbose enough
> 2) the cycle to rebuild/tests is too slow
> 
> As an example of 1, sometimes I see this:
> 
> apt install mariadb-client
>  The following packages have unmet dependencies:
>  mariadb-client : Depends: mariadb-client-10.5 (>= 1:10.5.10) but it
> is not going to be installed
> 
> apt install mariadb-client-10.5
>  Installing.. Done!
> 
> When this happens I have no idea why apt did not resolve the
> dependency by itself automatically, as there was no real conflict in
> installing it.
> 
> Do you have tips on how to debug the root cause of situations like these?
> 
> 
> For the problem 2, I hate to rebuild all of the packages (and
> binaries) just because there was a change in debian/control and go
> through the hassle of updating a test repo etc.
> 
> I wonder if there is some other "lighter" way to just edit the
> debian/control and produce new binary packages with them updated
> without having to actually build new binary packages (and no, editing
> the .deb files manually and repackaging them with 'ar' is not an option that
> would make life easier).

there is a way to throw arbitrary installation problems at the apt solver
without building any *.deb packages and without creating any repos with
Packages and Release files: by using /usr/lib/apt/solvers/apt directly. You do
it by going through the following steps:

1. Create an initial EDSP document that represents the current dependency
   situation. You can do this by running something like this:

   APT_EDSP_DUMP_FILENAME=/tmp/dump.edsp apt-get --simulate install 
--solver dump ghc

2. Open /tmp/dump.edsp in an editor and edit whatever you need. For example
   with the command above the first stanza will contain:

  Install: ghc:amd64

   And you probably want to change that into whatever you want to ask apt to
   install. Then go through the file and find the stanza for the package that
   you want to change the depends/breaks/replaces/conflicts/provides of and
   change whatever you like.

3. Evaluate the EDSP problem using the apt solver by calling:

  /usr/lib/apt/solvers/apt < /tmp/dump.edsp

   You can improve the output by throwing in options like:

  -oDebug::EDSP::WriteSolution=yes

   Or the options that David Kalnischkies mentioned in his mail.

The advantage is, that you only have to edit one file (the EDSP) and only have
to run one command (/usr/lib/apt/solvers/apt) to check what happens when you
change something. No need to build any actual Debian package or create a
repository with them.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Packages in contrib solely because they allow using non-free software

[Johannes Schauer Marin Rodrigues]

Quoting Simon McVittie (2021-04-04 13:55:21)
> On Sun, 04 Apr 2021 at 13:23:14 +0200, Joerg Jaspert wrote:
> > On 16093 March 1977, Dominik George wrote:
> > > That surprised me. If a package is free software, in ful laccordance
> > > with the DFSG, why is it put into contrib?
> > 
> > There is, as usual, no clear answer.
> > 
> > The policy for main is clear on that it needs to be self contained. So
> > software in main must not require something outside to work and do its job.
> > Contrib is the area where that is allowed. License wise its the same as
> > main, but it allows to depend on something not available for building or
> > working.
> 
> There was some discussion relevant to this on debian-devel-games earlier
> this year: the subthread starts at
> .
> 
> I don't think the rule can be as simple as "must not require something
> outside to work and do its job", because we have Free clients for non-Free
> network services (like all the instant messaging services that used to
> exist), and those were always in main. Similarly, it would be absurd to
> kick out email clients into contrib just because they are primarily used
> to read non-Free email messages like this one! :-)
> 
> As I mentioned on d-d-games, one of the major things I tend to ask myself
> when thinking about the borderline between main and contrib is: if the
> content that this package downloads was somehow in the Debian archive,
> would the downloader have a Depends or Recommends on it, or would it be
> a Suggests or no dependency at all?
> 
> Another factor in choosing main or contrib, for me, is whether the
> downloader is specifically hard-coded to work with particular content,
> or whether it generically works with any content in a particular
> format. game-data-packager and quakespasm are both close to the main/contrib
> borderline, and this factor is why I think they are in the correct archive
> areas: g-d-p downloads and repackages specific non-Free games, so it's
> in contrib, whereas quakespasm can play any Free or non-Free set of
> Quake-compatible levels, so it's in main.
> 
> I think winetricks is *probably* correctly in contrib, because it has
> hard-coded knowledge of how to best to download and install specific
> non-Free Windows DLLs? Like game-data-packager, it's quite close to the
> line between main and contrib, but I think it does make sense to consider
> it to be on the contrib side of that line (although I'm sure I could be
> convinced otherwise).
> 
> As Joerg says, this is all fairly subjective and unclear, but I think we
> do have an approximately coherent policy for what can and can't be in main,
> and that's realistically the best thing we are going to get.

another way to answer the question is to find some software similar to the one
that you want to package and see if that software is in Debian main or in
contrib. If it is in main, then at least one DD and FTP master already agreed
that packages of that nature are fit for main.

Maybe the comparison to winetricks is not very fitting because I think
winetricks can *only* download non-free software while lutris can also download
games that their developers distribute under a DFSG license?

Let me show you a package of mine that is still in Debian main even though it
is a "downloader [that is] specifically hard-coded to work with particular
[non-free] content" and thus fails Simon's test from above:

https://packages.debian.org/source/unstable/rss-bridge

I guess most people will use that package to access non-free services like
Facebook, Instagram or YouTube. After all, rss-bridge was specifically written
to work around the limitations of these non-free services:

https://github.com/RSS-Bridge/rss-bridge/#rant

Should anybody ever file an RC bug against the package I will point out that it
is also able to work with DFSG free software like Mediawiki. Maybe you can do
the same?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Need help with Multi-Arch in systemd

[Johannes Schauer Marin Rodrigues]

Quoting Helmut Grohne (2021-07-09 14:24:01)
> Another possibility (kudos to David Kalnischkies) would be exploiting
> something I might call a loophole in the Multi-Arch spec. While we don't
> currently use arch-qualified dependencies in the archive, the spec considers
> them and dpkg and apt support them (somewhat). So in theory, we could say
> that systemd-container Depends: systemd:${DEB_HOST_ARCH}.  I'm not sure
> whether all resolvers agree on this, but I'd expect this dependency to pull
> the right instance (despite being marked Multi-Arch: foreign). Before jumping
> onto this, please consider David's warning: here be dragons.

No, not all solvers agree on this. Imagine this situation:

Package: pkga
Architecture: amd64
Depends: pkgb:i386
Multi-Arch: no

Package: pkgb
Architecture: i386
Multi-Arch: foreign

dose3, apt and dpkg agree that pkga can be installed because its dependency is
satisfied by pkgb. But if we change pkgb:i386 to pkgb:amd64 to end up with
this:

Package: pkga
Architecture: amd64
Depends: pkgb:amd64
Multi-Arch: no

Package: pkgb
Architecture: i386
Multi-Arch: foreign

Then dose3 can satisfy the dependency of pkg while apt and dpkg cannot. This is
because when building the cudf representation of above two packages, dose3
considers pkg to provide pkgb for all architectures, including amd64, because
it is marked as m-a:foreign.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: merged /usr considered harmful (was Re: Bits from the Technical Committee)

[Johannes Schauer Marin Rodrigues]

Quoting Michael Biebl (2021-07-19 15:10:42)
> Am 19.07.21 um 03:36 schrieb Guillem Jover:
> > What I've also said multiple times, is that
> > merged-usr-via-moves-and-symlink-farms could have been implemented in
> > a fully automated way, by debhelper, w/o requiring any maintainer scripts,
> > all with full cooperation and managed by dpkg, with .debs shipping
> > actual tracked pathnames
> I'm convinced this view is way too naive and not implementable in 
> practice (and yes, openSUSE is a data point that confirms that)
> 
> What you propose is, that each and every package does its /usr-merge 
> transition on its own. This only works, if packages are independent 
> (enough) so this actually works.
> Unfortunately this is not the case. Take PAM for example. You can't just 
> recompile src:pam and have debhelper automatically move all files to 
> /usr. This would break all packages that install a PAM plugin. You have 
> a transition here, involving many packages.
> Same is true for udev rules, systemd service files, basically every 
> package that provides interfaces/hooks to other packages is affected.
> So it's not that simple unfortunately. You can't fully automate that.
> 
> According to
> apt-file search -x '^/(lib|bin|sbin)'
> on my Debian sid/amd64 system, we have 1747 packages shipping 24583 
> files in those directories.

more precisely, on amd64 unstable:

/bin 247 files
/lib{32,64,x32} 83 files
/lib/firmware 2379 files
/lib/live 115 files
/lib/modules 17500 files
/lib/systemd 2221 files
/lib/udev 614 files
/lib/x86_64-linux-gnu 343 files
/lib/* 441 files
/sbin 547 files

So most files come from /lib/modules, where only 14 packages are involved,
/lib/systemd which will be fixed by an update to dh_installsystemd, and
/lib/firmware where only 36 packages are involved. The remainder then doesn't
sound so scary anymore as it only involves 656 unique packages and not 1747.
And again many of those remaining packages will be fixed by an update to
debhelper, correct? Given that 90% of source packages use dh, that would reduce
the number to a very manageable size.

> There are *many* such entangled transitions 
> hidden in there, so I fear this is not manageable.
> As Luca pointed out, even distros with a much stricter governance model 
> were not able to do that.
> 
> The /usr-merge transition as described and decided on in the TC bug, 
> seems to me is the only viable way forward.
> 
> Yes, it does break dpkg -S, but your idea of using a list of mapped 
> paths as in [1] seems like an entirely reasonable approach to solve this.
> 
> Once we have this global switch to merged-usr, packages can bit by bit, 
> completely independent, update their debian/rules to use --prefix=/usr 
> and after a few years, we don't have any packages anymore installing any 
> files in /. We could aid this process with a lintian check that flags
> packages that install files in /(sbin|bin|lib)/.

So what what is actually the roadmap after the bullseye release? What is the
way forward? Should I rather file bugs with patches against individual packages
to move their files from /(sbin|bin|lib)/ to /usr/(sbin|bin|lib)/ or do we
already have a debhelper patch to do that move for us?

This would mean that we only have to bear with the problems mentioned by
guillem until that move is complete, correct?

And another question: once there are no more files shipped by any package in
/(sbin|bin|lib)/ we can let base-file create the symlinks?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Debian Policy 4.6.0.0 released

[Johannes Schauer Marin Rodrigues]

Quoting Sean Whitton (2021-08-18 22:21:15)
> On Wed 18 Aug 2021 at 11:10AM +02, Aurelien Jarno wrote:
> 
> >> 9.1.1
> >> No package is allowed to install files in ``/usr/lib64/``. Previously,
> >> this prohibition only applied to packages for 64-bit architectures.
> >
> > This path is used by the multilib 64-bit toolchain on 32-bit
> > architectures, for instance libc6-amd64:i386, or a few essential
> > libraries like ncurses.
> >
> > What kind of fix do you expect on the packages? Is it finally the time
> > to get rid of multilib and use pure multiarch instead?
> 
> We did not intend to make any packages buggy with this change.  It was
> thought to be just a clarification.
> 
> Russ, perhaps we should revert this?

Or maybe it is finally the time to get rid of multilib and use pure multiarch
instead?

signature.asc
Description: signature

Re: finally end single-person maintainership

[Johannes Schauer Marin Rodrigues]

Quoting Bernd Zeimetz (2024-05-21 00:54:07)
> On Wed, 2024-04-10 at 23:16 +0200, Johannes Schauer Marin Rodrigues
> wrote:
> > Quoting Andreas Tille (2024-04-10 22:44:25)
> > > > I do understand the argument that lots of different workflows
> > > > adds
> > > > friction. But I'm just still using what used to be _the_ standard
> > > > one
> > > > (insofar as we ever had such a thing). Putting everything in
> > > > salsa/git
> > > > doesn't standardise workflows in itself. I think Ian/Sean
> > > > identified 12
> > > > different git-based methods in their dgit review.
> > > 
> > > I agree that different workflows are not helpful.  We have DEP14[1]
> > > ... but we have no efficient processes to
> > >   a) accept DEPs
> > >   b) dedicate to accepted DEPs
> > 
> > or teach gbp about DEP14. See this git-buildpackage bug from *six*
> > years ago:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=829444
> 
> DEP14 is a candidate, I can't see that there was any consensus to accept it.
> Just because there is a DEP there is no need to implement it without having
> any consensus on it.

the current gbp defaults were also implemented without any consensus on them,
so in that regard, embracing DEP14 would not make the situation worse.

What is improved by DEP14 is that in contrast to the current gbp defaults, it
is backed up by a write-up of advice, best-practices and recommendations. We
should have more of these write-ups for the things we do. This would also make
it easier for new contributors to get into Debian.

Thanks!

cheers, josch

signature.asc
Description: signature

Documenting packaging workflows (was: finally end single-person maintainership)

[Johannes Schauer Marin Rodrigues]

Quoting Luca Boccassi (2024-05-22 01:45:54)
> On Wed, 22 May 2024 at 00:40, Russ Allbery  wrote:
> > For what it's worth, what I do for the packages for which I'm also
> > upstream is that I just add Salsa as another remote and, after I upload
> > a new version of the Debian package, I push to Salsa as well (yes,
> > including all the upstream branches; why not, the Debian branches are
> > based on that anyway, so it's not much more space).  One of these days
> > I'll get CI set up properly, and then it will be worthwhile to push to
> > Salsa *before* I upload the package and let it do some additional
> > checking.
> >
> > It's still an additional step, and I still sometimes forget to do it, but
> > after some one-time setup, it's a fairly trivial amount of work.
> >
> > It's more work to accept a merge request on Salsa and update the
> > repositories appropriately, since there are two repositories in play, but
> > in that case I'm getting a contribution out of it that I might not have
> > gotten otherwise, so to me that seems worth it.
> >
> > I used to try to keep the debian directory in a separate repository or try
> > to keep the Debian Git branches in a separate repository, and all of that
> > was just annoying and tedious and didn't feel like it accomplished much.
> > Just pushing the same branches everywhere is easy and seems to accomplish
> > the same thing.
> 
> Yeah I am doing the same, and gradually switching all my packages that
> used to have a separate upstream/downstream history to a single merged
> tree. This can be done post-facto with some one-time git rocket
> surgery, doesn't have to be the case from day one. It's a huge
> improvement, and with gpp patch-queue I can just cherry-pick upstream
> commits directly, with no hassle whatsoever. It works really nicely,
> and gbp supports it just fine as a workflow, even while still checking
> in upstream release tarballs in pristine-tar.

I would be *very* interested in more in-depth write-ups of the workflows other
DDs prefer to use, how they use them and what they think makes them better than
the alternatives.

Personally, I start packaging something without git, once I'm satisfied I use
"gbp import-dsc" to create a packaging git with pristine-tar (and that will
*not* have DEP14 branches and it will use "master" instead of "main") and then
I push that to salsa and do more fixes until the pipeline succeeds and lintian
is happy. My patches are managed using quilt in d/patches and upstream git is
not part of my packaging git. I upload using "dgit --quilt=gbp push-source".

Would another workflow make me more happy? Probably! But how would I get to
know them or get convinced that they are better? Maybe I'm missing out on
existing write-ups or video recordings which explain how others do their
packaging and why it is superior?

Any hints or future debconf workshop invitations welcome. :)

Thanks!

cheers, josch

signature.asc
Description: signature

Re: changing existing entries in debian/changelog

[Johannes Schauer Marin Rodrigues]

Quoting Andrey Rakhmatullin (2024-05-24 09:37:45)
> On Fri, May 24, 2024 at 09:30:44AM +0200, Bastian Venthur wrote:
> > I'm having troubles finding the relevant parts in the developers reference.
> > I've uploaded a version to experimental and later found out that this
> > version fixes several bugs.
> > 
> > Can I rewrite existing changelog entries for already uploaded versions with
> > the next upload, i.e. by adding the relevant "closes" line to the previsions
> > version?
> You can if you want but it won't close the bugs, you'll still need to
> close them properly with manual action. SO it may be easier to just not edit
> them.

When I forget to add the "closes:" line to d/changelog, I will add it
retroactively (after having closed the bugs with the correct version manually)
because it allows others as well as future-me to look up in d/changelog to
which changes correspond to which bugs and bugs will have much more context for
a problem than a line in d/changelog.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: changing existing entries in debian/changelog

[Johannes Schauer Marin Rodrigues]

Quoting Bill Allombert (2024-05-24 10:54:32)
> Le Fri, May 24, 2024 at 09:30:44AM +0200, Bastian Venthur a écrit :
> > Hi,
> > 
> > I'm having troubles finding the relevant parts in the developers reference.
> > I've uploaded a version to experimental and later found out that this
> > version fixes several bugs.
> > 
> > Can I rewrite existing changelog entries for already uploaded versions with
> > the next upload, i.e. by adding the relevant "closes" line to the previsions
> > version?
> 
> You need to use the option -v of dpkg-genchanges so that the relevant
> bugs are listed in the .changes file, so that they get automatically closed.

But then they get closed by the wrong version, no? Dev-ref says:

https://www.debian.org/doc/manuals/developers-reference/pkgs.en.html#when-bugs-are-closed-by-new-uploads

> To close any remaining bugs that were fixed by your upload, email the
> .changes file to xxx-d...@bugs.debian.org, where XXX is the bug number, and
> put Version: YYY and an empty line as the first two lines of the body of the
> email, where YYY is the first version where the bug has been fixed.

When I don't have the .changes file anymore, I just mail
xxx-d...@bugs.debian.org with a mail that has the "Version: YYY" line at the
top and then in the body explains that I'm manually closing it because I made a
mistake.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-28 01:54:08)
> Thanks for the useful input, the following has been done:
> 
> - existing installations pre-trixie will get an orphaned tmpfiles.d in
> /etc/ that keeps the existing behaviour unchanged (no cleanup of
> /var/tmp)
> - openssh and tmux have been fixed to provide a tmpfiles.d exception
> to retain their respective files
> - the /tmp/ description in debian-installer has been updated to note
> it is a tmpfs by default (via a commit in partman-basicfilesystems,
> will upload if nobody gets around to it before Trixie's freeze)
> - two paragraphs have been provided for the release notes ticket
> - the changes are also noted in NEWS, with instructions on how to
> override locally
> - as mentioned, the latest upload to unstable makes /tmp/ a tmpfs by
> default and for new installations 10+ days old files in /tmp/ and 30+ days
> old files in /var/tmp/ are cleaned up daily

thank you for having discussed this change on d-devel and for adding
documentation to NEWS and release notes to announce this change. I also think
it is sensible to roll this only out on new installations and to keep the
behaviour on existing setups. Thank you for implementing that as well.

That being said, maybe some Perl wizard knows how to do a flock on a directory
in Perl? I tried this:

use Fcntl qw(LOCK_EX);
opendir(my $handle, ".") or die "opendir: $!";
flock($handle, LOCK_EX) or die "flock: $!";

And got this:

flock() on unopened filehandle $handle at test.pl line 8.
(Are you trying to call flock() on dirhandle $handle?)
flock: Bad file descriptor at test.pl line 8.

Wrapping $handle in fileno() doesn't make a difference either. Perl sources
seem to indicate that directory handles are "evil_fh"?

https://sources.debian.org/src/perl/5.38.2-4/util.c/?hl=3783#L3783

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Peter Pentchev (2024-05-30 10:33:08)
> > thank you for having discussed this change on d-devel and for adding
> > documentation to NEWS and release notes to announce this change. I also
> > think it is sensible to roll this only out on new installations and to keep
> > the behaviour on existing setups. Thank you for implementing that as well.
> > 
> > That being said, maybe some Perl wizard knows how to do a flock on a 
> > directory
> > in Perl?
> 
> I wouldn't call myself a Perl wizard by a long stretch, but I can give it a 
> try :)
> 
> >  I tried this:
> > 
> > use Fcntl qw(LOCK_EX);
> > opendir(my $handle, ".") or die "opendir: $!";
> [snip]
> 
> Here lies your problem. The flock(2) syscall works on file descriptors,
> the things returned by open(2), not on the dirent structures returned by
> opendir(3). So you need something like this:
> 
> use v5.10;  # I really should switch to at least 5.16 if not 5.24
> use strict;
> use warnings;
> 
> use Fcntl qw(O_RDONLY O_DIRECTORY LOCK_EX);
> 
> my $dirname = "/tmp/to-be-locked";
> sysopen(my $handle, "/tmp/to-be-locked", O_RDONLY | O_DIRECTORY) or
> die "Could not open $dirname: $!\n";
> flock($handle, LOCK_EX) or
> die "Could not lock $dirname: $!\n";
> 
> say "locked, it seems";
> sleep(3600);'
> 
> I only put the sleep() part so I could check using lsof that
> the directory was indeed locked. And yeah, the v5.10 part is a leftover
> from the days (...until a month or two ago...) when I still had to
> support stock CentOS 7 systems; I really should train my fingers to
> put 5.24 there.
> 
> Hope that helps!

it absolutely does! Thank you! I was misled by `perldoc -f flock` which states
that it works on filehandles. I'll add your name to the git commit message
unless you object.  :)

I also found another issue with this change in systemd. After the upload to
unstable, 76 out of 264 mmdebstrap tests on jenkins.debian.net started to fail:

https://jenkins.debian.net/job/mmdebstrap-jenkins-worker/692/consoleText

The problem is, that debootstrap wants to mknod which will not work on a tmpfs
mounted with nodev:

+ debootstrap --no-merged-usr --variant=buildd oldstable /tmp/tmp.nWmx8YeAh3 
http://127.0.0.1/debian
/usr/sbin/debootstrap: 1840: cannot create /tmp/tmp.nWmx8YeAh3/test-dev-null: 
Permission denied
E: Cannot install into target '/tmp/tmp.nWmx8YeAh3' mounted with noexec or nodev

Maybe this affects more CI scripts and test setups which attempt to create a
temporary chroot with debootstrap in /tmp.

The fix which is documented in systemd NEWS makes everything work again:

--customize-hook='touch "$1/etc/systemd/system/tmp.mount"'

Thanks!

cheers, josch

signature.asc
Description: signature

Re: DEP17 /usr-move: debootstrap set uploaded

[Johannes Schauer Marin Rodrigues]

Hi Helmut,

Quoting Helmut Grohne (2024-06-06 09:28:52)
> I have just uploaded
>  * base-files
>  * bash
>  * dash
>  * glibc
>  * util-linux
> to unstable. These were the last remaining packages shipping aliased
> files inside the package set relevant to debootstrap.

thank you (and freexian for funding you) so much for finally reaching this
milestone!! Thank you also for doing all your work with incredible diligence
and attention to detail. This is really outstanding and what I consider to be a
model for how changes in Debian should be performed.

Your upload timing seems to have been great: the buildds seem to already have
gone through most of what you uploaded.

I cannot wait for the next dinstall in ~5 hours to test what you uploaded with
the mmdebstrap test suite (which is also testing debootstrap).

Too bad that it is just these days that snapshot.d.o is moving to the new
infrastructure (a big thanks to the team who is working on that) which means
that this crucial transition in unstable will not be part of
snapshot.debian.org. There are no timestamps recorded for June yet.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Simon Richter (2024-06-06 11:32:33)
> > Would it be possible to set in stone that packages are supposed to always
> > be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
> > builders must set LC_ALL=C.UTF-8?
> 
> This would be the opposite of the current rule.
> 
> Setting LC_ALL=C in debian/rules is an one-liner.
> 
> If your package is not reproducible without it, then your package is
> broken. It can go in with the workaround, but the underlying problem
> should be fixed at some point.
> 
> The reproducible builds checker explicitly tests different locales to
> ensure reproducibility. Adding this requirement would require disabling this
> check, and thus hide an entire class of bugs from detection.

this is one facet of a much bigger discussion (which we've had before). You can
argue both ways, depending on how you look at this problem.

It is the question of whether we want to:

 a) debian/rules is supposed to be runnable in a wide variety of environments.
 If your package FTBFS in a one specific environment, it is the job of d/rules
 to normalize the environment to cater for the specific needs of the package.

 b) debian/rules is supposed to be run in a well-defined environment. If your
 package FTBFS in this normalized environment, then it is the job of d/rules to
 add the specific needs of the package to d/rules.

So the question is whether you either want to have d/rules normalize
heterogeneous environments (a) or whether you want d/rules to make a normalized
environment specific to the build (b). This is of course a spectrum and I think
we currently doing much more of (a).

A question that goes in a similar direction is whether every d/rules that needs
it should have to do this:

export DPKG_EXPORT_BUILDFLAGS=y
include /usr/share/dpkg/buildflags.mk

Or whether we should switch the default and require that d/rules is run in an
environment (for example as set-up by dpkg-buildpackage) where these variables
are set?

Going back to the example of LC_ALL=C.UTF-8 and reproducibility: whether or not
this "hides" problem depends on the definition of what things are allowed to
change between two builds and what constitutes these things has changed already
in the past, for example for the build path which is not *not* changed anymore
but instead recorded in the buildinfo. The same could be argued for
LC_ALL=C.UTF-8 and the environment variables already are part of the buildinfo.

So I do not think that there is an easy answer to this question.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Hakan Bayındır (2024-06-06 12:32:27)
> On 6.06.2024 ÖS 1:08, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Simon Richter (2024-06-06 11:32:33)
> >>> Would it be possible to set in stone that packages are supposed to always
> >>> be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
> >>> builders must set LC_ALL=C.UTF-8?
> >>
> >> This would be the opposite of the current rule.
> >>
> >> Setting LC_ALL=C in debian/rules is an one-liner.
> >>
> >> If your package is not reproducible without it, then your package is
> >> broken. It can go in with the workaround, but the underlying problem
> >> should be fixed at some point.
> >>
> >> The reproducible builds checker explicitly tests different locales to
> >> ensure reproducibility. Adding this requirement would require disabling 
> >> this
> >> check, and thus hide an entire class of bugs from detection.
> > 
> > this is one facet of a much bigger discussion (which we've had before). You 
> > can
> > argue both ways, depending on how you look at this problem.
> > 
> > It is the question of whether we want to:
> > 
> >   a) debian/rules is supposed to be runnable in a wide variety of 
> > environments.
> >   If your package FTBFS in a one specific environment, it is the job of 
> > d/rules
> >   to normalize the environment to cater for the specific needs of the 
> > package.
> > 
> >   b) debian/rules is supposed to be run in a well-defined environment. If 
> > your
> >   package FTBFS in this normalized environment, then it is the job of 
> > d/rules to
> >   add the specific needs of the package to d/rules.
> > 
> > So the question is whether you either want to have d/rules normalize
> > heterogeneous environments (a) or whether you want d/rules to make a 
> > normalized
> > environment specific to the build (b). This is of course a spectrum and I 
> > think
> > we currently doing much more of (a).
> 
> I agree with Simon here.

And, if I understand your reply correctly, you do not disagree with me either?

> C, or C.UTF-8 is not a universal locale which > works for all.

Yes. If we imagine a hypothetical switch to LC_ALL=C.UTF-8 for all source
packages by default, then there will be bugs. The question is, which bugs do we
want to fix: Bugs that happen because of a problem that occurs because we did
*not* set LC_ALL=C.UTF-8 (like reproducible builds problems) or problems that
occur because we *did* set LC_ALL=C.UTF-8 as in the example that you are
describing below.

> While C.UTF-8 solves character representation part of 
> "The Turkish Test" [0], it doesn't solve capitalization and sorting  issues.
> 
> In short, Turkish is the reason why some English text has "İ" and "ı" in 
> it, because in Turkish, they're all present (ı, i, I,  İ), and their 
> capitalization rules are different (i becomes İ and ı  becomes I; i.e. 
> no loss/gain of dot during case changes).
> 
> This  creates tons of problems with software which are not aware of the 
> issue  (Kodi completely breaks for example, and some software needs 
> forced/custom environments to run).

As I'm curious: if your software breaks depending on the LC_ALL setting, how do
you make it produce reproducible binaries? If it breaks with a certain LC_ALL,
then during the build you have to set LC_ALL (or one of its friends) to some
specific value, right?

> So, all in all, if your software is expected to run in an international 
> environment, and its build/run behavior breaks in an environment is not 
> to its liking, I also argue that the software is broken to begin with. 
> Because when this problem takes hold in a codebase, it is nigh 
> impossible to fix.
> 
> So, I think it's better to strive to evolve the software to be a better 
> international citizen rather than give all the software we build an 
> artificially sterile environment, which is iteratively harder and harder 
> to build and maintain.

Just to make sure I'm not misunderstood: I also am tending towards *not*
setting LC_ALL=C.UTF-8 (but probably not as strongly as I understood Simon's
mail) just because I like dumping my time into figuring out why my software
does something different in a very specific environment. Figuring this out
does uncover bugs that should be fixed most of the time.

At the same time though, I also get annoyed of copy-pasting d/rules snippets
from one of my packages to the next instead of making use of a few more
defaults in our package build environment.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Reviving schroot as used by sbuild

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2024-06-25 10:16:20)
> In this work, limitations with --chroot-mode=unshare became apparent and that
> lead to Johannes, Jochen and me sitting down in Berlin pondering ideas on how
> to improve the situation. That is a longer story, but eventually Timo Röhling
> asked the innocuous question of why we cannot just use schroot and make it
> work with namespaces.

for those who are interested in the longer story behind all of this, let me
explain this here. Maybe this background helps to give a bit more of a context
about why we were thinking about this. If my memory serves me right, I think
the initial trigger for all of this was an idea that Julian Andres Klode had:
instead of having to manually run

$ mmdebstrap --variant=buildd unstable ~/.cache/sbuild/unstable-arm64.tar

manually before running sbuild for the first time and every once in a while
after that, could the sbuild unshare backend not run this command automatically
if the chroot tarball doesn't exist yet or became too old? If that were the
default, setting up a package builder on a new system would be as simple as
running:

$ sudo apt install sbulid
$ sbuild --chroot-mode=unshare -d unstable my_cool_package

Just install the package, no setup required, and just start building things. So
I wrote this MR:

https://salsa.debian.org/debian/sbuild/-/merge_requests/59

Besides automatically creating the chroot, updating the chroot and setting the
maximum age of the chroot tarball, this MR also allows passing custom options
to mmdebstrap depending on the chroot name. So somebody who wants an ubuntu
buildd chroot could have in their ~/.sbuildrc:

$UNSHARE_MMDEBSTRAP_EXTRA_ARGS = {
  "focal" => ["--components=universe,multiverse"]
}

And if you want a custom chroot for the rust packages you build, you could have
(using %d and %a as percent escapes for distribution and architecture):

$UNSHARE_MMDEBSTRAP_EXTRA_ARGS = {
  "debcargo-%d-%a" => [--include 
ccache,gnupg,dh-cargo,cargo,lintian,perl-openssl-defaults"]
}

But the big question now is: should all of this functionality and complexity
live in sbuild? Or should it be moved out of sbuild so that for example the
rust team can just have their custom sbuild chroot script doing all the setup
and customization they require without sbuild carrying that functionality? The
question of how to best allow sbuild to allow an external chroot manager lead
to Timo's idea of just replacing 'schroot' with something else which provides
the interface that sbuild uses to communicate with schroot but then in the back
does its own thing. We have such a thing now and that is Helmut's unschroot.py.

But this is not the only option forward. Another option would be to re-purpose
the sbuild autopkgtest backend for this.

Ultimately, what do we want to achieve? We want to make package building
easier, less tedious and more customizable. We are thinking about what the best
architecture would be to achieve this. We have multiple options:

 1. bolt the functionality we want into sbuild as extensions to the unshare
backend or by creating new backends with the desired functionality -- this
is https://salsa.debian.org/debian/sbuild/-/merge_requests/59
 2. replace the schroot binary with something else which shares the schroot
interface -- this is unschroot.py
 3. move functionality into autopkgtest backends and then make sbuild just a
wrapper around autopkgtest

Choosing one of these three options as the correct software engineering
approach becomes even more tricky when we start thinking of persistence.
Providing a persistent user and mount namespace (for example to be able to use
overlay filesystems on top of a unpacked chroot directory) will be *very*
tricky with the current design of the unshare backend and would probably need
to become its own backend, if choice 1. is the one we want to pursue.
Persistence is also something that would be useful for a backend around qemu.
Currently, there exists sbuild-qemu maintained by Christian Kastner which is
not a new backend but is a convenience wrapper on top of sbuild driving it with
the autopkgtest backend and autopkgtest-virt-qemu as the virt server. It would
be great if building packages inside a qemu vm became easier and option 2 would
allow users to create a backend that starts qemu and then communicates with a
process inside qemu efficiently via an AF_VSOCK. Lastly, persistence is also a
requirement for building packages inside a system container which runs an
actual init process when it spins up.

These last two bits (qemu and system containers) also become a very interesting
topic to think about because in contrast to minimal application containers or a
simple chroot directory, we do not want to build packages directly in them
(because we want to build packages in a minimal setup). So now sbuild would
have to manage 3 environments:

 1. the system on the outside
 2. the system inside qemu or the syst

Re: Reviving schroot as used by sbuild

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Simon McVittie (2024-06-27 15:59:01)
> On Thu, 27 Jun 2024 at 11:46:51 +0200, Helmut Grohne wrote:
> > I don't quite understand the need for a Dockerfile here. I suspect that
> > this is the obvious way that works reliably, but my impression was that
> > using podman import would be easier.
> 
> Honestly, the need for a Dockerfile here is: I already knew how to build
> containers from a Dockerfile, and I didn't read the documentation for
> the lower-level `podman import` because `podman build` can already do
> what I needed.
> 
> I see this as the same design principle as why we encourage package
> maintainers to use dh, even when building trivial "toy" packages like
> hello, and in preference to implementing debian/rules at a lower level
> in trivial cases. To build a non-trivial container with multiple layers,
> you'll likely need a Dockerfile (or docker-compose, or some similar thing)
> *anyway*, so a typical user expectation will be to have a Dockerfile, and
> anyone building a container will likely already have learned the basics
> of how to write one; and then we might as well follow the same procedure
> in the trivial case, rather than having the trivial case be different and
> require different knowledge.

I have never in my life written a Dockerfile and so far I've only used "podman
import" instead. Your explanation makes sense to me. I had no idea that "podman
build" is on a higher plumbing level. As a container noob it was always more
easy for me to write:

mmdebstrap [my customizations] unstable | podman import - debian

If I understand what you are saying, then what should instead be done is to
write a Dockerfile receiving a vanilla tarball and then do the customizations
via the Dockerfile?

Can a Dockerfile be read from stdin? It's a small wrinkle to me that I would
then need to create a private temporary directory with a Dockerfile first
instead of just shoving it in over a pipe.

> > Do I understand correctly that in this variant, you intend to use podman
> > without its image management capabilities and rather just use --rootfs
> > spawning two podman containers on the same --rootfs (one after another)
> > where the first one installs dependencies and the second one isolates the
> > network for building?
> 
> Maybe that; or maybe use its image management, tell the first podman command
> not to delete the container's root filesystem (don't use --rm), and then
> there's probably a way to tell podman to reuse the resulting filesystem
> with an additional layer in its overlayfs for the network-isolated run.
> 
> Please note that I am far from being an expert on podman or the
> "containers" family of libraries that it is based on, and I don't
> know everything it is capable of. Because Debian has a lot of pieces
> of infrastructure we have built for ourselves from first principles,
> I've had to spend time on understanding the finer points of sbuild,
> schroot, lxc and so on, so that I can replicate failure modes seen on
> the buildds and therefore fix release-critical bugs in the packages that
> I've taken responsibility for (and occasionally also try to improve the
> infrastructure itself, for example #856877 which recently passed its
> 7th birthday). That comes with an opportunity cost: the time I spent
> learning about schroot is time that I didn't spend learning about OCI.
> 
> One of the reasons I would like to have fewer Debian-specific pieces in
> our stack is so that other Debian developers don't have to do what I
> did, and can instead spend their time gaining transferrable knowledge
> that will be equally useful inside and outside the Debian bubble (for
> example the best ways to use OCI images, and OCI-based tools like
> Docker and Podman, which have a lot of overlap in how they are used even
> though they are rather different behind the scenes).

Thank you for this text as well as the one in your initial email in which you
caution against more Debian-isms with only very few maintainer(s) maintaining
them. As the author of the unshare backend I am guilty of having added another
Debian-specific thing instead of re-using existing solutions. Maybe my defense
can be that when I wrote that code in 2018, there was no podman in Debian yet?
I am not attached to the unshare code. I gladly throw it out for something
better. The less code I have to maintain the better for me. I do not dislike
podman either and I am happy that in contrast to docker, there is no persistent
service running in the background.

What I wanted to mainly bring up in this email are the following things:

Creating build chroots from things that are signed with the Debian archive
keyring is important to me. Even though, as Holger pointed out, the Debian
images that one can download can be reproduced independently, I rather make
sure that I receive what I think I receive by relying on creating my chroot via
mmdebstrap/apt verified by my local keyring. Maybe in the future debian.org can
publish build chroo

Re: Reviving schroot as used by sbuild

[Johannes Schauer Marin Rodrigues]

Simon,

Quoting Simon McVittie (2024-06-27 19:16:54)
> On Thu, 27 Jun 2024 at 17:26:20 +0200, Johannes Schauer Marin Rodrigues wrote:
> > But, if everybody is so excited about this, where are the sbuild 
> > contributors
> > implementing this?
> 
> I'm sorry, consider it added it to my list. As usual, there's no guarantee
> that I will get there within my lifetime, but I'll make sure to feel
> suitably guilty about my failure to achieve it.

if you want to do me a favour, please do not put it on your todo list. Even
more importantly: please try to not feel guilty for anything. If at all
possible, I'd like to assure you that you were not even close to being on the
list of people (if we imagine that such a list existed in the first place) that
I would make responsible.

> This is clearly not entirely true any more, because if it was, buildds would
> not be able to use sbuild's unshare backend - so perhaps now is the time to
> be proposing a sbuild podman backend, and I should probably be writing one
> instead of replying to this message.

Or you let other people take care of it. There are more than a dozen attempts
outside of sbuild. How hard can it be? I consider you one of the most capable
and clever people in the project and I greatly value your input into this
discussion. But were I to choose where to put your time, it would not be into
stretching your resources even more thinly by becoming the sbuild+podman
maintainer. If you are really eager I do not want to stop you either. But
please, please do not feel pressured by my last email.

> I'm sorry that I have failed to provide a concrete solution to this problem,
> and I will try to do better in future.

Please accept my apology for how I phrased my last email. I did not want you to
feel sorry for anything.

I'm sincerely sorry. I did not mean to make you feel guilty.

Sorry.

josch

signature.asc
Description: signature

Bug#1077486: ITP: fonts-iosevka-term -- Versatile typeface for code, from code

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org, debian-fo...@lists.debian.org

* Package name: fonts-iosevka-term
  Version : 31.0.0
  Upstream Contact: Renzhi Li aka. Belleve Invis 
* URL : https://github.com/be5invis/Iosevka
* License : OFL-1.1
  Programming Lang: TTF
  Description : Versatile typeface for code, from code

Iosevka [ˌjɔˈseβ.kʰa] is an open-source, sans-serif + slab-serif, monospace +
quasi‑proportional typeface family, designed for writing code, using in
terminals, and preparing technical documents.

This package provides the "Term" subfamily of Iosevka as ttf.

This is a dependency of the reform-tools package used by MNT for the Reform
family of open hardware computers.

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Johannes Schauer Marin Rodrigues]

Hi Otto,

Quoting Otto Kekäläinen (2024-07-28 00:38:40)
> I have drafted a new DEP at
> https://salsa.debian.org/dep-team/deps/-/merge_requests/8 titled "DEP-18:
> Enable true open collaboration on all Debian packages".
> 
> Direct link to raw text:
> https://salsa.debian.org/dep-team/deps/-/raw/798bfa5a1e1609afd4e48ee20aff80a82bcd4a2f/web/deps/dep18.mdwn
> 
> This would have been a great topic to discuss in person at DebConf, but
> unfortunately I can't attend this year, so I'll just kick this off on the
> mailing list.

thank you for working on this.

> This is continuation to the 'single maintainership' discussions earlier this
> year. I also think that more unified and open collaboration processes could
> help decrease maintainer burnout, lower barrier for existing and new
> maintainers to help with multiple packages, and overall perhaps also improve
> quality of uploads by having more attention on the source code prior to
> upload.

A slightly related topic: what is everybody's opinion on maybe starting with
some of the basic items of DEP-18, lets say items 1-3, and prescribe them to
only a very small set of packages in Debian. And which set of packages in
Debian should *especially* have easy open collaboration enabled if not the set
of source packages producing our Essential:yes packages? So why not start with
the source packages in the Essential:yes set, have them adhere to better
collaboration standards like packaging in git on salsa and with salsa CI
enabled?

I am aware that this might just be a pipe dream because at least in my personal
experience I observed very strong package ownership especially for packages in
the essential set. But I think it is the set of Essential:yes packages which
should set an example of how collaborative maintenance in Debian is supposed to
work because in contrast to other packages, the Essential:yes set is relevant
to literally all of us. Bugs in those packages affect us all, so we all should
have an easier time to collaborate on them, right?

Bugs like #1021336 had to be closed *twice* because of issues that would've
been caught by the salsa CI pipeline. It doesn't help that a MR enabling salsa
CI for that package is waiting to be acted upon by its maintainers since
October 2022. It is literally essential (pun intended) to all our work that
packages in the essential set do not accidentally break. The chance of this
happening can be greatly reduced by requiring a passing salsa ci pipeline with
successful autopkgtest, piuparts and lintian. What am I missing?

To those who refuse to work with salsa CI because it is slow (yes it is, I know
because my processor is probably 10 times slower than yours) I can warmly
recommend the glab utility from the package of the same name. My Debian
workflow has now these two commands added after I "git push" my work:

 - glab ci status --live
 - glab ci trace

The former will let me follow the pipeline for my last commit as it is running
without having to open it in my browser (which is painfully slow). The latter
will let me look at the stdout and stderr of the job of my choice (usually the
failing one) without some javascript deciding that it will remove the whole
text from my screen with the error message "An error occurred while fetching
the job."

> If you think this proposal makes sense, please press the thumbs up button.
> 
> If you have comments, please share them here or on the draft itself.

I think I like all items you brought up except for item 5. At least in the part
of Debian where I am active, I find it very hard to find anybody reviewing my
stuff, probably because everybody else is already overworked themselves. So I
think it's nice but I do not see us having very much volunteer time to really
have this work in practice. But maybe there are teams for which this really
works well? In any case, nothing I'd like you to remove but maybe it's too much
of a dream? :)

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Request for feedback on draft: DEP-18: Enable true open collaboration on all Debian packages

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Otto Kekäläinen (2024-08-02 17:23:51)
> I agree that Salsa is sometimes a bit sluggish
> (https://salsa.debian.org/salsa/support/-/issues/395),

what kind of hardware do you have? For people like me who are on slower
hardware, the web experience is absolutely not funny and "a bit sluggish"
doesn't begin to describe it. It is hard to find any other website I'm visiting
that is slower than gitlab. For example, when I run this on my Firefox I get a
score of 1.53: https://browserbench.org/Speedometer3.0/#summary

On an intel core i7 1280P you will get around 7. If the result on your machine
is towards the latter instead of the former, then I understand how you would
describe the gitlab experience only as "a bit sluggish" instead of "atrocious".

> but I hope we can do something to improve it and it is now a permanent reason
> to not use Salsa.

You now said "I hope we can do something to improve it and it is now a
permanent reason to not use Salsa." twice:

https://lists.debian.org/caou6tadwmnc__rvqpob7a1+zyysngotfiq4i+69hhpatfxe...@mail.gmail.com

Did you typo it twice or do you actually mean that it's now a permanent reason
to not use salsa?

I am not suggesting salsa use because I think it's the best thing since the
invention of sliced bread. But personally, I rather use something suboptimal if
it means that we can more or less agree on a "default" and I think that the
current situation (submission of patches by mail and the debian archive is the
bts) is deterring to newcomers. I know that most people probably have faster
machines than I.  As it was pointed out elsewhere in this thread, Debian work
already implies a lot more waiting than "a few seconds" for salsa to finish
loading a page or finish yet another animation, so meh. With the glab tool I
think I can be happy enough.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Ship a git .bundle in .dsc and .deb

[Johannes Schauer Marin Rodrigues]

Quoting Todd Zullinger (2024-08-15 01:12:01)
> Piper McCorkle wrote:
> > On Wednesday, 14 August 2024 16:43:54 CDT Agathe Porte wrote:
> >> Or maybe I could tar the upstream .git folder and store everything else
> >> as plain files? But still a binary blob in source package. And
> >> extracting the tar into a .git in the filesystem in /usr/share/ may
> >> raise a lot of Lintian warnings?
> > 
> > Perhaps you could include the contents of the git repo in the source 
> > package, 
> > then in d/rules run something like `cd usr/src/qmk_firmware && git init && 
> > cp 
> > files . && git add -A && git commit`. That way, you just have regular files 
> > in 
> > the source package, but you generate a synthetic Git repo in the build 
> > process.
> > 
> > Reproducibility might be difficult, given that git uses the current date 
> > and 
> > such for creating commits. Doesn't sound insurmountable though!
> 
> You can set GIT_AUTHOR_NAME and GIT_COMMITTER_DATE to use
> SOURCE_DATE_EPOCH and export those vars (if there isn't a helper which does
> this already in the Debian build tooling).

If you set GIT_COMMITTER_DATE, GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL then
you can create bit-by-bit reproducible git commits with --date, --author and
--message set to the desired value. We do this for metasnap.debian.net. Git has
no problem producing bit-by-bit identical commits, given the same input.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: debian/gbp.cnf analytics? (Re: Re: Accepting DEP14?)

[Johannes Schauer Marin Rodrigues]

Quoting Otto Kekäläinen (2024-08-19 03:45:37)
> I tried to use codesearch.debian.net to find out how many packages have a
> debian/gbp.conf but it seems it can't be used to simply list packages that
> have a specific file, it always also needs a search terms to look up inside
> the file.
> 
> With 
> https://codesearch.debian.net/search?q=path%3Adebian%2Fgbp.conf+debian-branch+%3F%3D+%3Fdebian%2Flatest&literal=0
> I was able to find that 1655 packages have either "debian-branch =
> debian/latest" or "debian-branch=debian/latest".
> 
> Is there some easy way to iterate every single Debian package and
> extract just one single file from them without having to download all
> packages?
> 
> I'd like to see how many % of all Debian packages have a gbp.conf file, and
> then download all of them to do stats on what they contain.

finding out which package contains a given file is better done via the Contents
files from our mirrors. The apt-file tool provides an easy interface to search
these contents files and answer the question "which package contains a file or
path that looks like this".

By default, apt-file will only download (and search) Contents files for binary
packages and not source packages. To change that, edit
/etc/apt/apt.conf.d/50apt-file.conf and change DefaultEnabled from "false" to
"true" in the section deb-src::Contents-dsc. Once that is done you run "apt
update" to download the newly enabled Contents files and then you can run a
search like this:

$ apt-file --index-names dsc search debian/gbp.conf

You need to add --index-names because the default value is "deb" and that would
only search through binary packages.

Thanks!

cheers, josch

signature.asc
Description: signature

Bug#1079047: ITP: reform-handbook -- MNT Reform Operator Handbook

[Johannes Schauer Marin Rodrigues]

Package: wnpp
Severity: wishlist
Owner: Johannes Schauer Marin Rodrigues 
X-Debbugs-Cc: debian-devel@lists.debian.org

* Package name: reform-handbook
  Version : 2024-08-19
  Upstream Contact: Lukas F. Hartmann 
* URL : https://source.mnt.re/reform/reform-handbook/
* License : GPL-3+ and CC-BY-SA-4.0 and CERN-OHL-S-2.0
  Programming Lang: LaTeX
  Description : MNT Reform Operator Handbook

The HTML and PDF version of the operator handbook for the MNT Reform open
hardware laptop. It contains topics like:

 - Linux basics (terminal, shell, filesystem, managing software with apt)
 - the graphical desktops sway and wayfire with waybar
 - MNT Reform specific software and scripts
 - the hardware (case, motherboard, processor modules, keyboard, trackball,
   trackpad, battery packs)
 - advanced topics (serial via uart, u-boot)
 - hardware schematics and bill of materials

Re: Removing more packages from unstable

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Helmut Grohne (2024-08-20 07:28:52)
> What do you think about the proposed criteria and suggested set of source
> packages? Is it reasonable to remove these packages from unstable? In a
> sense, it is extending the idea of the testing auto remover to unstable.
> Similarly, a package can be temporarily saved by mailing the respective bug.

if I remember correctly, a package can also become a key package by having a
high-enough popcon value. If that is correct, maybe there should also be the
inverse. Looking at your list, about 85% of those packages have a popcon lower
than 100. Taking the popcon value into account would also kinda make your
hand-curated list of exceptions obsolete as your current list has popcons well
above 100, for example. If the popcon is taken into consideration, that would
also give a little bit of insurance that only very few users will be affected.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: DEP-18: Git and GitLab usage in other Linux distros (Re: Representing Debian Metadata in Git)

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Otto Kekäläinen (2024-08-27 08:42:53)
> > Before pushing for new ways of representing Debian stuff in git, I think it
> > would be a good idea to learn from all the other distros and distro-like
> > systems successfully using git [1]. Debian is not the only distro that
> > wants to use git to capture changes and encourage contributions to its
> > packages.
> >
> > Chris
> >
> > [1] alpine, homebrew, freebsd ports come to mind immediately. nixos
> > and others too.
> 
> …this is the right attitude and I wanted to cater to it and summarize
> how packaging sources look in various distros.

thank you for your investigations.

> - The number of contributors/maintainers is low everywhere. Ending
> single-person maintainership is not going to happen any soon, but hopefully,
> we can work towards first increasing the pool of contributors who
> participate, and then expand on practices around Merge Requests and reviews
> and maybe have some kind of formal sign-offs from at least two people before
> upload. Initially, perhaps only for the top-150 packages. But before we can
> institute review workflows, we need to have more unification around the
> version control and basic packaging workflows.

I'm still dubious any "2 people sign-off" can work [1]. In your investigations,
did you find other distributions which implemented this successfully?

I think "work towards easier collaboration" and "require more than one person
for every commit/upload" are two very different things which should be
discussed independently.

Thanks!

cheers, josch

[1] My own experience with this comes from my contributions to devscripts which
is in the debian group, thus "team" maintained and probably all of you have it
installed and should feel responsible for it (right?).  Nevertheless, my MRs
mostly get zero replies, so I usually just merge them after waiting a couple of
months. The situation is a bit better for sbuild but not by much.

signature.asc
Description: signature