Re: RFC: Standardizing source package artifacts build paths

[Sean Whitton]

Hello Niels,

On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:

> Guillem and I have debated this and come to a solution, which we believe
> will be able to address the concerns about the path being "hidden". It
> also enables us to simplify parts of the migration in some cases.
>
> The short version of that solution is to enable debhelper to expose the
> relevant directories where you want it and under debian/.build at the
> same time (the latter being a symlink to the former)[1].
>   This can be made to cover the directories you mention above, but also
> e.g. the build directory for upstream builds, which was mentioned
> elsewhere in this thread.
>
> Based on the feedback so far, I will go with the following defaults in
> debhelper:
>
>  * build directories are exposed if -B/--builddirectory is used to
>request it.  If there is no explicit -B/--builddirectory, it will
>(in a new compat level) be hidden by default.

So, if you want the build directories to be exposed, you have to pass
--builddirectory to debhelper, so it becomes a choice made by the
maintainer of each package?

I'm concerned that this doesn't help address the issue of new
contributors struggling to find .build when trying to figure out where
their build artifacts have gone -- it's not going to be any easier for
them to find the --builddirectory option.

As has been said, a non-hidden 'build' is easier to notice if you're
looking around.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Sean Whitton:
> Hello,
> 
> On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:
> 
>> Guillem and I have debated this and come to a solution, which we believe
>> will be able to address the concerns about the path being "hidden". It
>> also enables us to simplify parts of the migration in some cases.
> 
> Thank you for your continued efforts!
> 

Thanks. :)

>> The short version of that solution is to enable debhelper to expose the
>> relevant directories where you want it and under debian/.build at the
>> same time (the latter being a symlink to the former)[1].
> 
> Sorry, but don't you mean the former (non-dotdirs) being a symlink to
> the latter (.build)?
> 

Originally, no.  Though I think I will leave it unspecified while we
experiment to see if I become wiser on the topic than I am today.


My thought was this would enable debhelper to see that the directory was
exposed and look it up without additional state tracking (which would in
turn enable debhelper to ensure that we clean up properly since both
"ends" must be removed during clean).

But worst case, it will cost me a compat level to flip the order of the
two if the epiphany comes to me after initial deployment.

~Niels

Re: RFC: Standardizing source package artifacts build paths

[Sean Whitton]

Hello,

On Tue 14 Apr 2020 at 10:54AM +02, Niels Thykier wrote:

> Guillem and I have debated this and come to a solution, which we believe
> will be able to address the concerns about the path being "hidden". It
> also enables us to simplify parts of the migration in some cases.

Thank you for your continued efforts!

> The short version of that solution is to enable debhelper to expose the
> relevant directories where you want it and under debian/.build at the
> same time (the latter being a symlink to the former)[1].

Sorry, but don't you mean the former (non-dotdirs) being a symlink to
the latter (.build)?

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Sam Hartman:
> I'm concerned about a leading . at least for:
> 
> * the debian/tmp replacement
> * the replacement for the package install directories under debian.
> 
> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.
> 
> I don't care as much about substvars, debhelper intermediates, debhelper
> logs and the like.
> 

Hi,

Guillem and I have debated this and come to a solution, which we believe
will be able to address the concerns about the path being "hidden". It
also enables us to simplify parts of the migration in some cases.

The short version of that solution is to enable debhelper to expose the
relevant directories where you want it and under debian/.build at the
same time (the latter being a symlink to the former)[1].
  This can be made to cover the directories you mention above, but also
e.g. the build directory for upstream builds, which was mentioned
elsewhere in this thread.

Based on the feedback so far, I will go with the following defaults in
debhelper:

 * build directories are exposed if -B/--builddirectory is used to
   request it.  If there is no explicit -B/--builddirectory, it will
   (in a new compat level) be hidden by default.

 * The staging directories a la d/tmp will be exposed if --destdir is
   used with dh_auto_install.
   - I will expose d/tmp by default for now but "parallel" d/tmp-X dirs
 are not (they currently require a --destdir to work as it is now).

 * d/ are exposed by default for now.


Any other debhelper artifact will move to a new path in a future compat
level (at this stage, we are talking debhelper/14 rather than debhelper/13).

~Niels

[1] Note that dpkg will migrate to always use the paths under
debian/.build.  This should not matter in practise unless you fiddle
with the symlinks that debhelper creates.

Re: RFC: Standardizing source package artifacts build paths

[Andreas Metzler]

On 2020-03-09 Sam Hartman  wrote:
> I'm concerned about a leading . at least for:

> * the debian/tmp replacement
> * the replacement for the package install directories under debian.

> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.

I would like to add a strong AOL. Please do not use a hidden toplevel
directory. debian/.build does not avoid clutter - it just makes it harder
to recognize a cluttered directory. Not "stomping over any existing
directories" is also a weak argument for ".build" vs any other name,
neither has a guarantee that there are no conflicts currently.

cu Andreas

Re: RFC: Standardizing source package artifacts build paths

[Guillem Jover]

On Mon, 2020-03-30 at 13:54:27 +0200, Simon Richter wrote:
> On 30.03.20 00:52, Guillem Jover wrote:
> > And, of course there are always going to be remaining sticking points
> > not covered by features I or others have in mind, but IMO their presence
> > will still mean there's something to improve somewhere.
> 
> This is the same debate we had in a lot of other places by now: do we
> want a descriptive interface that requires explicit provisions for every
> case, or an imperative interface that requires programming skill to use?

This proposal is orthogonal to that debate. Of course some things
might need declarative interfaces, but that does not mean that
excludes or would disallow programmatic ones. The proposal mostly
deals with defaults and conventions, for already existing interfaces.
For example for dpkg-shlibdeps or dh_install to look automatically under
these directories. Some of these commands might also grow new options
to override those defaults, when they do not yet have them. So it might
actually increase the control possibilities!

(But just to appease whoever else might be worried, while at least Niels
and I have been working and have plans to further improve the declarative
packaging support, we also are extremely aware that we'll always need
to support escape hatches for programmatic interfaces.)

> The namespace below debian/ has not been tightly regulated so far. Any
> name that isn't mentioned in Policy is free for the package to use for
> anything, and access to it is brokered again through debian/rules (with
> "clean" being pretty much the only operation allowed on it).

Again the proposal is not to tighly regulate anything. It would be a
matter of convention and defaults. If people do not want or cannot use
these, then that'd be fine too. Perhaps this was not really clear from
the initial proposal, or the fact that it was proposed at all made it
sound more forceful than it was intended.

Also policy does list many of these default pathnames already, most in
its appendices. Because that's what dpkg has used as defaults for a
very long time. Of course that has not disallowed debhelper, f.ex., to
use some different pathnames over time. The purpose of the proposal was
to try to create a new convention and coordinate over it.

> In that regard, the debian/ directory is not much different from the
> rest of the source tree. We still have quite a few packages doing
> in-tree builds because the upstream build system does not work for
> out-of-tree; these would still require cleaning through "debian/rules
> clean", so we won't get by without our escape hatch for quite a while.

Nothing would force antyhing to place stuff anywhere, as long as the
convention is not used. The point as mentioned above, is that if you
need, say, an out-of-tree pathname, then there would be conventions to
use, and the tools would have these as defaults or easy support for them
(for example with options to select different build flavors). If for
whatever reason these cannot be used then overrides would still be
possible for non-internal stuff, like it is currently the case.

> Unrelated to Policy, debhelper could move all of their intermediate
> files below debian/debhelper, that would probably get us 90% of the way
> without requiring a transition period.

This would still be quite unsatisfactory and not give the important
properties proposed, and moving all intermediate debhelper stuff would
still require a transition, or it would need to leave things behind.

Thanks,
Guillem

Re: RFC: Standardizing source package artifacts build paths

[Simon Richter]

Hi,

On 30.03.20 00:52, Guillem Jover wrote:

> And, of course there are always going to be remaining sticking points
> not covered by features I or others have in mind, but IMO their presence
> will still mean there's something to improve somewhere.

This is the same debate we had in a lot of other places by now: do we
want a descriptive interface that requires explicit provisions for every
case, or an imperative interface that requires programming skill to use?

For packaging, we have an imperative interface, and a declarative
wrapper on top that handles most of the simple cases. That works well,
and I'm not sure there is much potential for improvement there, because
even if we switched to a declarative-by-default system, we'd still need
an imperative escape hatch for those cases not covered yet.

The namespace below debian/ has not been tightly regulated so far. Any
name that isn't mentioned in Policy is free for the package to use for
anything, and access to it is brokered again through debian/rules (with
"clean" being pretty much the only operation allowed on it).

In that regard, the debian/ directory is not much different from the
rest of the source tree. We still have quite a few packages doing
in-tree builds because the upstream build system does not work for
out-of-tree; these would still require cleaning through "debian/rules
clean", so we won't get by without our escape hatch for quite a while.

Unrelated to Policy, debhelper could move all of their intermediate
files below debian/debhelper, that would probably get us 90% of the way
without requiring a transition period.

   Simon



signature.asc
Description: OpenPGP digital signature

Re: RFC: Standardizing source package artifacts build paths

[Guillem Jover]

On Sun, 2020-03-29 at 22:48:04 +0200, Marco d'Itri wrote:
> On Mar 29, Guillem Jover  wrote:
> > While it's true that we might need to use such pathnames in debian/rules
> > or debhelper fragment files (which some might consider ugly), IMO that
> > has always felt like a sign that there's something missing in our
> > packaging helpers/tools.

> If you believe this then it means that you have been maintaining too 
> modern or too much simple packages.

Aww thanks, you are, as always, way too kind, and you probably even
listed too many reasons there.

> https://salsa.debian.org/md/inn/-/blob/master/debian/rules
> https://salsa.debian.org/md/inn2/-/blob/master/debian/rules
> https://salsa.debian.org/md/kmod/-/blob/master/debian/rules
> https://salsa.debian.org/md/binkd/-/blob/master/debian/rules
> https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/rules

Most of these show some of the things I had in mind. Which have not
been supported (easily) in the past or yet by our packaging toolchain,
like:

  - declarative file metadata,
  - renaming files,
  - flavor builds (like udeb vs no-udeb, or w/ features disabled in
configure, etc.),

Some other usages there though, look like things that can really be
simplified in the packaging, by say, just:

  - using dh_install, dh_link and friends,
  - being explicit in things to install in the .install fragments
instead of just adding whole hierarchies,
  - using a temporary destdir (like debhelper does by default) and only
installing the desired pathnames to the final installation directory.

Other things there can be simplified by improving the upstream build
system, also by parametrizing things, and submitting these upstream or
just carrying as local patches.

So yes, thanks, to me these examples seem to still confirm that the
explicit usage of the staged directories is an indication of things that
need fixing or improving in either the packaging or the toolchain.

And, of course there are always going to be remaining sticking points
not covered by features I or others have in mind, but IMO their presence
will still mean there's something to improve somewhere.

Regards,
Guillem

Re: RFC: Standardizing source package artifacts build paths

[Marco d'Itri]

On Mar 29, Guillem Jover  wrote:

> While it's true that we might need to use such pathnames in debian/rules
> or debhelper fragment files (which some might consider ugly), IMO that
> has always felt like a sign that there's something missing in our
> packaging helpers/tools.
If you believe this then it means that you have been maintaining too 
modern or too much simple packages.

https://salsa.debian.org/md/inn/-/blob/master/debian/rules
https://salsa.debian.org/md/inn2/-/blob/master/debian/rules
https://salsa.debian.org/md/kmod/-/blob/master/debian/rules
https://salsa.debian.org/md/binkd/-/blob/master/debian/rules
https://salsa.debian.org/md/libxcrypt/-/blob/master/debian/rules

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Guillem Jover]

On Mon, 2020-03-09 at 09:23:46 -0400, Sam Hartman wrote:
> I'm concerned about a leading . at least for:
> 
> * the debian/tmp replacement
> * the replacement for the package install directories under debian.
> 
> I think that maintaining those directories such that ls shows them will
> be more friendly for new maintainers.
> So I'd prefer something like debian/build rather than debian/.build for
> at least those directories.
> 
> I don't care as much about substvars, debhelper intermediates, debhelper
> logs and the like.

As mentioned on the proposal the use of a hidden directory is to
reduce clutter (getting the packaged bits alongside the generated
pathnames when listing the debian/ directory contents has always
seemed extremely distracting, which also makes it too easy for them
to end up on a VCS, f.ex.) and to avoid collisions with existing
packaging (due to temporary directory usage or due to staged package
directories).

There's also precedent with hidden directories with debhelper, even
for the staged directories for dbgsyms packages. Making internal stuff
non-hidden would make it more distracting, and if this would mean having
at least two hierarchies (one hidden and one non-hidden), that can be
even more confusing and is more work to cleanup and ignore, which goes
in the opposite direction of simplifying things IMO.

While it's true that we might need to use such pathnames in debian/rules
or debhelper fragment files (which some might consider ugly), IMO that
has always felt like a sign that there's something missing in our
packaging helpers/tools. To me this calls f.ex. for making dpkg-dev and
debhelper support build flavors (for multiple builds) or for debhelper
to automatically look in the relevant build directories when looking up
for files, so in addition to look for them in the installation stage
directory and the source package root directory, it would look there
too, removing the major needs for having to list the full pathnames
explicitly.

Thanks,
Guillem

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Mattia Rizzolo]

(note, this is a barely structured brain dump)

On Tue, Mar 10, 2020 at 08:10:55AM +0100, Niels Thykier wrote:
> >> Though, can you elaborate a bit on why the above approach would be
> >> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
> >> easy way to declare additional artifacts to be extracted?
> > 
> > Mainly, I'd prefer something declarative with glob patterns (a bit like
> > debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
> > logic like these pseudo-patches:

Same.
Having such directory be hidden inside
/build/foo-1.2.3/debian/.build/artefacts/ is kind of a mouthful, but if
that gets to be standardized I think it would be awesome (builders
(sbuild, pbuilder, …) decide on the first '/build/foo-1.2.3/' part of
the path and they know of it; package building happens with CWD in that
place, so build tools should just try to stick to relative paths
'./debian/.build/artefacts/'; everything should Just Work that way).

One thing that strikes me of this proposal, is that you were trying to
"hide" that .build directory from the maintainer; doing this would be
going against that design decision.  This is the only "concern" I have
with the proposal.  Probably this can be avoided by providing a dh_
helper.

> Ack, I get the part of having a declarative mechanism for selecting files.

And then builder could just take out the whole directory.  If that gets
to be (g|x|)zipped or not would be an implementation detail of the
builders (sbuild, pbuilder, …) and of whatever frontend (launchpad,
buildd + wanna-build, …) is used.

> Just to clarify something related.  Should debhelper and other tools by
> default archive "certain files of possible interest" (e.g. config.log)?
> Or should we limit it to "on request only"?

That would be some nice automatism indeed, but I think it's something
for "later".  If you do, please consider these bits:
 * naming the files: you risk clashing with maintainer-set file names
 * deciding on whether to put those files there only on failure or all
   the time

> The former makes it simpler for people that are interested in the
> "default" parts but also bloats the archive for people that are
> interested in just one file.

"bloating" is indeed important.  If we start doing this, frontends need
to decide on a retaining policy.  Do we want maintainers to have a say
on this?  Like, adding a metadata file to the artifacts to indicate any
interest on those files (this is a successful build: keep for x
days/keep until next successful build + y days, etc etc).

> > I don't have any particular opinion on whether artifacts should be
> > collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
> > directly into some sort of archive (Gitlab and Jenkins seem to use zip
> > files, which have the advantage of being seekable, so web frontends
> > can presumably read individual logs directly out of the zip file if
> > that's desirable).
> 
> Thanks for clarifying.  This answered the question I was trying to write. :)


I think I took care of those thoughts above, but to reiterate:
 * IMHO ./debian/.build/artefacts/ (or artifacts? :P) is a cool and
   accessible place for all interested software
 * perhaps, you could consider using
   ${DPKG_ARTEFACTS:-$PWD/debian/.build/artefacts} so that some builders
   can override the directory if they find it more convenient for some
   reason, but otherwise I'd rather stick to a stable, non-changable
   path.
 * I think eventual tarball/compression should be left as a matter for
   the build driver (sbuild, pbuilder, …).

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Mattia Rizzolo]

On Tue, Mar 10, 2020 at 09:07:57AM +, Simon McVittie wrote:
> On Tue, 10 Mar 2020 at 07:19:59 +, Paul Wise wrote:
> > On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:
> > > Standardized way of extracting additional build-time artefacts
> > 
> > This reminds me of the BYHAND stuff, I forget how that works though.
[…]
> Similarly, we probably don't want to publish the build products to users
> if the build(-time tests) failed (because we can't be confident that any
> products that were already produced are good), although we might well
> want to make them available through a contributor-oriented interface to
> help to debug the failures; but we do want to publish build and test logs
> to contributors, regardless of success or failure.

And this highlights one important aspect of such interface: such
artifacts would be collected even after a build failure.
That's not possible at all now.

> The .buildinfo file is arguably already in the same category as build
> and test logs. We currently capture it in the .changes file and upload
> it to ftp-master, but it isn't reproducible, and ftp-master doesn't
> republish it through our user-facing interface (the archive). Ideally,
> failed builds would capture their .buildinfo as well as their log for
> subsequent analysis, although I don't know whether they actually do.

That's somewhat of a tough argument, that I'd try to keep separate (it
has to do with the semantic meaning of a .buildinfo (i.e., it tries to
attach a *built artifacts* to the way it was build, a .buildinfo without
any hashes would be quite meaningless when tied to its original meaning.
Also, we do want it to be published, but we are still waiting for the
ftp-masters to tell us their distribution requirements...).

-- 
regards,
Mattia Rizzolo

GPG Key: 66AE 2B4A FCCF 3F52 DA18  4D18 4B04 3FCD B944 4540  .''`.
More about me:  https://mapreri.org : :'  :
Launchpad user: https://launchpad.net/~mapreri  `. `'`
Debian QA page: https://qa.debian.org/developer.php?login=mattia  `-


signature.asc
Description: PGP signature

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Simon McVittie]

On Tue, 10 Mar 2020 at 07:19:59 +, Paul Wise wrote:
> On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:
> > Standardized way of extracting additional build-time artefacts
> 
> This reminds me of the BYHAND stuff, I forget how that works though.

I think how that works is that at the appropriate time during a successful
build, you run dpkg-distaddfile to insert extra entries that are not a
recognised file type (.deb, .udeb etc.) into debian/files?

The difference (as I understand it) is that BYHAND is for extra build
products that are listed in the .changes file and intended to be published
to Debian users via ftp.debian.org; whereas in this thread we're talking
about non-essential things that are produced as a side-effect of the
build, are potentially useful to Debian contributors for debugging and
analysis of the build process itself, but are not actually the "product".

Some important trade-offs are different. For example, for the build
products mentioned in the .changes file (whether .deb or BYHAND) we want
reproducible builds that don't capture unnecessary information like the
properties of the build system; whereas in build and test logs, we *do*
want to capture system-specific information in case it's relevant, for
example to help a Debian contributor to realise correlations like "this
test fails whenever we're building on a btrfs filesystem" that can help
them to find and fix bugs.

Similarly, we probably don't want to publish the build products to users
if the build(-time tests) failed (because we can't be confident that any
products that were already produced are good), although we might well
want to make them available through a contributor-oriented interface to
help to debug the failures; but we do want to publish build and test logs
to contributors, regardless of success or failure.

The .buildinfo file is arguably already in the same category as build
and test logs. We currently capture it in the .changes file and upload
it to ftp-master, but it isn't reproducible, and ftp-master doesn't
republish it through our user-facing interface (the archive). Ideally,
failed builds would capture their .buildinfo as well as their log for
subsequent analysis, although I don't know whether they actually do.

smcv

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Simon McVittie]

On Tue, 10 Mar 2020 at 08:10:55 +0100, Niels Thykier wrote:
> Just to clarify something related.  Should debhelper and other tools by
> default archive "certain files of possible interest" (e.g. config.log)?
> Or should we limit it to "on request only"?

I think it would probably make most sense for dpkg (which doesn't know
about specific build systems) to not archive anything by default, or to
archive only things it produced itself.

For debhelper it might make sense for build system classes to archive
well-known logs like Autotools' ${builddir}/config.log and Meson's
${builddir}/meson-logs/ by default, but probably not logs that are in
an unpredictable location like Autotools' test logs.

If there's an exclusion mechanism for packages that know a particular
artifact is not useful and monstrously large ("!meson-logs/big.log"?) then
it doesn't necessarily matter much either way. If artifacts aren't kept
forever then the damage from archiving too much will be temporary.

> The former makes it simpler for people that are interested in the
> "default" parts but also bloats the archive for people that are
> interested in just one file.

Have you seen the UI Gitlab-CI and Jenkins provide for this? If you look
at a Gitlab-CI job like ,
there's a Download link that gives you the complete bundle of
artifacts in a zip file, but there's also a Browse link like

that lets you look at individual logs online (which is often enough to
debug an issue). Jenkins has a similar system.

On Salsa, ci.debian.net (for tests with the needs-build restriction),
and similar systems, I think it would make most sense to drop the
artifacts into somewhere the "larger" CI system will pick them up, and
let the "larger" CI system handle browsing and expiry. On
buildd.debian.org, I think build logs are kept forever(?) but artifacts
should probably have some sort of expiration mechanism, similar to the
way ci.debian.net remembers test results indefinitely but discards old
logs after a while.

smcv

Re: Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Paul Wise]

On Tue, Mar 10, 2020 at 7:12 AM Niels Thykier wrote:

>

Standardized way of extracting additional build-time artefacts


This reminds me of the BYHAND stuff, I forget how that works though.

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Standardized way of extracting additional build-time artefacts (was: Re: RFC: Standardizing source package artifacts build paths)

[Niels Thykier]

Simon McVittie:
> On Mon, 09 Mar 2020 at 20:45:13 +0100, Niels Thykier wrote:
>> Simon McVittie:
>>> For example, dpkg-buildpackage could perhaps read one glob per
>>> line from debian/artifacts and hardlink matched files (if any) into
>>> debian/.build/artifacts for collection by a "larger" framework like
>>> sbuild or pbuilder, or individual packages could copy/link files into there
>>> as they go, or debhelper build-system classes like Autotools and Meson
>>> could know the names of common log files from their build system, or
>>> some combination of those.
>>
>> Though, can you elaborate a bit on why the above approach would be
>> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
>> easy way to declare additional artifacts to be extracted?
> 
> Mainly, I'd prefer something declarative with glob patterns (a bit like
> debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
> logic like these pseudo-patches:
> 
> [...]
> 

Ack, I get the part of having a declarative mechanism for selecting files.

Just to clarify something related.  Should debhelper and other tools by
default archive "certain files of possible interest" (e.g. config.log)?
Or should we limit it to "on request only"?

The former makes it simpler for people that are interested in the
"default" parts but also bloats the archive for people that are
interested in just one file.

> I don't have any particular opinion on whether artifacts should be
> collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
> directly into some sort of archive (Gitlab and Jenkins seem to use zip
> files, which have the advantage of being seekable, so web frontends
> can presumably read individual logs directly out of the zip file if
> that's desirable).
> 
> smcv
> 

Thanks for clarifying.  This answered the question I was trying to write. :)

~Niels

Re: RFC: Standardizing source package artifacts build paths

[Simon McVittie]

On Mon, 09 Mar 2020 at 20:45:13 +0100, Niels Thykier wrote:
> Simon McVittie:
> > For example, dpkg-buildpackage could perhaps read one glob per
> > line from debian/artifacts and hardlink matched files (if any) into
> > debian/.build/artifacts for collection by a "larger" framework like
> > sbuild or pbuilder, or individual packages could copy/link files into there
> > as they go, or debhelper build-system classes like Autotools and Meson
> > could know the names of common log files from their build system, or
> > some combination of those.
> 
> Though, can you elaborate a bit on why the above approach would be
> better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
> easy way to declare additional artifacts to be extracted?

Mainly, I'd prefer something declarative with glob patterns (a bit like
debian/clean or Gitlab-CI's artifacts:paths) rather than having to write
logic like these pseudo-patches:

-./configure
+if ./configure; then
+: # OK
+else
+error=$?
+if [ -n "$DPKG_ARTIFACTS" ]; then
+cp config.log "$DPKG_ARTIFACTS/config.log" || :
+fi
+exit "$error"
+fi

-make check
+if make check; then
+: # OK
+else
+error=$?
+if [ -n "$DPKG_ARTIFACTS" ]; then
+find tests/ -name '*.log' | something something xargs cp
+fi
+exit "$error"
+fi

(doubly so if I have to sprinkle backslashes, tabs and double $ over
it to make it a valid Makefile), and I'm sure you think similarly from
the perspective of writing debhelper buildsystem classes.

In particular, if you want to capture Automake test logs without also
capturing unrelated files that happen to be named *.log, as far as
I'm aware there is no general way to do that without knowledge of the
specific package and how its tests are named or arranged, so individual
maintainers are likely to have to be involved in designating which logs
are of interest.

I don't have any particular opinion on whether artifacts should be
collected into debian/.build/artifacts/, into $DPKG_ARTIFACTS/, or
directly into some sort of archive (Gitlab and Jenkins seem to use zip
files, which have the advantage of being seekable, so web frontends
can presumably read individual logs directly out of the zip file if
that's desirable).

smcv

Re: RFC: Standardizing source package artifacts build paths

[Sean Whitton]

Hello Julian,

On Mon 09 Mar 2020 at 09:43PM +01, Julian Andres Klode wrote:

> On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote:
>> We'd like to standardize on a new set of artifact build pathnames
>> for our deb toolchain. [...]
> [...]
>> The use of a hidden directory is to reduce clutter and stomping over any
>
> Love the hidden directory.

Fair enough, but what do you think about the counterarguments that have
been raised?  What's good about having it hidden?

Thanks.

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: RFC: Standardizing source package artifacts build paths

[Julian Andres Klode]

On Mon, Mar 09, 2020 at 10:09:46AM +0100, Guillem Jover wrote: 
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. [...]
[...]
> The use of a hidden directory is to reduce clutter and stomping over any

Love the hidden directory.

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer  i speak de, en

Re: RFC: Standardizing source package artifacts build paths

[Niels Thykier]

Simon McVittie:

Hi :)

> On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
>> We'd like to standardize on a new set of artifact build pathnames
>> for our deb toolchain. These would have the following form:
>>
>>   - debian/.build/upstream*
>>
>> These could be used for out-of-tree builds replacing current
>> build-tree/ and similar directories.
> 
> Presumably the '*' means it's considered OK for packages that do separate
> production/debug/udeb builds to use multiple directories matching that
> glob?
> 
> For example, a simple debhelper build might eventually (in a future compat
> level) default to "dh $@ --builddirectory=debian/.build/upstream",
> but because src:glib2.0 does separate deb and udeb builds
> (with non-essential features disabled in the udeb), it
> would use "--builddirectory=debian/.build/upstream-deb"
> and "--builddirectory=debian/.build/upstream-udeb" instead
> of its current "--builddirectory=debian/build/deb" and
> "--builddirectory=debian/build/udeb"?
> 

Yes.  :)

Among other, it will hopefully also enable debhelper to support multiple
builds natively or better, when we have a standardized path for these
things.

>>   - We could just (say) .gitignore a single pathname.
>>   - Cleaning the Debian artifacts would imply doing just
>> «rm -rf debian/.build/» (even though this could be abstracted via a
>> new dpkg-clean command).
>>   - Namespacing would avoid any collisions.
>>   - Would make parallelization way easier and faster.
>>   - dpkg-dev tools could start defaulting to use package specfific
>> directories, instead of requiring the user/tool to specify them.
> 
> This all seems good to have.
> 
> The Salsa-CI pipeline (which runs in a git clone of a package, and
> currently uses debian/output/ for intermediate build stuff) could maybe
> also move its scratch directory to debian/.build/salsa-ci.
> 
> Have you seen  and
> related discussions in the context of sbuild and pbuilder? It seems
> like a related topic. [...]
> 
> For example, dpkg-buildpackage could perhaps read one glob per
> line from debian/artifacts and hardlink matched files (if any) into
> debian/.build/artifacts for collection by a "larger" framework like
> sbuild or pbuilder, or individual packages could copy/link files into there
> as they go, or debhelper build-system classes like Autotools and Meson
> could know the names of common log files from their build system, or
> some combination of those.
> 
> smcv
> 

I could see how having a well-defined method for extracting artifacts
will be useful (also for CI purposes).

Though, can you elaborate a bit on why the above approach would be
better than a standard ENV variable a la AUTOPKGTEST_ARTIFACTS and some
easy way to declare additional artifacts to be extracted?
  I am fine either way, but I could image that the
debian/.build/artifacts will feature interact with e.g.
"dpkg-buildpackage -tc" where a AUTOPKGTEST_ARTIFACTS replacement would not.

~Niels

Re: RFC: Standardizing source package artifacts build paths

[Weatherby,Gerard]

FWIW:

We have been using debian packaging from a private repository for the last 
three years to configure virtual machines.

As a subversion shop, at least, the extraneous files generated by a build 
aren't an issue because we simply svn-clean after a build. I believe "git 
clean" does the equivalent operation. 

I concur that hiding the build directory with a leading dot is less friendly. 
It takes a while to understand how package creation / building works, and 
hiding information from users will make it more difficult. 

--
Gerard Weatherby| Application Architect
NMRbox | Department of Molecular Biology and Biophysics | UConn Health
263 Farmington Avenue, Farmington, CT 06030-6406
Phone: 860 679 8484
uchc.edu


From: Sam Hartman 
Sent: Monday, March 9, 2020 9:23 AM
To: debian-devel@lists.debian.org
Cc: debian-d...@lists.debian.org; debhel...@packages.debian.org
Subject: Re: RFC: Standardizing source package artifacts build paths

*** Attention: This is an external email. Use caution responding, opening 
attachments or clicking on links. ***

I'm concerned about a leading . at least for:

* the debian/tmp replacement
* the replacement for the package install directories under debian.

I think that maintaining those directories such that ls shows them will
be more friendly for new maintainers.
So I'd prefer something like debian/build rather than debian/.build for
at least those directories.

I don't care as much about substvars, debhelper intermediates, debhelper
logs and the like.

Re: RFC: Standardizing source package artifacts build paths

[Sam Hartman]

I'm concerned about a leading . at least for:

* the debian/tmp replacement
* the replacement for the package install directories under debian.

I think that maintaining those directories such that ls shows them will
be more friendly for new maintainers.
So I'd prefer something like debian/build rather than debian/.build for
at least those directories.

I don't care as much about substvars, debhelper intermediates, debhelper
logs and the like.

Re: RFC: Standardizing source package artifacts build paths

[Simon McVittie]

On Mon, 09 Mar 2020 at 10:09:46 +0100, Guillem Jover wrote:
> We'd like to standardize on a new set of artifact build pathnames
> for our deb toolchain. These would have the following form:
> 
>   - debian/.build/upstream*
> 
> These could be used for out-of-tree builds replacing current
> build-tree/ and similar directories.

Presumably the '*' means it's considered OK for packages that do separate
production/debug/udeb builds to use multiple directories matching that
glob?

For example, a simple debhelper build might eventually (in a future compat
level) default to "dh $@ --builddirectory=debian/.build/upstream",
but because src:glib2.0 does separate deb and udeb builds
(with non-essential features disabled in the udeb), it
would use "--builddirectory=debian/.build/upstream-deb"
and "--builddirectory=debian/.build/upstream-udeb" instead
of its current "--builddirectory=debian/build/deb" and
"--builddirectory=debian/build/udeb"?

>   - We could just (say) .gitignore a single pathname.
>   - Cleaning the Debian artifacts would imply doing just
> «rm -rf debian/.build/» (even though this could be abstracted via a
> new dpkg-clean command).
>   - Namespacing would avoid any collisions.
>   - Would make parallelization way easier and faster.
>   - dpkg-dev tools could start defaulting to use package specfific
> directories, instead of requiring the user/tool to specify them.

This all seems good to have.

The Salsa-CI pipeline (which runs in a git clone of a package, and
currently uses debian/output/ for intermediate build stuff) could maybe
also move its scratch directory to debian/.build/salsa-ci.

Have you seen  and
related discussions in the context of sbuild and pbuilder? It seems
like a related topic. Briefly, there's a desire to be able to have
"artifacts" that are not part of the main build result, are produced by
both successful and failing builds, are kept for a while by the buildd
infrastructure, and are probably not uploaded to the Debian archive,
as a generalization of the build log.

For example, Autotools packages have ${builddir}/config.log, which is not
a useful thing to publish in a .deb (not at all reproducible!), but can
be useful to diff between successful and failing builds to debug why a
build failed. At the moment there's a special case in debhelper to cat
config.log if configure fails, but it is forgotten about after the build
if configure succeeds. Meson has ${builddir}/meson-logs/meson-log.txt,
which is analogous.

Similarly, Autotools packages normally output a test summary on success
or a more detailed log on failure (${builddir}/**/test-suite.log
describes all tests that failed, were partially or entirely skipped,
or were otherwise not 100% successful, and there are also detailed logs
that capture diagnostics from even the completely successful tests),
and Meson ${builddir}/meson-logs/testlog.txt is analogous.

At the moment the only artifact, other than the .changes file and the
files it references when the build is successful, is the build log itself,
which means the build process must either cat these other artifacts to
get them into the build log (resulting in large build logs where it's
difficult to find what you're looking for), or lose them forever.

Some packages have machine-readable test output that is used to track
regressions and intermittent failures between builds, for which cat'ing
the test output into the build log is less than ideal: to compare test
results between runs it's necessary to screen-scrape the test output
from among the unstructured/human-readable messages that make up most
of the build log. A few packages work around this by packaging their
test results, which usually makes them non-reproducible.

autopkgtest's ${AUTOPKGTEST_ARTIFACTS} (a directory into which tests are
encouraged to write or copy more detailed results than the ones shown
on stdout) and Gitlab-CI's artifacts:paths: parameter (a list of globs
matching paths in the built directory) are conceptually similar ideas.

For example, dpkg-buildpackage could perhaps read one glob per
line from debian/artifacts and hardlink matched files (if any) into
debian/.build/artifacts for collection by a "larger" framework like
sbuild or pbuilder, or individual packages could copy/link files into there
as they go, or debhelper build-system classes like Autotools and Meson
could know the names of common log files from their build system, or
some combination of those.

smcv