Re: nocheck (don't run) vs nodoc (don't build)

[Sam Hartman]

> "Sean" == Sean Whitton  writes:

Sean> Hello,
Sean> On Wed 26 Apr 2023 at 04:48PM -06, Sam Hartman wrote:

>> I guess that's consistent with RFC 2119.  And RFC 2119 SHOULD
>> means that the requirement is RECOMMENDED, and an implementation
>> that does not follow the SHOULD needs to have a reason for not
>> following the recommendation.

Sean> Just to note that Debian Policy's definition of these terms is
Sean> not quite the same as the RFC process definitions (I know you
Sean> know this -- just wanted to note that they're not the most
Sean> relevant definitions).

Agreed.
I rated the chance that Simon knew the difference between RFC 2119 and
policy language and spoke with precision at about 80%.  But to reinforce
that I picked a flamboyant usage of RFC 2119 in a manner that did not
fit policy to make sure that it fit Simon's usage, and for Simon to
realize the difference and say "hey no I meant policy language," if on
reading my text  he realized RFC 2119was not what he meant.

Responding to a developer with somewhat less experience I would have
just asked whether they really meant to be using policy language.

Re: nocheck (don't run) vs nodoc (don't build)

[Sean Whitton]

Hello,

On Wed 26 Apr 2023 at 04:48PM -06, Sam Hartman wrote:

> I guess that's consistent with RFC 2119.
> And RFC 2119 SHOULD means that the requirement is RECOMMENDED, and an
> implementation that does not follow the SHOULD needs to have a reason
> for not following the recommendation.

Just to note that Debian Policy's definition of these terms is not quite
the same as the RFC process definitions (I know you know this -- just
wanted to note that they're not the most relevant definitions).

-- 
Sean Whitton


signature.asc
Description: PGP signature

Re: nocheck (don't run) vs nodoc (don't build)

[Simon McVittie]

On Wed, 26 Apr 2023 at 16:48:43 -0600, Sam Hartman wrote:
> Simon> - the nocheck option SHOULD NOT alter the contents of any
> Simon> binary package
> 
> I agree this is true--possibly even as a MUST--for the nocheck build
> profile, but
> I think DEB_BUILD_OPTIONS are allowed to modify the contents of binary
> packages.
> As an example nostrip certainly modifies the size of things, and noopt
> can modify the behavior and performance.

Yes, I'm not making a general statement about DEB_BUILD_OPTIONS here:
some of the other options clearly do alter binary packages.

I think that as a design principle, DEB_BUILD_OPTIONS=nocheck,
specifically, should not affect the contents of binary packages (because
like the build-profile of the same name, it's about whether to run tests,
and not about what to package).

> My assumption was that if you were specifying the nocheck build option,
> you don't want to run tests, possibly because they are flaky and/or you
> are trying a build for some local reason even though you know the tests
> will fail.
> 
> If you actually want to avoid building the tests, use the nocheck build
> profile (assuming that can be done without modifying binary packages).

I agree, although if the package has installed-tests (like dbus-tests,
fwupd-tests, systemd-tests) then you would also need to turn those off
with the noinsttest build profile to get the desired effect.

> If my reasoning is correct, my interpretation is that nocheck option
> can build tests or not, depending on whatever is convenient.

Yes, I have the same interpretation.

smcv

Re: nocheck (don't run) vs nodoc (don't build)

[Simon McVittie]

On Wed, 26 Apr 2023 at 23:24:58 +0200, Christian Kastner wrote:
> On 2023-04-26 20:42, Russ Allbery wrote:
> > It's just
> > less common (although certainly not unheard of) for test suites to have
> > test-suite-only build dependencies (as opposed to test-only runtime
> > dependencies, which are very common in at least the Perl world).
> 
> We (the ROCm team) have a few instances of such test suites, namely
> header-only libraries. Upstream ships tests that require compilation,
> and we'd like to ship these tests for autopkgtest purposes.

I think test-suite-only build dependencies are actually quite common.
At least, I see them a lot.

But DEB_BUILD_OPTIONS are not a mechanism to reduce build-dependencies:
they *cannot* reduce build dependencies. If you want to reduce
build-dependencies, you have to use build profiles.

nocheck is both a build option and a build profile, with a dependency
between them: using the build profile documented as requiring also using
the option (in other words, if you try to build with DEB_BUILD_OPTIONS=""
and DEB_BUILD_PROFILES=nocheck, that's undefined behaviour and it's OK
for packages to FTBFS). So the three valid things are:

DEB_BUILD_OPTIONS="" DEB_BUILD_PROFILES="" - normal

DEB_BUILD_OPTIONS="nocheck" DEB_BUILD_PROFILES="" - don't run
the test suite to save some time or work around known failures

DEB_BUILD_OPTIONS="nocheck" DEB_BUILD_PROFILES="nocheck" - same, but
also try to avoid installing test-only build-dependencies where feasible

Any of these can be combined with DEB_BUILD_PROFILES += " noinsttest"
to disable the build of "as-installed" tests like fwupd-tests.

> Strictly speaking, [the build-profiles spec] says build dependencies
> for the purposes of
> _running_ should be ignored, which makes perfect sense of course. But
> what if there is another purpose, specifically shipping autopkgtests?
> Like packages 'systemd-tests', or 'fwupd-tests'?

Then you probably need the same logic as in src:dbus and src:glib2.0,
which is this pseudocode, so that you'll compile with tests enabled
whenever there is at least one reason to want them:

Package: foobar
Build-Depends: thing-needed-to-compile-tests  ,
   thing-needed-to-run-tests 

Package: foobar-tests
Build-Profiles: 

# note: OPTIONS not PROFILES
want build-time tests := not (nocheck in DEB_BUILD_OPTIONS)

# note: PROFILES not OPTIONS
want installed-tests := not (noinsttest in DEB_BUILD_PROFILES)
# or equivalently and often clearer:
# want installed-tests := (foobar-tests in dh_listpackages output)

override_dh_auto_configure:
if want build-time tests or want installed-tests:
--enable-tests or equivalent
else:
--disable-tests or equivalent

# recent dh_auto_test does this internally, but you might still need
# this logic if the tests have non-trivial setup/teardown
override_dh_auto_test:
if want build-time tests:
test setup
dh_auto_test
test teardown
else:
do nothing

I can never remember without looking it up whether
  or  is the right syntax
for this, but I'm reasonably sure I got it right in dbus and glib2.0 (by
testing all four combinations), so it must be  .

Any build-profile is OK to leave unimplemented (as long as nobody is
relying on it to avoid cyclic dependencies during bootstrapping, and
if they are, they'll tell you); so if you don't care about minimizing
build-dependencies, it's OK to simplify this by behaving as though
noinsttest was never used. But in that case you would always have to
build your tests (or at least the subset of them that end up installed
in foobar-tests), even if they will not be run.

> But that will take me a while, as I clearly need to think this through a
> bit more. Because it just dawned me that for a header-only library, the
> argument could be made that autopkgtests should always build the tests
> from scratch anyways. Or maybe even both (run, and build-and-run).

For the special case of a header-only library, yes I think the autopkgtest
should ideally rebuild the test executable every time, because that's
what real consumers of your API have to do, and it confirms that if your
header-only library is doing some tricky C++ thing, it hasn't been broken
by a newer toolchain. (But for autopkgtest, it should really be compiled
against the header-only library from /usr/include, and not against the copy
of hopefully the same headers in the source tree!)

For the more common case of a shared library, I think it's desirable to
*not* rebuild the tests every time, as a way to verify that the ABI of
the new shared library does not cause pre-existing binaries to regress.
If the test was rebuilt every time, it would silently accept and ignore
ABI breaks, by always being compiled with the new header that describes
the new ABI.

Header-only libraries are special because they don't have ABI, only 

Re: nocheck (don't run) vs nodoc (don't build)

[Sam Hartman]

> "Simon" == Simon McVittie  writes:

Simon> On Wed, 26 Apr 2023 at 18:59:46 +0200, Christian Kastner wrote:
>> Policy 4.9.1 states that (emphases mine): * "[nocheck] says not
>> to *run* any build-time test suite" * "[nodoc] says to skip any
>> *build* steps"
>> 
>> My reading with regards to 'nocheck' was that where tests were
>> available and needed to be built, then they should always be
>> built, just not run.
>> 
>> A typical example might be a C library package that builds those
>> tests and ships them as autopkgtests, maybe even in a dedicated
>> package.
>> 
>> I thought this line of reasoning was sound, but then I remembered
>> the 'nodoc' tag and now I am no longer sure. Maybe I'm taking the
>> 'nocheck' description too literally.

Simon> With RFC 2119 terms, my opinion would be:

Simon> - packages built with the nocheck option SHOULD NOT run tests

Agreed.

Simon> - the nocheck option SHOULD NOT alter the contents of any
Simon> binary package

I agree this is true--possibly even as a MUST--for the nocheck build
profile, but
I think DEB_BUILD_OPTIONS are allowed to modify the contents of binary
packages.
As an example nostrip certainly modifies the size of things, and noopt
can modify the behavior and performance.

My assumption was that if you were specifying the nocheck build option,
you don't want to run tests, possibly because they are flaky and/or you
are trying a build for some local reason even though you know the tests
will fail.

If you actually want to avoid building the tests, use the nocheck build
profile (assuming that can be done without modifying binary packages).

If my reasoning is correct, my interpretation is that nocheck option
can build tests or not, depending on whatever is convenient.

I guess that's consistent with RFC 2119.
And RFC 2119 SHOULD means that the requirement is RECOMMENDED, and an
implementation that does not follow the SHOULD needs to have a reason
for not following the recommendation.

Re: nocheck (don't run) vs nodoc (don't build)

[Simon McVittie]

On Wed, 26 Apr 2023 at 18:59:46 +0200, Christian Kastner wrote:
> Policy 4.9.1 states that (emphases mine):
>   * "[nocheck] says not to *run* any build-time test suite"
>   * "[nodoc] says to skip any *build* steps"
> 
> My reading with regards to 'nocheck' was that where tests were available
> and needed to be built, then they should always be built, just not run.
> 
> A typical example might be a C library package that builds those tests
> and ships them as autopkgtests, maybe even in a dedicated package.
> 
> I thought this line of reasoning was sound, but then I remembered the
> 'nodoc' tag and now I am no longer sure. Maybe I'm taking the 'nocheck'
> description too literally.

With RFC 2119 terms, my opinion would be:

- packages built with the nocheck option SHOULD NOT run tests
- the nocheck option SHOULD NOT alter the contents of any binary package
- packages built with the nocheck option MAY avoid building the tests at
  all, but only if that doesn't change the binary packages

The nocheck option (in Policy) is related to the nocheck build-profile
(not part of Policy). https://wiki.debian.org/BuildProfileSpec documents
that DEB_BUILD_PROFILES=nocheck SHOULD NOT change either the contents of
any binary package or the set of binary packages that are compiled, and
also documents that DEB_BUILD_PROFILES=nocheck MUST be accompanied by
DEB_BUILD_OPTIONS=nocheck.

For the use-case of choosing whether to build tests and ship them
in a .deb for autopkgtest or manual testing, we have the noinsttest
build-profile ("Disable binary packages consisting entirely of automated
tests, manual tests, example/demo programs and test tools").

For example, see dbus, which uses GLib in many of its automated tests,
but not in its production code. It build-depends on

libglib2.0-dev  

which means it usually needs GLib, but if it is built with *both* nocheck
and noinsttest (meaning don't run build-time tests, *and* don't build
dbus-tests_*.deb for later use by autopkgtest) then GLib isn't required.

smcv

Re: nocheck (don't run) vs nodoc (don't build)

[Christian Kastner]

Hi Russ,

thanks for the fast reply!

On 2023-04-26 20:42, Russ Allbery wrote:
> Christian Kastner  writes:
>> I thought this line of reasoning was sound, but then I remembered the
>> 'nodoc' tag and now I am no longer sure. Maybe I'm taking the 'nocheck'
>> description too literally.
> 
> I think this may be semi-accidental stemming from the affect on build
> dependencies.  One of the points of nodoc is to avoid needing to install
> (and thus have available) all the build dependencies that are used only
> for building the documentation, which often involves a full TeXLive
> installation and thus poses time, space, and bootstrapping issues.  So I
> think we made specific note that the documentation wouldn't be built so
> that those build dependencies wouldn't be required.

Yep, that was my understanding, too.

> The same argument probably applies to the test suite, I think?  It's just
> less common (although certainly not unheard of) for test suites to have
> test-suite-only build dependencies (as opposed to test-only runtime
> dependencies, which are very common in at least the Perl world).

We (the ROCm team) have a few instances of such test suites, namely
header-only libraries. Upstream ships tests that require compilation,
and we'd like to ship these tests for autopkgtest purposes.

[Tangential: our autopkgtests cannot be run on official infra yet as
they require an AMD GPU to run, but that's another avenue we're working
on. The issue here should also apply to any other header-only library.]

> Anyway, I think this warrants a bug and some work on figuring out what we
> really want to say, and I'd want the folks working on bootstrapping and
> build profiles to weigh in.  The nocheck build profile sort of implies
> that the test suite shouldn't be built either:
> 
> No test suite should be run, and build dependencies used only for that
> purpose should be ignored. Builds that set this profile must also add
> nocheck to DEB_BUILD_OPTIONS
> 
> It still doesn't say this explicitly, but it says build dependencies
> should be ignored, so presumably the build would fail if it required any
> build dependcies, implying that it shouldn't be run.

Strictly speaking, it says build dependencies for the purposes of
_running_ should be ignored, which makes perfect sense of course. But
what if there is another purpose, specifically shipping autopkgtests?
Like packages 'systemd-tests', or 'fwupd-tests'?

For a Python package, this is easy: I can skip any build dependencies
needed for build-time testing (eg: python3-pytest), and since Python
test suites generally don't need builds, I can still ship full test
suite as an autopkgtest. So nocheck "just works".

For a C or C++ library though, skipping build dependencies for tests
means empty autopkgtests. So far, I just assumed that this was
wrong/contradictory, and thus always built them.

But then I remembered nodoc, and how that too can lead to possibly empty
and thus unusable packages. And how that's ok, since such builds aren't
meant for general use, but for the specific use of the caller who
explicitly requested nocheck/nodoc in the first place, so they know what
they are doing.

> Anyway, I think this warrants a bug

I'll try to improve and to compact my reasoning, and will file one when
it's done -- unless you'd prefer me not to do it myself, or somebody
else beats me to it.

But that will take me a while, as I clearly need to think this through a
bit more. Because it just dawned me that for a header-only library, the
argument could be made that autopkgtests should always build the tests
from scratch anyways. Or maybe even both (run, and build-and-run).

Best,
Christian

Re: nocheck (don't run) vs nodoc (don't build)

[Russ Allbery]

Christian Kastner  writes:

> Policy 4.9.1 states that (emphases mine):
>   * "[nocheck] says not to *run* any build-time test suite"
>   * "[nodoc] says to skip any *build* steps"

> My reading with regards to 'nocheck' was that where tests were available
> and needed to be built, then they should always be built, just not run.

> A typical example might be a C library package that builds those tests
> and ships them as autopkgtests, maybe even in a dedicated package.

> I thought this line of reasoning was sound, but then I remembered the
> 'nodoc' tag and now I am no longer sure. Maybe I'm taking the 'nocheck'
> description too literally.

I think this may be semi-accidental stemming from the affect on build
dependencies.  One of the points of nodoc is to avoid needing to install
(and thus have available) all the build dependencies that are used only
for building the documentation, which often involves a full TeXLive
installation and thus poses time, space, and bootstrapping issues.  So I
think we made specific note that the documentation wouldn't be built so
that those build dependencies wouldn't be required.

The same argument probably applies to the test suite, I think?  It's just
less common (although certainly not unheard of) for test suites to have
test-suite-only build dependencies (as opposed to test-only runtime
dependencies, which are very common in at least the Perl world).

Anyway, I think this warrants a bug and some work on figuring out what we
really want to say, and I'd want the folks working on bootstrapping and
build profiles to weigh in.  The nocheck build profile sort of implies
that the test suite shouldn't be built either:

No test suite should be run, and build dependencies used only for that
purpose should be ignored. Builds that set this profile must also add
nocheck to DEB_BUILD_OPTIONS

It still doesn't say this explicitly, but it says build dependencies
should be ignored, so presumably the build would fail if it required any
build dependcies, implying that it shouldn't be run.

-- 
Russ Allbery (r...@debian.org)  

nocheck (don't run) vs nodoc (don't build)

[Christian Kastner]

Hi,

Policy 4.9.1 states that (emphases mine):
  * "[nocheck] says not to *run* any build-time test suite"
  * "[nodoc] says to skip any *build* steps"

My reading with regards to 'nocheck' was that where tests were available
and needed to be built, then they should always be built, just not run.

A typical example might be a C library package that builds those tests
and ships them as autopkgtests, maybe even in a dedicated package.

I thought this line of reasoning was sound, but then I remembered the
'nodoc' tag and now I am no longer sure. Maybe I'm taking the 'nocheck'
description too literally.


Is there some established consensus on this that just needs to be
codified? Or are there some common practices?

Is is something worth opening a wishlist bug for?

Best,
Christian