Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
Mathieu Lirzin skribis: > As suggested, I've created a “wip-check” for this patch. Cool. [...] >> Is GNU cpio available in $PATH? Which version is that? Could you run >> this test in ‘master’ to see if the problem shows up? > > Yeah the test pass in master. So I suppose I can leave the burden on you to find out what’s going on? :-) Ludo’.
Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
Hi, As suggested, I've created a “wip-check” for this patch. l...@gnu.org (Ludovic Courtès) writes: > Mathieu Lirzin skribis: > >> l...@gnu.org (Ludovic Courtès) writes: > > [...] > One caveat is that ‘tests/cpio.scm‘ is now failing. >>> >>> Why is that? Does it relate to this change? >> >> I didn't try to debug the problem but it has appeared when I started >> messing with redirection of the output/error ports. Here is the failure >> output. >> >> test-name: bit-identical to GNU cpio's output >> location: /home/mthl/src/gnu/guix/tests/cpio.scm:49 > > Is GNU cpio available in $PATH? Which version is that? Could you run > this test in ‘master’ to see if the problem shows up? Yeah the test pass in master. $ which cpio /bin/cpio $cpio --version cpio (GNU cpio) 2.11 Copyright (C) 2010 Free Software Foundation, Inc. [...] Thanks for looking into it, -- Mathieu Lirzin
Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
Mathieu Lirzin skribis: > l...@gnu.org (Ludovic Courtès) writes: [...] >>> One caveat is that ‘tests/cpio.scm‘ is now failing. >> >> Why is that? Does it relate to this change? > > I didn't try to debug the problem but it has appeared when I started > messing with redirection of the output/error ports. Here is the failure > output. > > test-name: bit-identical to GNU cpio's output > location: /home/mthl/src/gnu/guix/tests/cpio.scm:49 Is GNU cpio available in $PATH? Which version is that? Could you run this test in ‘master’ to see if the problem shows up? >>> Since this script is not intented to be exclusively used by Guix, I have >>> used a generic copyright notice. I guess Guix is the best place to >>> challenge and improve it, but IMO it will be better hosted elsewhere >>> like in Gnulib. Opinions? >> >> I think we could start using it and testing it for a while in Guix, and >> eventually submit it for inclusion in Gnulib once we are more confident. > > So you recommend to add ”this file is part of GNU Guix” and use “GNU > Guix” instead of “this program” for now? I think you can leave “this program” so nothing will need to be changed when you move it elsewhere. >> However, I’m unsure if we should push it now, or after the release. On >> one hand, I’d rather avoid potentially disrupting changes like this >> now. On the other hand, since it makes it easier (and different) to >> report test failures, it’s nice. >> >> How confident are you? :-) > > IMHO we should wait after the next release in order to make this test > driver more bullet proof. Sounds reasonable. Thanks! Ludo’.
Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
Mathieu Lirzin skribis: > Mathieu Lirzin writes: > >> l...@gnu.org (Ludovic Courtès) writes: >>> Awesome! Are the “inner” tests displayed by default? Or is there some >>> environment variable to control that? >> >> Yes, each test case is displayed by default. >> >>> I’d prefer the default to display only file names, as is currently the >>> case (it’s more concise.) >> >> IMO This would be reasonable to have an option for making the output more >> concise since there are 563 test cases in Guix. this is not part of the >> test driver API specification, but it does not seem too hard to add an >> additional option to the test driver script like what the TAP driver is >> doing: >> >> >> https://www.gnu.org/software/automake/manual/automake.html#Use-TAP-with-the-Automake-test-harness >> >> I think ‘--brief’ could be a good name for this option. Default >> verbosity would specified by adding “AM_SCM_LOG_DRIVER= --brief=yes” in >^^^ >AM_SCM_LOG_DRIVER_FLAGS Great. > From 8a1f52e08c8d33a33066271f0f39d6072baf9854 Mon Sep 17 00:00:00 2001 > From: Mathieu Lirzin > Date: Mon, 26 Oct 2015 23:47:24 +0100 > Subject: [PATCH] build: Add a scheme custom test driver using SRFI-64. > > This provides support for multiple scheme test cases in a unique file and > fixes the fragmentation of '.log' files. > > * build-aux/test-driver.scm: New file. > * Makefile.am (SCM_LOG_DRIVER, AM_SCM_LOG_DRIVER_FLAGS): New variables. > (SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables. > (AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0. > * doc/guix.texi (Running the Test Suite): Reporting a bug does not > require to provide a specific '.log' file anymore. > * tests/base32.scm, tests/build-utils.scm, tests/builders.scm, > tests/challenge.scm, tests/containers.scm, tests/cpan.scm, > tests/cpio.scm, tests/cran.scm, tests/derivations.scm, tests/elpa.scm, > tests/file-systems.scm, tests/gem.scm, tests/gexp.scm, > tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm, > tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm, > tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm, > tests/publish.scm, tests/pypi.scm, tests/records.scm, > tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm, > tests/snix.scm, tests/store.scm, tests/substitute.scm, > tests/syscalls.scm, tests/ui.scm, tests/union.scm, tests/utils.scm: > Don't exit at the end of each test. Looks nice. If you want you can push it to a wip- branch, and we’ll apply it after the release. Or you can keep it locally for later. Thanks, Ludo’.
Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
Mathieu Lirzin writes: > l...@gnu.org (Ludovic Courtès) writes: >> Awesome! Are the “inner” tests displayed by default? Or is there some >> environment variable to control that? > > Yes, each test case is displayed by default. > >> I’d prefer the default to display only file names, as is currently the >> case (it’s more concise.) > > IMO This would be reasonable to have an option for making the output more > concise since there are 563 test cases in Guix. this is not part of the > test driver API specification, but it does not seem too hard to add an > additional option to the test driver script like what the TAP driver is > doing: > > > https://www.gnu.org/software/automake/manual/automake.html#Use-TAP-with-the-Automake-test-harness > > I think ‘--brief’ could be a good name for this option. Default > verbosity would specified by adding “AM_SCM_LOG_DRIVER= --brief=yes” in ^^^ AM_SCM_LOG_DRIVER_FLAGS > Makefile.am, then the user can overide that with: > > make check -j4 TESTS=tests/rocket-science.scm SCM_LOG_DRIVER="--brief=no" ^^^ SCM_LOG_DRIVER_FLAGS > [...] >> Nice. “Running the Test Suite” needs to be updated accordingly. > > Thanks for reminding me this. ;) Here is an updated patch with the implementation of what is described above. As a bonus There is a bug fix for the ‘--color-tests’ option and a few docstring/comment changes. -- Mathieu Lirzin >From 8a1f52e08c8d33a33066271f0f39d6072baf9854 Mon Sep 17 00:00:00 2001 From: Mathieu Lirzin Date: Mon, 26 Oct 2015 23:47:24 +0100 Subject: [PATCH] build: Add a scheme custom test driver using SRFI-64. This provides support for multiple scheme test cases in a unique file and fixes the fragmentation of '.log' files. * build-aux/test-driver.scm: New file. * Makefile.am (SCM_LOG_DRIVER, AM_SCM_LOG_DRIVER_FLAGS): New variables. (SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables. (AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0. * doc/guix.texi (Running the Test Suite): Reporting a bug does not require to provide a specific '.log' file anymore. * tests/base32.scm, tests/build-utils.scm, tests/builders.scm, tests/challenge.scm, tests/containers.scm, tests/cpan.scm, tests/cpio.scm, tests/cran.scm, tests/derivations.scm, tests/elpa.scm, tests/file-systems.scm, tests/gem.scm, tests/gexp.scm, tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm, tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm, tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm, tests/publish.scm, tests/pypi.scm, tests/records.scm, tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm, tests/snix.scm, tests/store.scm, tests/substitute.scm, tests/syscalls.scm, tests/ui.scm, tests/union.scm, tests/utils.scm: Don't exit at the end of each test. --- Makefile.am | 7 +- build-aux/test-driver.scm | 174 ++ doc/guix.texi | 8 +-- tests/base32.scm | 3 - tests/build-utils.scm | 3 - tests/builders.scm| 3 - tests/challenge.scm | 3 - tests/containers.scm | 3 - tests/cpan.scm| 3 - tests/cpio.scm| 3 - tests/cran.scm| 3 - tests/derivations.scm | 3 - tests/elpa.scm| 3 - tests/file-systems.scm| 3 - tests/gem.scm | 3 - tests/gexp.scm| 3 - tests/graph.scm | 3 - tests/gremlin.scm | 3 - tests/hackage.scm | 3 - tests/hash.scm| 3 - tests/lint.scm| 3 - tests/monads.scm | 3 - tests/nar.scm | 3 - tests/packages.scm| 3 - tests/pk-crypto.scm | 3 - tests/pki.scm | 3 - tests/profiles.scm| 3 - tests/publish.scm | 3 - tests/pypi.scm| 3 - tests/records.scm | 3 - tests/scripts.scm | 3 - tests/services.scm| 3 - tests/sets.scm| 3 - tests/size.scm| 3 - tests/snix.scm| 3 - tests/store.scm | 3 - tests/substitute.scm | 3 - tests/syscalls.scm| 3 - tests/ui.scm | 3 - tests/union.scm | 3 - tests/utils.scm | 3 - 41 files changed, 181 insertions(+), 122 deletions(-) create mode 100644 build-aux/test-driver.scm diff --git a/Makefile.am b/Makefile.am index b2ee324..ce4cd53 100644 --- a/Makefile.am +++ b/Makefile.am @@ -268,10 +268,11 @@ TESTS = $(SCM_TESTS) $(SH_TESTS) TEST_EXTENSIONS = .scm .sh -AM_TESTS_ENVIRONMENT = abs_top_srcdir="$(abs_top_srcdir)" +AM_TESTS_ENVIRONMENT = abs_t
Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
allenge and improve it, but IMO it will be better hosted elsewhere >> like in Gnulib. Opinions? > > I think we could start using it and testing it for a while in Guix, and > eventually submit it for inclusion in Gnulib once we are more confident. So you recommend to add ”this file is part of GNU Guix” and use “GNU Guix” instead of “this program” for now? >> From 977e0586a6689df204fd9b522f8508ec318180c0 Mon Sep 17 00:00:00 2001 >> From: Mathieu Lirzin >> Date: Mon, 26 Oct 2015 23:47:24 +0100 >> Subject: [PATCH] build: Add a scheme custom test driver using SRFI-64. >> >> This provides support for multiple scheme test cases in a unique file and >> fixes the fragmentation of '.log' files. >> >> * build-aux/test-driver.scm: New file. >> * Makefile.am (SCM_LOG_DRIVER): New variable. >> (SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables. >> (AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0. >> * tests/base32.scm, tests/build-utils.scm, tests/builders.scm, >> tests/challenge.scm, tests/containers.scm, tests/cpan.scm, >> tests/cpio.scm, tests/cran.scm, tests/derivations.scm, tests/elpa.scm, >> tests/file-systems.scm, tests/gem.scm, tests/gexp.scm, >> tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm, >> tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm, >> tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm, >> tests/publish.scm, tests/pypi.scm, tests/records.scm, >> tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm, >> tests/snix.scm, tests/store.scm, tests/substitute.scm, >> tests/syscalls.scm, tests/ui.scm, tests/union.scm, tests/utils.scm: >> Don't exit at the end of each test. > > AIUI each tests/foo.scm file is still run with its own Guile instance, > in ./test-env, right? (This is important. It also means tests can run > in parallel.) > > If this is OK, it LGTM. I think it is OK :) > However, I’m unsure if we should push it now, or after the release. On > one hand, I’d rather avoid potentially disrupting changes like this > now. On the other hand, since it makes it easier (and different) to > report test failures, it’s nice. > > How confident are you? :-) IMHO we should wait after the next release in order to make this test driver more bullet proof. Thanks for your comments! -- Mathieu Lirzin
Re: [PATCH] build: Add a scheme custom test driver using SRFI-64.
Mathieu Lirzin skribis:
> While reading the Automake manual I have discovered that it provides an
> API for custom test drivers, so I've tried to implement something for
> the scheme API for test suites (SRFI-64) which Guix is using. The
> result provides support for multiple test cases in a single file, and
> allow ‘make check’ display the results individually:
>
> PASS: tests/base32.scm - bytevector->base32-string
> PASS: tests/base32.scm - base32-string->bytevector
> PASS: tests/base32.scm - nix-base32-string->bytevector
> PASS: tests/base32.scm - &invalid-base32-character
> SKIP: tests/base32.scm - sha256 & bytevector->nix-base32-string
> PASS: tests/base64.scm - empty string
Awesome! Are the “inner” tests displayed by default? Or is there some
environment variable to control that?
I’d prefer the default to display only file names, as is currently the
case (it’s more concise.)
> And the ‘.log’ files now use somekind of recutils format:
Cool!
> Finally the ‘.log’ files are no longer fragmented between
> “${top_srcdir}” and “tests/”. If something went wrong (FAIL, SKIP,
> XPASS) then the entire log file is copied in ‘test-suite.log’ so no need
> to ask for an additional file when someone is reporting some ‘make
> check’ error.
Nice. “Running the Test Suite” needs to be updated accordingly.
> One caveat is that ‘tests/cpio.scm‘ is now failing.
Why is that? Does it relate to this change?
> Since this script is not intented to be exclusively used by Guix, I have
> used a generic copyright notice. I guess Guix is the best place to
> challenge and improve it, but IMO it will be better hosted elsewhere
> like in Gnulib. Opinions?
I think we could start using it and testing it for a while in Guix, and
eventually submit it for inclusion in Gnulib once we are more confident.
> From 977e0586a6689df204fd9b522f8508ec318180c0 Mon Sep 17 00:00:00 2001
> From: Mathieu Lirzin
> Date: Mon, 26 Oct 2015 23:47:24 +0100
> Subject: [PATCH] build: Add a scheme custom test driver using SRFI-64.
>
> This provides support for multiple scheme test cases in a unique file and
> fixes the fragmentation of '.log' files.
>
> * build-aux/test-driver.scm: New file.
> * Makefile.am (SCM_LOG_DRIVER): New variable.
> (SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables.
> (AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0.
> * tests/base32.scm, tests/build-utils.scm, tests/builders.scm,
> tests/challenge.scm, tests/containers.scm, tests/cpan.scm,
> tests/cpio.scm, tests/cran.scm, tests/derivations.scm, tests/elpa.scm,
> tests/file-systems.scm, tests/gem.scm, tests/gexp.scm,
> tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm,
> tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm,
> tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm,
> tests/publish.scm, tests/pypi.scm, tests/records.scm,
> tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm,
> tests/snix.scm, tests/store.scm, tests/substitute.scm,
> tests/syscalls.scm, tests/ui.scm, tests/union.scm, tests/utils.scm:
> Don't exit at the end of each test.
AIUI each tests/foo.scm file is still run with its own Guile instance,
in ./test-env, right? (This is important. It also means tests can run
in parallel.)
If this is OK, it LGTM.
However, I’m unsure if we should push it now, or after the release. On
one hand, I’d rather avoid potentially disrupting changes like this
now. On the other hand, since it makes it easier (and different) to
report test failures, it’s nice.
How confident are you? :-)
What do others think?
Thanks!
Ludo’.
[PATCH] build: Add a scheme custom test driver using SRFI-64.
Hello Guix,
While reading the Automake manual I have discovered that it provides an
API for custom test drivers, so I've tried to implement something for
the scheme API for test suites (SRFI-64) which Guix is using. The
result provides support for multiple test cases in a single file, and
allow ‘make check’ display the results individually:
PASS: tests/base32.scm - bytevector->base32-string
PASS: tests/base32.scm - base32-string->bytevector
PASS: tests/base32.scm - nix-base32-string->bytevector
PASS: tests/base32.scm - &invalid-base32-character
SKIP: tests/base32.scm - sha256 & bytevector->nix-base32-string
PASS: tests/base64.scm - empty string
It is now possible to define individual XFAIL test cases:
(test-expect-fail "absurd")
(test-assert "absurd" (= 0 1))
And the ‘.log’ files now use somekind of recutils format:
test-name: url-fetch, file
location: /home/mthl/src/gnu/guix/tests/builders.scm:78
source:
+ (test-assert
+ "url-fetch, file"
+ (let* ((file (search-path %load-path "guix.scm"))
+ (hash (call-with-input-file file port-sha256))
+ (out (url-fetch* %store file 'sha256 hash)))
+ (and (file-exists? out) (valid-path? %store out
value: #t
result: PASS
Finally the ‘.log’ files are no longer fragmented between
“${top_srcdir}” and “tests/”. If something went wrong (FAIL, SKIP,
XPASS) then the entire log file is copied in ‘test-suite.log’ so no need
to ask for an additional file when someone is reporting some ‘make
check’ error.
One caveat is that ‘tests/cpio.scm‘ is now failing.
Since this script is not intented to be exclusively used by Guix, I have
used a generic copyright notice. I guess Guix is the best place to
challenge and improve it, but IMO it will be better hosted elsewhere
like in Gnulib. Opinions?
Comments are welcome!
--
Mathieu Lirzin
>From 977e0586a6689df204fd9b522f8508ec318180c0 Mon Sep 17 00:00:00 2001
From: Mathieu Lirzin
Date: Mon, 26 Oct 2015 23:47:24 +0100
Subject: [PATCH] build: Add a scheme custom test driver using SRFI-64.
This provides support for multiple scheme test cases in a unique file and
fixes the fragmentation of '.log' files.
* build-aux/test-driver.scm: New file.
* Makefile.am (SCM_LOG_DRIVER): New variable.
(SCM_LOG_COMPILER, AM_SCM_LOG_FLAGS): Delete variables.
(AM_TESTS_ENVIRONMENT): Set GUILE_AUTO_COMPILE to 0.
* tests/base32.scm, tests/build-utils.scm, tests/builders.scm,
tests/challenge.scm, tests/containers.scm, tests/cpan.scm,
tests/cpio.scm, tests/cran.scm, tests/derivations.scm, tests/elpa.scm,
tests/file-systems.scm, tests/gem.scm, tests/gexp.scm,
tests/graph.scm, tests/gremlin.scm, tests/hackage.scm, tests/hash.scm,
tests/lint.scm, tests/monads.scm, tests/nar.scm, tests/packages.scm,
tests/pk-crypto.scm, tests/pki.scm, tests/profiles.scm,
tests/publish.scm, tests/pypi.scm, tests/records.scm,
tests/scripts.scm, tests/services.scm, tests/sets.scm, tests/size.scm,
tests/snix.scm, tests/store.scm, tests/substitute.scm,
tests/syscalls.scm, tests/ui.scm, tests/union.scm, tests/utils.scm:
Don't exit at the end of each test.
---
Makefile.am | 6 +-
build-aux/test-driver.scm | 157 ++
tests/base32.scm | 3 -
tests/build-utils.scm | 3 -
tests/builders.scm| 3 -
tests/challenge.scm | 3 -
tests/containers.scm | 3 -
tests/cpan.scm| 3 -
tests/cpio.scm| 3 -
tests/cran.scm| 3 -
tests/derivations.scm | 3 -
tests/elpa.scm| 3 -
tests/file-systems.scm| 3 -
tests/gem.scm | 3 -
tests/gexp.scm| 3 -
tests/graph.scm | 3 -
tests/gremlin.scm | 3 -
tests/hackage.scm | 3 -
tests/hash.scm| 3 -
tests/lint.scm| 3 -
tests/monads.scm | 3 -
tests/nar.scm | 3 -
tests/packages.scm| 3 -
tests/pk-crypto.scm | 3 -
tests/pki.scm | 3 -
tests/profiles.scm| 3 -
tests/publish.scm | 3 -
tests/pypi.scm| 3 -
tests/records.scm | 3 -
tests/scripts.scm | 3 -
tests/services.scm| 3 -
tests/sets.scm| 3 -
tests/size.scm| 3 -
tests/snix.scm| 3 -
tests/store.scm | 3 -
tests/substitute.scm | 3 -
tests/syscalls.scm| 3 -
tests/ui.scm | 3 -
tests/union.scm | 3 -
tests/utils.scm | 3 -
40 files changed, 160 insertions(+), 117 deletions(-)
create mode 100644 build-aux/test-driver.scm
diff --git a/Makefile.am b/Makefile.am
index 4f90b1d..2db6b25 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -268,10 +268,10 @@ TESTS = $(SCM_TESTS) $(SH_TESTS)
TEST_EXTENSIONS = .scm .sh
-AM_TESTS_ENVIRONMENT = abs
