Hi,
One of the things motivating me to work on the meson conversion is the ability
to run tests in an easily understandable way. Meson has a testrunner that
both allows to run all tests at once, and run subsets of tests.
= Test and testsuite naming =
Each test has a unique name, and 0-n labels (called 'suites'). One can run
tests by their name, or select tests by suites.
The way I've organized it so far is that tests are named like this:
main/pg_regress
main/isolation
recovery/t/018_wal_optimize.pl
pg_prewarm/t/001_basic.pl
test_shm_mq/pg_regress
psql/t/001_basic.pl
we could also name tests by their full path, but that makes the display very
unwieldy.
At the moment there's three suites differentiating by the type of test:
'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites,
describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc.
That currently works out to each test having two suites, although I've
wondered about adding a 'contrib' suite as well.
Perhaps the pg_regress suite should just be named 'regress'? And perhaps the
t/ in the tap tests is superfluous - the display is already fairly wide?
= Log and Data locations =
To make things like the selection of log files for a specific test easier,
I've so far set it up so that test data and logs are stored in a separate
directory from the sources.
testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...>
The runner now creates a test.start at the start of a test and either
test.success or test.failure at the end. That should make it pretty easy for
e.g. the buildfarm and CI to make the logs for a failed test easily
accessible. I've spent far too much time going through the ~hundred logs in
src/test/recovery/ that the buildfarm displays as one thing.
I really like having all the test data separately from the sources, but I get
that that's not what we've done so far. It's doable to just mirror the current
choice, but I don't think we should. But I won't push too hard against keeping
things the same.
I do wonder if we should put test data and log files in a separate directory
tree, but that'd be a bit more work probably.
Any comments on the above?
= Example outputs =
Here's an example output that you mostly should be able to make sense of now:
$ m test --print-errorlogs
ninja: Entering directory `/tmp/meson'
ninja: no work to do.
1/242 postgresql:setup / tmp_install
OK 0.33s
2/242 postgresql:tap+pg_upgrade / pg_upgrade/t/001_basic.pl
OK 0.35s 8 subtests passed
3/242 postgresql:tap+recovery / recovery/t/011_crash_recovery.pl
OK 2.67s 3 subtests passed
4/242 postgresql:tap+recovery / recovery/t/013_crash_restart.pl
OK 2.91s 18 subtests passed
5/242 postgresql:tap+recovery / recovery/t/014_unlogged_reinit.pl
OK 3.01s 23 subtests passed
6/242 postgresql:tap+recovery / recovery/t/022_crash_temp_files.pl
OK 3.16s 11 subtests passed
7/242 postgresql:tap+recovery / recovery/t/016_min_consistency.pl
OK 3.43s 1 subtests passed
8/242 postgresql:tap+recovery / recovery/t/021_row_visibility.pl
OK 3.46s 10 subtests passed
9/242 postgresql:isolation+tcn / tcn/isolation
OK 3.42s
10/242 postgresql:tap+recovery / recovery/t/023_pitr_prepared_xact.pl
OK 3.63s 1 subtests passed
...
241/242 postgresql:isolation+main / main/isolation
OK 46.69s
242/242 postgresql:tap+pg_upgrade / pg_upgrade/t/002_pg_upgrade.pl
OK 57.00s 13 subtests passed
Ok: 242
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to /tmp/meson/meson-logs/testlog.txt
The 'postgresql' is because meson supports subprojects (both to provide
dependencies if needed, and "real" subprojects), and their tests can be run at
once.
If a test fails it'll show the error output at the time of test:
39/242 postgresql:pg_regress+cube / cube/pg_regress
FAIL 3.74s exit status 1
>>> REGRESS_SHLIB=/tmp/meson/src/test/regress/regress.so MALLOC_PERTURB_=44
>>> PG_TEST_EXTRA='kerberos ldap ssl'
>>> PG_REGRESS=/tmp/meson/src/test/regress/pg_regress
>>> PATH=/tmp/meson/tmp_install/tmp/meson-install/bin:/tmp/meson/contrib/cube:/home/andres/bin/perl5/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/games
>>> /usr/bin/python3 /home/andres/src/postgresql/src/tools/testwrap --srcdir
>>> /home/andres/src/postgresql/contrib/cube --basedir /tmp/meson --builddir
>>> /tmp/meson/contrib/cube --testgroup cube --testname pg_regress
>>> /tmp/meson/src/test/regress/pg_regress --temp-instance
>>> /tmp/meson/testrun/cube/pg_regress/tmp_check --inputdir
>>> /home/andres/src/postgresql/contrib/cube --expecteddir
>>> /home/andres/src/postgresql/contrib/cube --outputdir
>>> /tmp/meson/testrun/cube/pg_regress --bindir '' --dlpath
>>> /tmp/meson/contrib/cube --max-concurrent-tests=20 --port=40012 cube cube_sci
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# executing test in /tmp/meson/testrun/cube/pg_regress group cube test
pg_regress, builddir /tmp/meson/contrib/cube
============== creating temporary instance ==============
============== initializing database system ==============
============== starting postmaster ==============
running on port 40012 with PID 354981
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
============== running regression test queries ==============
test cube ... FAILED 418 ms
test cube_sci ... ok 16 ms
============== shutting down postmaster ==============
======================
1 of 2 tests failed.
======================
The differences that caused some tests to fail can be viewed in the
file "/tmp/meson/testrun/cube/pg_regress/regression.diffs". A copy of the test
summary that you see
above is saved in the file "/tmp/meson/testrun/cube/pg_regress/regression.out".
# test failed
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
40/242 postgresql:pg_regress+citext / citext/pg_regress
OK 3.77s
...
A list of failed tests is listed at the end of the test:
Summary of Failures:
39/242 postgresql:pg_regress+cube / cube/pg_regress
FAIL 3.74s exit status 1
Ok: 241
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0
Full log written to /tmp/meson/meson-logs/testlog.txt
Greetings,
Andres Freund
