Hello, Thanks for the input!
On 7/1/25 15:24, Bob Friesenhahn wrote:
GraphicsMagick uses a custom TAP-test script (patterned on the example
one from Automake) where there is support for deciding if the test is
expected to fail based on a list of features. For example, the support
for a file format may depend on an optional library and that library is
a "feature". The test_command_fn() function has this usage description:
# Test a simple command where pass/fail depends on command exit status
#
# Usage:
# test_command_fn description [ -F 'feat1 feat2 ...'] command
#
# Where:
# description -- test description
# -F -- optional space-delimited required features
# command -- command to execute
Well, I forgot to clarify in the original email, but my particular target
use-case is is "unit tests" written in code, say for a particular
function or
class. In that case it gets cumbersome to have one binary per test
(atleast IMO).
My idea is to add support for a TAP output like so:
[not] ok 5 - Call sys_open with negative file descriptor # XFAIL
Where the trailing XFAIL directive would put it in the output for XFAIL,
and consider the test "passing". If it were to be an "ok" it would count
as a
failing test.
A typical test expression looks like:
test_command_fn "MNG ${type}" -F PNG ${MEMCHECK} ${rwfile} -filespec
"out_${type}_%d" "${SRCDIR}/input_${type}.miff" MNG
so in the above one can see that "PNG" is a feature (in this case
depending on libpng being present).
This strategy has been working well for a great many years.
The ability to supply a custom test functions script which is matched to
the software being tested is quite useful.
The idea of adding a directive that certain tests are expected to fail
is interesting, but the immediate feature-based approach seems much
easier (and more efficient) than trying to add indirect support, such as
via an algorithm or database lookup.
Well the algorithm I had in mind is quite simple - if a test out starts with
"ok NUM ... " but contains directive "# XFAIL" then its a failing test,
otherwise
is a "XFAIL" test which is considered passing.
Essentially its encoding the expected-fail information in the TAP
message itself
rather than in the surrounding runner script or harness.
