Hey All.
The following gives a mini tutorial on writing test programs for GLib
and Gtk+ with the new framework. We have a good number of example test
programs in SVN now and appreciate help from everyone in implementing
new tests.
First, we'll have a quick introduction into the main rationale on
test writing.
The main goals in writing tests are:
- In example (test) driven development (EDD or TDD), an example or test
program is written first, and then the newly used API gets implemented.
This ensures early testability and gives programmers early feedback on
their implementation.
- For untested legacy code, new tests are written to get about to be changed
code portions under automated tests to catch behavior changes as code is
changed.
- Most tests are written to ensure basic functionality checks of the code
area in change. Tests usually cannot perform comprehensive checks, but
can check for specific known to be tricky corner cases or often test a
representative subset of an API.
In general, working on code that is sufficiently under automated tests
makes programmers feel much more confident about their changes and helps
to spot errors shortly after introduction. So well tested code bases
tend to increase productivity and fun in working with the code.
The following list of steps is hopefully helpful when approaching the
implementation of a new test for GLib or Gtk+:
1) Figure a place for the test case. For this it's useful to keep in mind
that make check will traverse CWD recursively. So tests that should be
run often when glib, gdk or gtk changed should go into glib/glib/tests/,
gtk+/gtk/tests/ or gtk+/gdk/tests/. Tests more thorough or planned to
be run less frequently can go into glib/tests/ or gtk+/tests/. This is
e.g. the case for the generic object property tester in
gtk+/tests/objecttests.c. To sum up:
glib/tests/ # less frequently run GLib tests
glib/glib/tests/ # frequent GLib testing
glib/gobject/tests/ # frequent GObject testing
gtk+/tests/ # less frequently run Gdk & Gtk+ tests
gtk+/gdk/tests/ # frequent Gdk testing
gtk+/gtk/tests/ # frequent Gtk+ testing
Also, not all tests need to go into an extra test binary. Building and
linking many test binaries can be quite time consuming, so linking
multiple .c files with tests into a single test binary can be advisable.
2) Write the test fixture setup and teardown code if necessary.
See e.g. ScannerFixture in glib/tests/scannerapi.c for a simple
example fixture that creates and sets up an object to be used
freshly in various different tests.
3) Implement the actual test function, possibly taking a fixture argument.
Tests should try to avoid duplicating logic or tests and often consist
of a series of calls and checks to use a component and verify its
behavior, e.g.:
string = g_string_new ("first");
g_assert_cmpstr (string->str, ==, "first");
g_string_append (string, "last");
g_assert_cmpstr (string->str, ==, "firstlast");
The current set of useful test assertions provided by GLib is:
g_assert_not_reached ();
g_assert (expression);
g_assert_cmpstr (s1, cmpop, s2);
g_assert_cmpint (n1, cmpop, n2);
g_assert_cmpuint (n1, cmpop, n2);
g_assert_cmphex (n1, cmpop, n2);
g_assert_cmpfloat (n1, cmpop, n2);
Where 'cmpop' is the compare operator, such as '==' or '>='.
Of course g_error() can also be used once a test error is discovered.
Note that g_warning() will usually also abort test programs, because
tests generally run with --g-fatal-warnings enabled.
4) Verify stdout and stderr output or assert failures.
Tests can be started in a separate forked off sub process to capture
premature failure, exit status and output. Here is a sample snippet:
if (g_test_trap_fork (0, G_TEST_TRAP_SILENCE_STDOUT |
G_TEST_TRAP_SILENCE_STDERR))
{
g_warning ("harmless warning with parameters: %d %s %#x", 42, "Boo",
12345);
exit (0); // should never be triggered
}
g_test_trap_assert_failed(); // we have fatal-warnings enabled
g_test_trap_assert_stderr ("*harmless warning*");
More example uses of the test_trap API can be found in:
glib/tests/testglib.c
glib/tests/scannerapi.c
glib/glib/tests/testing.c
5) Conditionalize slow or fragile tests.
While unit tests are most effective if they are fast, to allow quick
turn around times during development, slow or more thorough tests
also have their place. Test routines can be conditionalized in case
they contain fragile or slow code with the following API:
gboolean g_test_perf (); // TRUE to enable (slow) performance tests
gboolean g_test_slow (); // TRUE to execute possibly slow test code
gboolean g_test_thorough (); // TRUE to execute possibly fragile code
gboolean g_test_verbose (); // TRUE to enable additional info output
For instance gtk+/tests/objecttests.c has a black list of "suspected to
be buggy" Gtk+ property implementation. Testing and verification of the
properties on this blacklist is conditionalized with g_test_thorough(),
so testing these properties doesn't break "make check", but the errors
still show up when doing "make full-report" (more on testing related
Makefile rules later).
6) Hook up the test in the test program. The simplest test program is:
int
main (int argc,
char *argv[])
{
gtk_test_init (&argc, &argv); // initialize test program
g_test_add_func ("/TestProgramName/Test Case Name",
test_case_test_func);
return g_test_run();
}
The g_test_add() function can be used to hook up tests with Fixtures:
g_test_add ("/scanner/symbols", // test case name
ScannerFixture, // fixture structure type
NULL, // unused data argument
scanner_fixture_setup, // fixture setup
test_scanner_symbols, // test function
scanner_fixture_teardown); // fixture teardown
7) Integrate the test binary into the build and test rules.
For GLib and Gtk+, all that is needed is to add a few lines to
the respective Makefile.am, e.g.:
+++ gtk+/gtk/tests/Makefile.am
@@ -24,7 +24,11 @@ noinst_PROGRAMS = $(TEST_PROGS)
TEST_PROGS += testing
testing_SOURCES = testing.c
testing_LDADD = $(progs_ldadd)
+TEST_PROGS += liststore
+liststore_SOURCES = liststore.c
+liststore_LDADD = $(progs_ldadd)
+
TEST_PROGS += treestore
treestore_SOURCES = treestore.c
treestore_LDADD = $(progs_ldadd)
8) Execute the tests.
Currently, GLib and Gtk+ support four Makefile targets related
to tests, one of which is hooked up to automake's check rule:
make test # run all tests recursively from $(TEST_PROGS),
# abort on first error
make test-report # run all tests recursively,
# ignore errors, generate test-report.xml
make perf-report # run all tests recursively, enable performance tests,
# ignore errors, generate perf-report.xml
make full-report # run all tests recursively, enable performance tests,
# enable slow/thorough tests,
# ignore errors, generate full-report.xml
make check # run make test in addition to automake checks
For Gtk+, the tests from $(TEST_PROGS) will be executed within an Xvfb(1)
server, to avoid interactions with a currently running session.
On an aside, the XML files generated by gtester from the *-report rules
are of course not that interesting for humans. The last bit of our testing
framework implementation in GLib and Gtk+ is to generate an overview of
all the test results of a test run in HTML files from the XML logs.
---
ciaoTJ
_______________________________________________
gtk-devel-list mailing list
gtk-devel-list@gnome.org
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
