ABSTRACT: The Test Anything Protocol (TAP) is a simple text-based protocol that allows communication between test scripts and a test harness.
The primary target of this project is the implementation of a TAP-consumer testsuite harness in Automake-generated Makefiles. This way, a software package using Automake and having a scripts-based testsuite would be able to run multiple checks in each test script, and communicate the result of each of those checks separately to the test harness, which could then issue more faithful statistics about the number of failed/passed/skipped checks. A secondary objective is to enhance Autotest-generated test scripts to allow them to become TAP producers. DETAILS: See attachements. -*-*- Now, in all honesty, I must say that I've chosen TAP not only for its objective qualities and merits, but also because I have some previous knowledge of it (which allowed me to present a more concrete proposal and a meaningful roadmap) and I personally like it (which will probably be a powerful motivator to overcome the unavoidable hurdles I'll encounter down the road). Still, there might be very valid competing alternatives to TAP out there, which I might not know about, but that in the long run would offer Automake more advantages and interoperabilty, thus outweighting the two "personal" advantages of TAP I've reported above. So, if anyone who's reading this has proposals about viable alternatives to TAP, please speak up -- your contribution is appreciated! Thanks, Stefano
Student Identification ====================== Name: Stefano Lattarini Email address: stefano.lattar...@gmail.com The name of the project ======================= automake - Interfacing with a test protocol like TAP or subunit Abstract ======== The Test Anything Protocol (TAP) is a simple text-based protocol that allows communication between test scripts and a test harness. The primary target of this project is the implementation of a TAP-consumer testsuite harness in Automake-generated Makefiles. This way, a software package using Automake and having a scripts-based testsuite would be able to run multiple checks in each test script, and communicate the result of each of those checks separately to the test harness, which could then issue more faithful statistics about the number of failed/passed/skipped checks. A secondary objective is to enhance Autotest-generated test scripts to allow them to become TAP producers. The parallel-tests driver and its limits ======================================== In the recent years, Automake's built-in support for the generation of testsuite harnesses has become quite sophisticated. In particular, the "parallel-tests" driver has performed very good. That driver is targeted at testsuites comprised of test scripts which are mostly independent one from the other, and safe to run in parallel. It is implemented in portable make (obviously with the contribution of portable POSIX utilities, such as the shell, awk, sed, etc.), but still manages to offer many useful and pretty advanced features (please refer to the Automake documentation for more details). Still, the parallel driver still has one major limit. In a scripts-based testsuite, it's common for a single test script to run checks for multiple (usually related) features, and/or against multiple bugs at once -- especially in case these checks require a similar (or even identical) setup. And while it would usually be feasible to separate such related checks into different test scripts, that might require a too-high overhead, either in test execution time or in programmer coding time. The problem with the current Automake parallel-tests driver (and also with its precursor, the "simple-tests" driver) is that the results of all the checks run by an individual test script are hopelessly condensed into the single final exit status of the script. So there is no way to distinguish between, say, a test script that failed because one check out of N in it failed, from one which failed because all the N checks in it failed. Possible enhancements with the TAP protocol ============================================ The Test Anything Protocol (TAP) is a simple protocol that allows communication between test scripts and a test harness. It allows individual tests (TAP producers) to communicate test results to the testing harness (TAP consumer) in a language-agnostic way. What is important for us is that, using TAP, a single test script can safely run multiple checks and report the outcome of each of them separately to the harness; the results will then be properly displayed and accounted for by the harness, with the right level of granularity. So, for example, if we have a TAP-aware test script that fails because one check out of 20 in it failed, the test harness won't count that as "1 failure, 0 passes", but as a "1 failure, 19 passes". Thus, Automake-generated testsuite harnesses could become TAP consumers, for either third-party testsuites and for Automake-generated parallel testsuite drivers. Also, Autoconf's Autotest currently use its own custom output to report test results. It could benefit of better interoperability if it start supporting TAP; in particular, the integration with Automake harnesses would be greatly improved (and probably also simplified). Some useful links on TAP: * <http://search.cpan.org/~petdance/Test-Harness/lib/Test/Harness/TAP.pod> Documentation for the TAP format understood by the perl module Test::Harness (kind of "de-facto standard"); * <http://testanything.org/wiki/index.php/Main_Page> A wiki on TAP, with useful discussions, pointers, and also bits of history. * <http://www.eyrie.org/~eagle/software/c-tap-harness/> A C library implementing both a TAP producer and a TAP consumer; should be quite mature and actively maintained. Benefits of this proposal ========================= We have already clearly described how the use of TAP would an improved granularity in the presentation/summary of test results. The TAP model also offers what I deem to be a better mental model of the typical "testing with test scripts" scenario: a testsuite contains test scripts, each of which can run multiple checks. The smallest meaningful piece of result is the outcome of a check, and *not* the exit status of a test script. This model is both more granular and more natural. TAP is well-established in the Perl community, and thus the protocol have already proven itself as valid "in the real world". This seems confirmed by the existence of quite many third-party TAP producers and consumers for different languages. There are already C/C++ libraries aimed at writing TAP-producing test programs. Thus the addition of the new TAP support in Automake would allow an easier and more proficient integration of custom C unit tests into Automake-generated testsuite harnesses. This would be particularly useful, considering that projects based on C and/or C++ are the primary client base of Automake. Finally, the TAP support in Automake could rely on the existing implementation and APIs of the parallel-tests driver, extending them in a mostly backward-compatible way. The new functionality would thus build on a well-established foundation, and place itself in thriving paradigm. Deliverables ============ Once this proposal is implemented, Automake should be able to generate TAP-consumer testsuite harnesses that works portably on many systems (ideally all those Automake strives to support) and with many vendor make implementations (ideally all those Automake strives to support). Failing this, the "fallback plan" is to have the generated testsuite harnesses to work at least with recent and older GNU make version on as many systems as possible (ideally all those Automake strives to support). Finally, if there have been enough enough time, Autotest should be enhanced so that the test scripts it generates become able to produce TAP-compatible output. Plan and Roadmap ================ Q: How will you and your mentor track your progress as you work on the project? A: Since technical discussions about design and implementation of new Automake features take place on the public lists <automake@gnu.org> and/or <automake-patc...@gnu.org>, most measurable progresses are publicly available there, and thus quite easy to judge. Also, every new feature going in Automake must have proper testsuite coverage, so that we can be confident that the new code will really works. Finally, the project mentor has been the Automake maintainer since 4/5 years now, so that he's surely able to judge how a partial implementation if faring, and what the chances are for it to be completed in time. Q: How the mid-term evaluation of your project will be made? A: I must have an implementation of Automake-generated TAP-consumer testsuite harness that works at least with non-obsolete versions of GNU make at least on GNU/Linux, Solaris 10 and FreeBSD 8. This implementation must comprise test cases (maybe non exhaustive) and documentation (also unpolished, but basically complete). Q: Remember to mention any periods during the summer when you won't actually be available to work on the project (though remember, the Summer of Code project is expected to be your main activity). A: I plan to be available for basically all the period of the project. If not the whole day, at least half a day, basically each day (either morning and early afternoon, or later afternoon and evening). There might be *very few* short periods (one, max two days) when I'll be completely off-line and "away from keyboard" -- I'll let the mentor know about them in advance, obviously. Communication ============= First, I'm already in contact with the mentor, due to my on-going involvement with the Automake project. I've also been in contact with the backup mentor, due to my (small) contributions to Autoconf. In the Automake and Autoconf projects, discussions about design and documentation, and proposals and reviews of patches are public, taking place on the projects' pre-existing and well-established mailing lists. The mentor reads them regularly, so that posting patches an ideas there is a sure way to have them considered and reviewed by him (and possibly also by other mailing list readers, which might contribute with their own useful insights, questions, and experiences). Finally, my experiences have showed that real-time interaction with the mentor to work on a particular issue (either on-list or off-list) is not only possible, but also efficient and effective. Qualification ============= I'm quite familiar with the policies of both the Automake and Autoconf projects (especially coding standards, patch review process, and portability concerns). Moreover, the copyright assignment to the Free Software Foundation that allows me to contribute code to the Automake and Autoconf official repositories is already in place (and has been for more than a year and a half now). I've started contributing to the Automake project an year and a half ago. During the last year, my involvement with the project has steadily increased: I've posted more patches and participated more actively to discussions on the mailing lists, I've been granted commit rights, and I've started helping with the bug tracker and related issues. I plan to continue my involvement with Automake in the foreseeable future. I have access to some systems to test my work on: Debian GNU/Linux (with custom installation of older version of tools and compilers), FreeBSD 8.0, and Solaris 10. This can help me in finding out early potential portability problems. I believe that powerful, easy-to-use testing frameworks are vital for the development of modern software. Also, I find software testing an interesting, and sometimes even fascinating, topic. So I can say I'm motivated enough to work in this area.
Tentative roadmap for GSoC project "automake - Interfacing with a test protocol like TAP or subunit". 1. COLLECT INFORMATION: Before April 20 ---------------------------------------- Read carefully the TAP specifications available. Skim the wiki at <http://testanything.org/>, and pick out useful pointers and information. Take a look at real-word usages of TAP (mostly the distribution of Perl and some major perl packages); familiarize as much as possible with the corner cases and the "rough edges", jot down some notes about things that are unclear might be deemed to cause potential problem. Finally, take a look at existing implementation (chiefly Test::Harness and c-tap-harness -- their licences are GPL-compatible, so that's OK from a legal point of view). 2. EXPERIMENTING: April 20 - May 15 (Before the official coding time) ----------------------------------------------------------------------- Write scripts (in shell and/or awk) and Makefiles that implement TAP consumers (it's OK to do so by tweaking Automake-generated Makefiles, and it's ok if they're not 100% functional). Try to make those examples as portable as possible, to spot potential portability issues early. Test at least on GNU/Linux, FreeBSD 8 and Solaris 10, and their native make implementation(s). Exchange early ideas and doubts with the mentor, and, if appropriate, also with the official mailing lists. If there's still time, take a deeper look at Autotest documentation, implementation, and history, so that I will be able to (try to) extend it to produce TAP-compatible output (time permitting). This phase should help with the subsequent "real" implementation, by providing some ideas, and quite likely also code (incomplete of course, but probably good as a starting point and/or for "cut & paste"), and tentative testcases. 3. BASIC DESIGN: May 15 - June 5 (Official coding period starts May 23) ------------------------------------------------------------------------- We'll have to decide which is the best way to integrate the TAP consumer code into Automake -- will we use new primary, a new option, or maybe some new special syntax and/or variables? This is something to be discussed publicly on the Automake mailing lists. Also, how much of the parallel-tests implementation and scaffolding can be reused? What about parallelization and/or synchronization issues? What about other issues that came up in the experimentation phase? All these doubts should be worked out. If there's time, we might also start planning how Autotest can be extended to allow it to produce TAP output. This is something to be discussed publicly on the Autoconf mailing lists. BTW, at this point, we'll probably need to go back and do some more experimenting (but more purpose-oriented than in the previous phase). And we could also write some early test cases and documentation. 4. WRITE BASIC TESTCASES AND DOCUMENTATION: June 5 - June 12 ------------------------------------------------------------- Documentation about design and general concepts shouldn't be long nor verbose, but it's important to get it right and to make it as clear as possible -- which can be tricky, and take some time. Also, condensing the most important use cases into tests is not immediate (although it shouldn't be very difficult in this phase). 5. CODING, PART 1: June 12 - July 6 (Until mid-term evaluation) ---------------------------------------------------------------- Here I'll have to extend Automake to provide TAP-aware test harnesses. The resulting code must behave as to match the existing documentation and design. It should also be able to generate Makefiles that work on at least GNU/Linux, FreeBSD and Solaris, with their native make implementation(s). If this fails, I *must* at least be able to ensure that the generated Makefiles will work with all non-obsolete versions of GNU make on all those platforms. This is not a clear-cut phase. During it, I'll have to be ready to extend and adjust the new documentation; adapt the existing testcases and add some more (probably many more); read and understand parts of the Automake codebase I'm not familiar with yet; and maybe doing more experimentation. In the very worst case, the mentor and I might even be forced to revisit and modify our previous design. 5. CODING, PART 2: July 6 - July 23 ------------------------------------ If the support for TAP-aware test harnesses I wrote in the previous phase for Automake is complete and wholly functional (which include portability to at least BSD and Solaris make, but allows for some rough edges), in this phase I'll try to enhance Autotest to enable TAP-compatible output in its generated test scripts. Otherwise, I'll have to work more on Automake TAP support. 6. DOCUMENTATION POLISHING: July 23 - July 28 ---------------------------------------------- Polish/complete documentation of all the implemented features. Maybe add some examples, and if there's time contribute them to existing third-party tutorials, FAQs and/or wikis about the autotools. 7. STRESS AND PORTABILITY TESTING: July 28 - Until done -------------------------------------------------------- Test the code on other systems which Automake strives to support (e.g., Cygwin, MSYS/MinGW, AIX, HP-UX, IRIX), and (try to) fix all the problems and incompatibilities that show up. This might require some more help from the mentor, which has access to an impressive array of testing systems (all the major proprietary Unix implementations are among them). 8. FINAL CLEANUP: All the time that's left (if any) ---------------------------------------------------- Note that this last pass is not strictly necessary, and might be omitted altogether in case we're out of time. Here we could add more tests about corner cases; polish documentation and/or code; and maybe even convert (part of) Automake's own testsuite to the use of the newly implemented TAP-based harness.