On Thu, 31 Oct 2013 07:17:03 -0400 (EDT) Josef Skladanka <jskla...@redhat.com> wrote:
Bah, just realized this only went to Josef the first time I sent it. He gets two copies! > Tim (and of course the rest of the gang ;)), > > During our chat with Tim, we agreed that we'd really like to use some > standardized 'output format' for the tests in Taskbot, to be a bit > more programming-language/results-store-implementation agnostic. We > knew about two options - TAP > <http://testanything.org/wiki/index.php/Main_Page> and Subunit > <https://pypi.python.org/pypi/python-subunit>. > > = Subunit = > > At least in Python is quite tightly coupled with unittest, both > ideologically and practically. I was unable to find a way to just > produce a Subunit stream without the need of actually running a > testsuite. > > The format is (basically) just plain PASS/FAIL/INFO/..., and there is > possibility to add some 'details'results. It should also be possible > to add an attachment to the end of the stream, but no result-specific > data can be added (IMO). > > Also Subunit is now in the process of transition to new > implementation, which now should fix some issues with concurrency, > add more result-states, etc., but will be binary, so human > readability would quite suffer. > > I do not really feel that this is a good match for our needs (at > least without any significant hacking) Yeah, this matches what I remember finding when I looked into this a couple of months ago. It looks like a better protocol overall but it's going to be quite a bit more upfront work to support than TAP. I know that openstack's test suite uses subunit for nova and neutron [1] [1] https://wiki.openstack.org/wiki/Testr Subunit has been recently added to the fedora repos but I suppose that's of little utility for us if we end up having to re-implement parts to get away from UnitTest. > = TAP = > > TAP is not unittest-specific, and is human-readable plaintext format. > > It also has just PASS/FAIL logic, but there is a possibility to add > YAML 'metadata' to any result (since TAP v. 13). > > The real issue with TAP is Python support. > There is a TAP-consumer library created as an example for PyParsing > <http://pyparsing.wikispaces.com/file/detail/TAP.py>, but it does not > support the v13 protocol, and is quite useless as such. TAP-producer > library for Python <https://github.com/rjbs/pytap> is also using the > old version (i.e. no YAML extensions), and seems to be dead (2 years > since last update). It is also quite badly written. Another option for TAP emission is bayeux [2] which was created for a better, less perl-ish interface to tap [3]. It might be worth asking mcepl if he has any thoughts on TAP vs. subunit vs. something else. [2] http://luther.ceplovi.cz/git/bayeux.git/ [3]http://lists.idyll.org/pipermail/testing-in-python/2012-March/004881.html > = Result = > > Although neither choice is ideal, I feel that TAP would be the better > choice, even though it would mean implementing our own > producer/parser. Also the TAP is really simplistic format, so > creating a TAP output should be quite easy even in any programming > language. I think that TAP is certainly appears to be the simpler solution for now and I suspect it's a bit more widely used than subunit. As long as the amount of work required to have a TAP emitter and parser isn't crazy, I agree that TAP is a better choice. > It is possible that I somehow utterly misunderstood the Subunit > concept, so it might be useful to contact some QAs currently using it > (I thing Tim mentioned something about OpenStack?), or contact the > developer directly. I suspect that you found this when searching but there's a direct comparison of the two by someone with more of a TAP background: http://www.kinoshita.eti.br/2011/06/04/a-comparison-of-tap-test-anything-protocol-and-subunit.html Thanks for looking into this, Tim
signature.asc
Description: PGP signature
_______________________________________________ qa-devel mailing list qa-de...@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/qa-devel