Hi
Has anyone considered using TAP (Test Anything Protocol) for test reporting?
[1][2]
disclaimer: I'm the maintainer of the Jenkins TAP plug-in and tap4j Java library
I know that pointing at something created around 1988 to fix a problem with the
current testing tools may sound weird, but I think using TAP rather than JUnit
could work.
# Test reporting
The syntax is human friendly.
```
1..3ok 1 - everything working as expected
not ok 2 - something went wrongok 3```
# Concurrency
Even though TAP states that is must support concurrency, we are discussing how
to make it more explicit in the next spec [3], what doesn't make it impossible
to report tests concurrently in near real time with TAP 13, with limitations to
test order, test suites, etc.
# Logging more information in the test output
With JUnit/TestNG/xUnit it can be quite hard to include extra information to
the test output. Using Jenkins, you can include extra information using a not
standard syntax and the JUnit Attachments Plug-in [4]. But with TAP you can use
YAMLish [5][6]. It was never included in any TAP specification, but has been
the most widely used way to extend test results with TAP (I started using it in
2010).
```
1..3ok 1 - everything working as expectednot ok 2 - something went wrong---
date: 2001-11-23 15:03:17 -5
user: ed
fatal:
Unknown variable "bar"
stack:
- file: TopClass.py
line: 23
code: |
x = MoreObject("345\n")
- file: MoreClass.py
line: 58
code: |-
foo = bar...ok 3
Jenkins TAP Plug-in [7] is an example of how to extend Jenkins to support
different report formats. The YAMLish can already be used to attach files in
the test result. It uses tap4j [8], which would have to be changed to support
tests in parallel, but it wouldn't be so hard.
It is just an example, so it could be possible to write a custom reporter and
not use the TAP plug-in or tap4j, but since Perl has used TAP for so long, and
people have tried running tests in parallel with Test::More, Test::Parallel and
prove, maybe something could be re-used for Hadoop? I think Git is also using
TAP to report its tests, though I've never compiled and tested it, and I think
MySQL or some of its modules/tools uses TAP as well. Another possibility would
be to take a look at the existing issue in Jenkins for real time tests [9]
Just my 0.02 cents.
Bruno
[1] http://testanything.org/philosophy.html[2] https://github.com/testanything
[3] https://github.com/TestAnything/Specification/issues/15[4]
https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin
[5] https://www.npmjs.com/package/yamlish
[6] https://github.com/TestAnything/Specification/issues/11
[7] https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin
[8] http://tap4j.org/[9] https://issues.jenkins-ci.org/browse/JENKINS-19154
From: Steve Loughran <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Monday, July 13, 2015 9:52 PM
Subject: Re: [Test-Patch TLP] consensus on naming
> On 12 Jul 2015, at 22:32, Sean Busbey <[email protected]> wrote:
>
> sure. what did you have in mind?
>
> Last time it was discussed we were going to wait to overhaul the repo until
> we have a new repo to move to.
>
> On Sat, Jul 11, 2015 at 7:40 AM, Steve Loughran <[email protected]>
> wrote:
>
>> +1,
>>
>> could you structure the source tree/build so that adding new modules is
>> easy?
nothing particular, except I want to do something better with test running and
reporting
http://steveloughran.blogspot.co.uk/2015/05/distributed-system-testing-where-now.html
http://steveloughran.blogspot.co.uk/2015/06/why-is-so-much-of-my-life-wasted.html
I actually think that Yetus may be the wrong place; I want to do some baby
steps with
-something to import ant Junit XML and generate some aggregate (avro?) format
-spark libs for analysis in ipython, zepplin of test runs and test runs across
time and configurations.
ultimately I'd like to be able to stream test events from test runners & log
events from (distributed) processes together for a linearized event log which
could go into better analysis & some webapp to view better, with some
as-test-run reporting.
I think overall this wouldn't fit into Yetus, something way downstream -but I
might like some of the test & log reporting in there if that helps get the
dependency graph right.
Like I said: want. Spare time work, which is why I didn't propose a talk on it
at apachecon EU data.
