Re: [Test-Patch TLP] consensus on naming
On Jul 17, 2015, at 3:07 AM, Steve Loughran ste...@hortonworks.com wrote: On 16 Jul 2015, at 16:05, Allen Wittenauer a...@altiscale.com wrote: “convert this directory of TAP-formatted files to JUnit XML”. FWIW it's Ant JUnit XML Reporter-formatted XML. Just to make the origins clear: it's an ASF format written in, 2000 or 2001, probably by sballiez, who also did the XSL stylesheets. we own it, but the way its widely used now means we can't fix it directly It actually occurred to me this morning that I'm solving the wrong problem. Our development workflows don't care if Jenkins or Maven understand TAP. I just need to add TAP support to test-patch directly and make sure that the build fails when TAP tests say it should fail. This simplifies the problem tremendously.
Re: [Test-Patch TLP] consensus on naming
On 16 Jul 2015, at 16:05, Allen Wittenauer a...@altiscale.com wrote: “convert this directory of TAP-formatted files to JUnit XML”. FWIW it's Ant JUnit XML Reporter-formatted XML. Just to make the origins clear: it's an ASF format written in, 2000 or 2001, probably by sballiez, who also did the XSL stylesheets. we own it, but the way its widely used now means we can't fix it directly
Re: [Test-Patch TLP] consensus on naming
Hi Allen Just to be clear: yetus is about the layer between Jenkins and the unit tests. Many Apache communities don’t bother looking at Jenkins and it’s reporting because a) it’s extremely noisy when you’re testing literally hundreds to thousands of patches a week (plus full builds!) and b) why log into something when one can have the results sent (passive vs. active)? Plus it’s much easier to run a pre-existing script across a ton of different source bases than work on integrating xyz tool into an existing source tree. Good points. This layer between Jenkins and the unit tests seems useful. That said, I did look at tap4j last year when I was looking to integrate bats into Hadoop’s test framework. As someone who isn’t particularly proficient in Java, it wasn’t clear to me how exactly I would turn bats TAP output into something JUnit could process using tap4j. You can use something like this Perl script [1] to convert the output of bats TAP to JUnit. I tested it locally and the XML looked good. It would be possible to use tap4j for that too, but I think it would be not very efficient unless you already had a JVM running or needed to integrate it with Jenkins. Cheers Bruno [1] http://taint.org/2008/03/26/124602a.html From: Allen Wittenauer a...@altiscale.com To: common-dev@hadoop.apache.org; Bruno P. Kinoshita brunodepau...@yahoo.com.br Sent: Wednesday, July 15, 2015 4:43 AM Subject: Re: [Test-Patch TLP] consensus on naming On Jul 14, 2015, at 3:08 AM, Bruno P. Kinoshita brunodepau...@yahoo.com.br wrote: Hi Has anyone considered using TAP (Test Anything Protocol) for test reporting? [1][2] disclaimer: I'm the maintainer of the Jenkins TAP plug-in and tap4j Java library Just to be clear: yetus is about the layer between Jenkins and the unit tests. Many Apache communities don’t bother looking at Jenkins and it’s reporting because a) it’s extremely noisy when you’re testing literally hundreds to thousands of patches a week (plus full builds!) and b) why log into something when one can have the results sent (passive vs. active)? Plus it’s much easier to run a pre-existing script across a ton of different source bases than work on integrating xyz tool into an existing source tree. That said, I did look at tap4j last year when I was looking to integrate bats into Hadoop’s test framework. As someone who isn’t particularly proficient in Java, it wasn’t clear to me how exactly I would turn bats TAP output into something JUnit could process using tap4j.
Re: [Test-Patch TLP] consensus on naming
On Jul 15, 2015, at 11:22 PM, Bruno P. Kinoshita brunodepau...@yahoo.com.br wrote: Good points. This layer between Jenkins and the unit tests seems useful. Thanks! We think so too! It’s why we’re working on pulling the code out of Hadoop to make it more generalized for lots of different projects. You can use something like this Perl script [1] to convert the output of bats TAP to JUnit. I tested it locally and the XML looked good. It would be possible to use tap4j for that too, but I think it would be not very efficient unless you already had a JVM running or needed to integrate it with Jenkins. JVM startup costs aren’t a problem because we’d want to do this inside maven. So it’d be *extremely* useful if tap4j’s maven plugin had a hook that said “convert this directory of TAP-formatted files to JUnit XML”. [We post-process the JUnit XML.] I’ll take a look at the perl code. Thanks!
Re: [Test-Patch TLP] consensus on naming
Hi Has anyone considered using TAP (Test Anything Protocol) for test reporting? [1][2] disclaimer: I'm the maintainer of the Jenkins TAP plug-in and tap4j Java library I know that pointing at something created around 1988 to fix a problem with the current testing tools may sound weird, but I think using TAP rather than JUnit could work. # Test reporting The syntax is human friendly. ``` 1..3ok 1 - everything working as expected not ok 2 - something went wrongok 3``` # Concurrency Even though TAP states that is must support concurrency, we are discussing how to make it more explicit in the next spec [3], what doesn't make it impossible to report tests concurrently in near real time with TAP 13, with limitations to test order, test suites, etc. # Logging more information in the test output With JUnit/TestNG/xUnit it can be quite hard to include extra information to the test output. Using Jenkins, you can include extra information using a not standard syntax and the JUnit Attachments Plug-in [4]. But with TAP you can use YAMLish [5][6]. It was never included in any TAP specification, but has been the most widely used way to extend test results with TAP (I started using it in 2010). ``` 1..3ok 1 - everything working as expectednot ok 2 - something went wrong--- date: 2001-11-23 15:03:17 -5 user: ed fatal: Unknown variable bar stack: - file: TopClass.py line: 23 code: | x = MoreObject(345\n) - file: MoreClass.py line: 58 code: |- foo = bar...ok 3 Jenkins TAP Plug-in [7] is an example of how to extend Jenkins to support different report formats. The YAMLish can already be used to attach files in the test result. It uses tap4j [8], which would have to be changed to support tests in parallel, but it wouldn't be so hard. It is just an example, so it could be possible to write a custom reporter and not use the TAP plug-in or tap4j, but since Perl has used TAP for so long, and people have tried running tests in parallel with Test::More, Test::Parallel and prove, maybe something could be re-used for Hadoop? I think Git is also using TAP to report its tests, though I've never compiled and tested it, and I think MySQL or some of its modules/tools uses TAP as well. Another possibility would be to take a look at the existing issue in Jenkins for real time tests [9] Just my 0.02 cents. Bruno [1] http://testanything.org/philosophy.html[2] https://github.com/testanything [3] https://github.com/TestAnything/Specification/issues/15[4] https://wiki.jenkins-ci.org/display/JENKINS/JUnit+Attachments+Plugin [5] https://www.npmjs.com/package/yamlish [6] https://github.com/TestAnything/Specification/issues/11 [7] https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin [8] http://tap4j.org/[9] https://issues.jenkins-ci.org/browse/JENKINS-19154 From: Steve Loughran ste...@hortonworks.com To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org Sent: Monday, July 13, 2015 9:52 PM Subject: Re: [Test-Patch TLP] consensus on naming On 12 Jul 2015, at 22:32, Sean Busbey bus...@cloudera.com wrote: sure. what did you have in mind? Last time it was discussed we were going to wait to overhaul the repo until we have a new repo to move to. On Sat, Jul 11, 2015 at 7:40 AM, Steve Loughran ste...@hortonworks.com wrote: +1, could you structure the source tree/build so that adding new modules is easy? nothing particular, except I want to do something better with test running and reporting http://steveloughran.blogspot.co.uk/2015/05/distributed-system-testing-where-now.html http://steveloughran.blogspot.co.uk/2015/06/why-is-so-much-of-my-life-wasted.html I actually think that Yetus may be the wrong place; I want to do some baby steps with -something to import ant Junit XML and generate some aggregate (avro?) format -spark libs for analysis in ipython, zepplin of test runs and test runs across time and configurations. ultimately I'd like to be able to stream test events from test runners log events from (distributed) processes together for a linearized event log which could go into better analysis some webapp to view better, with some as-test-run reporting. I think overall this wouldn't fit into Yetus, something way downstream -but I might like some of the test log reporting in there if that helps get the dependency graph right. Like I said: want. Spare time work, which is why I didn't propose a talk on it at apachecon EU data.
Re: [Test-Patch TLP] consensus on naming
On 12 Jul 2015, at 22:32, Sean Busbey bus...@cloudera.com wrote: sure. what did you have in mind? Last time it was discussed we were going to wait to overhaul the repo until we have a new repo to move to. On Sat, Jul 11, 2015 at 7:40 AM, Steve Loughran ste...@hortonworks.com wrote: +1, could you structure the source tree/build so that adding new modules is easy? nothing particular, except I want to do something better with test running and reporting http://steveloughran.blogspot.co.uk/2015/05/distributed-system-testing-where-now.html http://steveloughran.blogspot.co.uk/2015/06/why-is-so-much-of-my-life-wasted.html I actually think that Yetus may be the wrong place; I want to do some baby steps with -something to import ant Junit XML and generate some aggregate (avro?) format -spark libs for analysis in ipython, zepplin of test runs and test runs across time and configurations. ultimately I'd like to be able to stream test events from test runners log events from (distributed) processes together for a linearized event log which could go into better analysis some webapp to view better, with some as-test-run reporting. I think overall this wouldn't fit into Yetus, something way downstream -but I might like some of the test log reporting in there if that helps get the dependency graph right. Like I said: want. Spare time work, which is why I didn't propose a talk on it at apachecon EU data.
Re: [Test-Patch TLP] consensus on naming
On Jul 13, 2015, at 2:52 AM, Steve Loughran ste...@hortonworks.com wrote: nothing particular, except I want to do something better with test running and reporting ... ultimately I'd like to be able to stream test events from test runners log events from (distributed) processes together for a linearized event log which could go into better analysis some webapp to view better, with some as-test-run reporting. ... I think overall this wouldn't fit into Yetus, something way downstream -but I might like some of the test log reporting in there if that helps get the dependency graph right. We've started to build some code to support more than one reporting mechanism at a time[1]. It probably wouldn't be that hard to write a reporter that takes the data from a finished run and ships it off to Kafka or something similar. Downstream systems could pull from Kafka (or whatever) the data and do whatever with it. Like I said: want. Spare time work, which is why I didn't propose a talk on it at apachecon EU data. [1] - technically, we already do given that we're pumping to the screen and optionally JIRA. Groundwork is laid for github PR support, but I'm not particularly happy with the design given some of the requirements I didn't fully comprehend at the time.
Re: [Test-Patch TLP] consensus on naming
sure. what did you have in mind? Last time it was discussed we were going to wait to overhaul the repo until we have a new repo to move to. On Sat, Jul 11, 2015 at 7:40 AM, Steve Loughran ste...@hortonworks.com wrote: +1, could you structure the source tree/build so that adding new modules is easy? On 10 Jul 2015, at 06:08, Kengo Seki sek...@gmail.com wrote: +1 for Yetus. Simple and distinctive. On Friday, July 10, 2015, Kengo Seki sek...@gmail.com wrote: On Wednesday, July 8, 2015, Tsuyoshi Ozawa oz...@apache.org javascript:_e(%7B%7D,'cvml','oz...@apache.org'); wrote: Hi Sean, +1 for Yetus since it sounds good name to me. Thanks - Tsuyoshi On Wed, Jul 8, 2015 at 2:42 PM, Sean Busbey bus...@cloudera.com wrote: Hi folks! It's almost time for the July board meeting, so we need to get the ball rolling on the proposal for a new TLP focused on QA[1]. Once issue outstanding from the original discussion is consensus on a name. We need to get some consensus together so that I can start verifying that the name is usable for an ASF project via a podling name search jira [2]. A brief review, so far the only stated naming preferences are Allen W's desire for something shell related and my desire for a project name beginning with the letter Y. We have a proposed name of Yetus; it's an archaic genus of sea snail and from my initial review should be usable. Any other strong feelings on naming? Any specific objections to Yetus? [1]: http://s.apache.org/yetus-discuss-hadoop [2]: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ -- Sean -- Sean
Re: [Test-Patch TLP] consensus on naming
+1, could you structure the source tree/build so that adding new modules is easy? On 10 Jul 2015, at 06:08, Kengo Seki sek...@gmail.com wrote: +1 for Yetus. Simple and distinctive. On Friday, July 10, 2015, Kengo Seki sek...@gmail.com wrote: On Wednesday, July 8, 2015, Tsuyoshi Ozawa oz...@apache.org javascript:_e(%7B%7D,'cvml','oz...@apache.org'); wrote: Hi Sean, +1 for Yetus since it sounds good name to me. Thanks - Tsuyoshi On Wed, Jul 8, 2015 at 2:42 PM, Sean Busbey bus...@cloudera.com wrote: Hi folks! It's almost time for the July board meeting, so we need to get the ball rolling on the proposal for a new TLP focused on QA[1]. Once issue outstanding from the original discussion is consensus on a name. We need to get some consensus together so that I can start verifying that the name is usable for an ASF project via a podling name search jira [2]. A brief review, so far the only stated naming preferences are Allen W's desire for something shell related and my desire for a project name beginning with the letter Y. We have a proposed name of Yetus; it's an archaic genus of sea snail and from my initial review should be usable. Any other strong feelings on naming? Any specific objections to Yetus? [1]: http://s.apache.org/yetus-discuss-hadoop [2]: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ -- Sean
Re: [Test-Patch TLP] consensus on naming
On Wednesday, July 8, 2015, Tsuyoshi Ozawa oz...@apache.org wrote: Hi Sean, +1 for Yetus since it sounds good name to me. Thanks - Tsuyoshi On Wed, Jul 8, 2015 at 2:42 PM, Sean Busbey bus...@cloudera.com javascript:; wrote: Hi folks! It's almost time for the July board meeting, so we need to get the ball rolling on the proposal for a new TLP focused on QA[1]. Once issue outstanding from the original discussion is consensus on a name. We need to get some consensus together so that I can start verifying that the name is usable for an ASF project via a podling name search jira [2]. A brief review, so far the only stated naming preferences are Allen W's desire for something shell related and my desire for a project name beginning with the letter Y. We have a proposed name of Yetus; it's an archaic genus of sea snail and from my initial review should be usable. Any other strong feelings on naming? Any specific objections to Yetus? [1]: http://s.apache.org/yetus-discuss-hadoop [2]: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ -- Sean
Re: [Test-Patch TLP] consensus on naming
+1 for Yetus. Simple and distinctive. On Friday, July 10, 2015, Kengo Seki sek...@gmail.com wrote: On Wednesday, July 8, 2015, Tsuyoshi Ozawa oz...@apache.org javascript:_e(%7B%7D,'cvml','oz...@apache.org'); wrote: Hi Sean, +1 for Yetus since it sounds good name to me. Thanks - Tsuyoshi On Wed, Jul 8, 2015 at 2:42 PM, Sean Busbey bus...@cloudera.com wrote: Hi folks! It's almost time for the July board meeting, so we need to get the ball rolling on the proposal for a new TLP focused on QA[1]. Once issue outstanding from the original discussion is consensus on a name. We need to get some consensus together so that I can start verifying that the name is usable for an ASF project via a podling name search jira [2]. A brief review, so far the only stated naming preferences are Allen W's desire for something shell related and my desire for a project name beginning with the letter Y. We have a proposed name of Yetus; it's an archaic genus of sea snail and from my initial review should be usable. Any other strong feelings on naming? Any specific objections to Yetus? [1]: http://s.apache.org/yetus-discuss-hadoop [2]: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ -- Sean
Re: [Test-Patch TLP] consensus on naming
Hi Sean, +1 for Yetus since it sounds good name to me. Thanks - Tsuyoshi On Wed, Jul 8, 2015 at 2:42 PM, Sean Busbey bus...@cloudera.com wrote: Hi folks! It's almost time for the July board meeting, so we need to get the ball rolling on the proposal for a new TLP focused on QA[1]. Once issue outstanding from the original discussion is consensus on a name. We need to get some consensus together so that I can start verifying that the name is usable for an ASF project via a podling name search jira [2]. A brief review, so far the only stated naming preferences are Allen W's desire for something shell related and my desire for a project name beginning with the letter Y. We have a proposed name of Yetus; it's an archaic genus of sea snail and from my initial review should be usable. Any other strong feelings on naming? Any specific objections to Yetus? [1]: http://s.apache.org/yetus-discuss-hadoop [2]: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ -- Sean
[Test-Patch TLP] consensus on naming
Hi folks! It's almost time for the July board meeting, so we need to get the ball rolling on the proposal for a new TLP focused on QA[1]. Once issue outstanding from the original discussion is consensus on a name. We need to get some consensus together so that I can start verifying that the name is usable for an ASF project via a podling name search jira [2]. A brief review, so far the only stated naming preferences are Allen W's desire for something shell related and my desire for a project name beginning with the letter Y. We have a proposed name of Yetus; it's an archaic genus of sea snail and from my initial review should be usable. Any other strong feelings on naming? Any specific objections to Yetus? [1]: http://s.apache.org/yetus-discuss-hadoop [2]: https://issues.apache.org/jira/browse/PODLINGNAMESEARCH/ -- Sean
