Making TODO Tests Fail
We currently have over 30,000 tests in our system. It's getting harder to manage them. In particular, it's getting harder to find out which TODO tests are unexpectedly passing. It would be handy have to some option to force TODO tests to die or bailout if they pass (note that this behavior MUST be optional). Now one might think that it would be easy to track down missing TODOs, but with 15,000 tests aggregated via Test::Aggregate, I find the following unhelpful: TODO passed: 2390, 2413 If those were in individual tests, it would be a piece of cake to track them down, aggregated tests get lumped together. Lacking proper subtest support (which might not mitigate the problem) or structured diagnostics (which could allow me to attach a lot more information to TODO tests) at the end of the day, I need an easier way of tracking this. Suggestions? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: Making TODO Tests Fail
On Mon, Jul 13, 2009 at 4:56 PM, Ovid wrote: > > We currently have over 30,000 tests in our system. It's getting harder to > manage them. In particular, it's getting harder to find out which TODO tests > are unexpectedly passing. It would be handy have to some option to force > TODO tests to die or bailout if they pass (note that this behavior MUST be > optional). > > Now one might think that it would be easy to track down missing TODOs, but > with 15,000 tests aggregated via Test::Aggregate, I find the following > unhelpful: > > > TODO passed: 2390, 2413 > > If those were in individual tests, it would be a piece of cake to track them > down, aggregated tests get lumped together. Lacking proper subtest support > (which might not mitigate the problem) or structured diagnostics (which could > allow me to attach a lot more information to TODO tests) at the end of the > day, I need an easier way of tracking this. I think it would be better to have a tool (Smolder) be able to display various drill-downs from the aggregated test report. e.g. list of all the TODOs list of all the TODOs that pass etc... That way you don't need to run your test suit again with an option but can get this information from the report of the regular run. Gabor
Re: Making TODO Tests Fail
- Original Message > From: Gabor Szabo > > I think it would be better to have a tool (Smolder) be able to display > various drill-downs from the aggregated test report. > e.g. list of all the TODOs > list of all the TODOs that pass > etc... How would Smolder (which we're not using since we use Hudson) help with this? With over 15,000 tests being reported for t/aggregate.t, I think a drill-down would be problematic here. Plus, tying the TODO to the appropriate test file being aggregated is needed. Thus, I thought about a BAILOUT or forced failure for the TODO at that point. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Tech blog- http://use.perl.org/~Ovid/journal/ Twitter - http://twitter.com/OvidPerl Official Perl 6 Wiki - http://www.perlfoundation.org/perl6
Re: Making TODO Tests Fail
Gabor Szabo wrote: I think it would be better to have a tool (Smolder) be able to display various drill-downs from the aggregated test report. If you want to see what Smolder would do to your tests, create a TAP archive and then you can upload it to the "Junk" project at http://smolder.plusthree.com (here's the upload form: http://smolder.plusthree.com/app/public_projects/add_report/9) Then you can click the "OK" or "FAILED" boxes for each stream and it will expand them out so you can see the details. TODOs are light green. So find the the one you might be interested in, click it and then see it's details. -- Michael Peters Plus Three, LP
Re: Making TODO Tests Fail
On Mon, Jul 13, 2009 at 5:10 PM, Michael Peters wrote: > Gabor Szabo wrote: > >> I think it would be better to have a tool (Smolder) be able to display >> various drill-downs from the aggregated test report. > > If you want to see what Smolder would do to your tests, create a TAP archive > and then you can upload it to the "Junk" project at > http://smolder.plusthree.com (here's the upload form: > http://smolder.plusthree.com/app/public_projects/add_report/9) > > Then you can click the "OK" or "FAILED" boxes for each stream and it will > expand them out so you can see the details. TODOs are light green. So find > the the one you might be interested in, click it and then see it's details. > AFAIK due to the number of tests it won't work well in Smolder - but I have not tried it. I was referencing to a future version of it ;-) aka another feature request... Gabor
Re: Making TODO Tests Fail
Gabor Szabo wrote: AFAIK due to the number of tests it won't work well in Smolder - but I have not tried it. I was referencing to a future version of it ;-) It's worth a try. Our main test suite at $work has 23,000+ tests and Smolder handles it just fine. -- Michael Peters Plus Three, LP
Re: Making TODO Tests Fail
* On Mon, Jul 13 2009, Ovid wrote: > How would Smolder (which we're not using since we use Hudson) help > with this? With over 15,000 tests being reported for t/aggregate.t, I > think a drill-down would be problematic here. Plus, tying the TODO to > the appropriate test file being aggregated is needed. Thus, I thought > about a BAILOUT or forced failure for the TODO at that point. Can't you just put the TAP into a database, and do a quick query to extract the file, test number, and line of passing TODO tests? Surely some code might need to be written, but it doesn't sound too hard. -- print just => another => perl => hacker => if $,=$"
Re: Making TODO Tests Fail
On Monday 13 July 2009 06:56:15 Ovid wrote: > We currently have over 30,000 tests in our system. It's getting harder to > manage them. In particular, it's getting harder to find out which TODO > tests are unexpectedly passing. It would be handy have to some option to > force TODO tests to die or bailout if they pass (note that this behavior > MUST be optional). What does dying or bailing out give you that you don't already get through TAP? Is it "The ease of a human being looking at the console window or the logs and seeing that something failed and parsing it visually?" How does a human figure out which TODO passed from: prove test.pl test.pl .. ok All tests successful. Test Summary Report --- test.pl (Wstat: 0 Tests: 2 Failed: 0) TODO passed: 1-2 There's no diagnostic information there. > Now one might think that it would be easy to track down missing TODOs, but > with 15,000 tests aggregated via Test::Aggregate, I find the following > unhelpful: > > > TODO passed: 2390, 2413 That sounds like a problem with Test::Aggregate, compounding the lack of TODO diagnostic information. > If those were in individual tests, it would be a piece of cake to track > them down, aggregated tests get lumped together. Lacking proper subtest > support (which might not mitigate the problem) or structured diagnostics > (which could allow me to attach a lot more information to TODO tests) at > the end of the day, I need an easier way of tracking this. > > Suggestions? Add diagnostics to TODO tests and let your test harness do what it's supposed to do. Shoving yet more optional behavior in the test process continues to violate the reasons for having separate test processes and TAP analyzers. -- c
