[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
SCM/JIRA link daemon commented on JENKINS-17781 TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapParser.java src/main/java/org/tap4j/plugin/TapPublisher.java src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31 Log: FIXED JENKINS-17960 FIXED JENKINS-17781 Adding option to validate number of planned tests and number of tests executed This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
SCM/JIRA link daemon commented on JENKINS-17781 TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Code changed in jenkins User: Bruno P. Kinoshita Path: src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d Log: JENKINS-17960 JENKINS-17781 Fixing tests This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Bruno P. Kinoshita resolved JENKINS-17781 as Fixed TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Change By: Bruno P. Kinoshita (26/Jul/13 8:25 PM) Status: InProgress Resolved Resolution: Fixed This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Bruno P. Kinoshita closed JENKINS-17781 as Fixed TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Fixed in 1.13 Change By: Bruno P. Kinoshita (27/Jul/13 1:57 AM) Status: Resolved Closed This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Bruno P. Kinoshita updated JENKINS-17781 TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Change By: Bruno P. Kinoshita (24/Jul/13 12:44 PM) Labels: 1.13 parse-errortap This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Bruno P. Kinoshita started work on JENKINS-17781 TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Change By: Bruno P. Kinoshita (11/Jul/13 6:42 PM) Status: Open InProgress This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Bruno P. Kinoshita commented on JENKINS-17781 TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Successfully reproduced the issue here. I'll write a quick fix for it later. Thanks Steve! This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for
Steve Purkis
created JENKINS-17781
TAP Plugin: fails to report errors for
Issue Type:
Bug
Affects Versions:
current
Assignee:
Bruno P. Kinoshita
Attachments:
Jenkins-TAP-no-parse-errors-bug.jpg
Components:
tap
Created:
29/Apr/13 7:44 PM
Description:
Thanks for your work on the Jenkins TAP Plugin!
In our tests, we've found it ignores this parse error. Given this perl test:
$ cat fail.t
use Test::More tests = 2;
ok('before die');
die "eek!";
ok('after die');
And a Jenkins Job with this shell build step:
cd $JENKINS_HOME
mkdir -p $WORKSPACE/tap-output
set +e
prove -m --archive $WORKSPACE/tap-output/ fail.t
echo "prove exited with: $?"
Plus a Publish TAP step...
Running the job you get this TAP output:
$ cat ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t
1..2
ok 1
eek! at fail.t line 4.
Looks like you planned 2 tests but ran 1.
Looks like your test exited with 255 just after 1.
This should result in a Parse Error (because we planned 2 tests, but ran only 1). TAP::Harness confirms:
$ cp ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t ~/fail.tap
$ prove -v ~/fail.tap
... snip - TAP ...
Failed 1/2 subtests
Test Summary Report
---
fail.tap (Wstat: 0 Tests: 1 Failed: 0)
Parse errors: Bad plan. You planned 2 tests but ran 1.
Files=1, Tests=1, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU)
Result: FAIL
But the Jenkins TAP plugin reports a successful build:
1 tests, 1 ok, 0 not ok, 0 skipped, 0 Bail Out!.
...
No parse errors found.
See screenshot for more details.
I've marked this as critical: as it stands Jenkins TAP will report false positives in these cases. As a worst-case scenario this may result in some users deploying bugs to production.
Environment:
Jenkins: 1.500
TAP Plugin: 1.10
Optional: perl: 5.8.9, TAP::Harness: 3.26, TAP::Harness::Archive: 0.14
Project:
Jenkins
Labels:
tap
parse-error
Priority:
Critical
Reporter:
Steve Purkis
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for
Bruno P. Kinoshita commented on JENKINS-17781 TAP Plugin: fails to report errors for Hi Steve! Thanks for reporting the issue, and for including details. At moment I'm working on other open source projects and private projects. But I'll have a new cycle to work on Jenkins plug-ins within the next days. I'll take a look on this issue and will try to cut a release as soon as I've found a fix for it. In the meantime, feel free to comment on this issue and/or, in case you have time, submit a pull request via GitHub All the best, Bruno This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [tap] (JENKINS-17781) TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan
Steve Purkis updated JENKINS-17781 TAP Plugin: fails to report errors for TAP streams where # tests run doesnt match plan Change By: Steve Purkis (29/Apr/13 7:52 PM) Summary: TAPPlugin:failstoreporterrorsfor TAPstreamswhere#testsrundoesntmatchplan Description: ThanksforyourworkontheJenkinsTAPPlugin!Inourtests,wevefounditignores this parse error errorswhere#testsrundoesnotmatchtheplan .Giventhisperltest:$catfail.tuseTest::Moretests=2;ok(beforedie);dieeek!;ok(afterdie);AndaJenkinsJobwiththisshellbuildstep:cd$JENKINS_HOMEmkdir-p$WORKSPACE/tap-outputset+eprove-m--archive$WORKSPACE/tap-output/fail.techoproveexitedwith:$?PlusaPublishTAPstep...RunningthejobyougetthisTAPoutput:$cat~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t1..2ok1eek!atfail.tline4.#Lookslikeyouplanned2testsbutran1.#Lookslikeyourtestexitedwith255justafter1.ThisshouldresultinaParseError(becauseweplanned2tests,butranonly1).TAP::Harnessconfirms:$cp~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t~/fail.tap$prove-v~/fail.tap...snip-TAP...Failed1/2subtestsTestSummaryReport---fail.tap(Wstat:0Tests:1Failed:0)Parseerrors:Badplan.Youplanned2testsbutran1.Files=1,Tests=1,0wallclocksecs(0.02usr+0.03sys=0.05CPU)Result:FAILButtheJenkinsTAPpluginreportsasuccessfulbuild:1tests,1ok,0notok,0skipped,0BailOut!Noparseerrorsfound.Seescreenshotformoredetails.Ivemarkedthisascritical:asitstandsJenkinsTAPwillreportfalsepositivesinthesecases.Asaworst-casescenariothismayresultinsomeusersdeployingbugstoproduction. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
