[JIRA] [tap-plugin] (JENKINS-27037) Identifying output files that don't match their plan

[jonathan.rajotte-jul...@efficios.com (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Jonathan Rajotte-Julien commented on  JENKINS-27037 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
  Re: Identifying output files that don't match their plan  
 
 
 
 
 
 
 
 
 
 
Any news on this ? 
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   





-- 
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JIRA] [tap-plugin] (JENKINS-27037) Identifying output files that don't match their plan

[brunodepau...@yahoo.com.br (JIRA)]

Title: Message Title
 
 
 
 
 
 
 
 
 
 
  
 
 Bruno P. Kinoshita commented on  JENKINS-27037 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
  Re: Identifying output files that don't match their plan  
 
 
 
 
 
 
 
 
 
 
Hi Andrew; 
Good points. Let me work on a fix for this. Will post a possible solution in the next days  
 
 
 
 
 
 
 
 
 
 
 
 

 
 Add Comment 
 
 
 
 
 
 
 
 
 
 

 
 
 
 
 
 
 
 
 
 

 This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) 
 
 
 
 
  
 
 
 
 
 
 
 
 
   





-- 
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[JIRA] [tap-plugin] (JENKINS-27037) Identifying output files that don't match their plan

[a...@aps.anl.gov (JIRA)]

Andrew Johnson
 created  JENKINS-27037


Identifying output files that dont match their plan















Issue Type:


Bug



Assignee:


Bruno P. Kinoshita



Components:


tap-plugin



Created:


19/Feb/15 5:32 PM



Description:


The output from the TAP Plugin does not identify which .tap files don't meet their plan, it just marks the whole job as unstable. I have jobs with several thousand tests which have no failures, but where one or more of the test programs have died before they finished running. There are test results missing from the output files from those tests which could easily be identified by the fact that they don't meet their plan, but all I can see from the TAP reports is that the job is unstable, not which of my many test files the problem came from.

When a .tap file includes a plan (and the plugin has been configured to check the plans), any missing test lines should be treated as failures and counted as such in the test reports (e.g. if the plan says "1..7" but there are only 6 "ok" lines below it and no "not ok" lines at all, that should be counted as 6 passing and one failed test).

This is really important IMHO, it's a fundamental issue with the meaning of the TAP output which the plugin is not following. The https://testanything.org/tap-specification.html gives an example with a missing test and shows how the harness should respond to this, see the section TESTS LINES AND THE PLAN/The test line/Test number where the harness reports the missing test #6 as having failed.




Environment:


Jenkins 1.552, tap-plugin 1.18, RHEL 6.6




Project:


Jenkins



Priority:


Major



Reporter:


Andrew Johnson

























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira







-- 
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.