[jira] Issue Comment Edited: (STDCXX-683) implement notion of expected failures in the test suite

[Martin Sebor (JIRA)]

[ 
https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553857
 ] 

sebor edited comment on STDCXX-683 at 12/20/07 9:25 PM:
---

I left these out on purpose, but on second thought I agree that XFMAT should be 
added. I don't think XNOUT makes sense because NOUT is an expected state for 
regression tests and would be unexpected for any other kind.

  was (Author: sebor):
I left these out on purpose, but on second thought I agree that XFMAT 
should be added. I don't think XNOUT makes sense because NOUT is an expected 
state for regression tests and would unexpected for any other kind.
  
> implement notion of expected failures in the test suite
> ---
>
> Key: STDCXX-683
> URL: https://issues.apache.org/jira/browse/STDCXX-683
> Project: C++ Standard Library
>  Issue Type: New Feature
>  Components: Test Driver, Tests
>Affects Versions: 4.2.0
>Reporter: Martin Sebor
>Assignee: Martin Sebor
>Priority: Critical
> Attachments: codes.html, xcodes.html
>
>
> Tests (or examples) that fail for known reasons that we haven't been able to 
> deal with should be distinguished from failures that haven't been analyzed 
> yet. For example, an example program that fails to compile on an older target 
> platform because of a compiler bug that we can't find a simple/elegant 
> workaround should be flagged as such in the test results. Similarly, a test 
> that fails one or more assertions due to compiler or libc bugs on a specific 
> platform (or a set of platforms) that we are unable to work around should be 
> reported as such.
> This is important in order to reduce the currently fairly large number of 
> unexpected failures and to be able to make changes without having to worry 
> about regressions as much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Issue Comment Edited: (STDCXX-683) implement notion of expected failures in the test suite

[Martin Sebor (JIRA)]

[ 
https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553737
 ] 

sebor edited comment on STDCXX-683 at 12/20/07 10:05 AM:


4. In addition to the requirements listed above, the system itself (as opposed 
to the user) must indicate when a successful outcome is not expected. I.e., 
when a component such as a test is expected to fail but succeeds it must be 
highlighted as such so as to distinguish it from an ordinary success and make 
it easy to remove the "expected failure" markup.

  was (Author: sebor):
4) In addition to the requirements listed above, the system itself (as 
opposed to the user) must indicate when a successful outcome is not expected. 
I.e., when a component such as a test is expected to fail but succeeds it must 
be highlighted as such so as to distinguish it from an ordinary success and 
make it easy to remove the "expected failure" markup.
  
> implement notion of expected failures in the test suite
> ---
>
> Key: STDCXX-683
> URL: https://issues.apache.org/jira/browse/STDCXX-683
> Project: C++ Standard Library
>  Issue Type: New Feature
>  Components: Test Driver, Tests
>Affects Versions: 4.2.0
>Reporter: Martin Sebor
>Assignee: Martin Sebor
>Priority: Critical
> Attachments: codes.html
>
>
> Tests (or examples) that fail for known reasons that we haven't been able to 
> deal with should be distinguished from failures that haven't been analyzed 
> yet. For example, an example program that fails to compile on an older target 
> platform because of a compiler bug that we can't find a simple/elegant 
> workaround should be flagged as such in the test results. Similarly, a test 
> that fails one or more assertions due to compiler or libc bugs on a specific 
> platform (or a set of platforms) that we are unable to work around should be 
> reported as such.
> This is important in order to reduce the currently fairly large number of 
> unexpected failures and to be able to make changes without having to worry 
> about regressions as much.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.