[jira] Issue Comment Edited: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553857 ] sebor edited comment on STDCXX-683 at 12/20/07 9:25 PM: --- I left these out on purpose, but on second thought I agree that XFMAT should be added. I don't think XNOUT makes sense because NOUT is an expected state for regression tests and would be unexpected for any other kind. was (Author: sebor): I left these out on purpose, but on second thought I agree that XFMAT should be added. I don't think XNOUT makes sense because NOUT is an expected state for regression tests and would unexpected for any other kind. > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html, xcodes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553857 ] Martin Sebor commented on STDCXX-683: - I left these out on purpose, but on second thought I agree that XFMAT should be added. I don't think XNOUT makes sense because NOUT is an expected state for regression tests and would unexpected for any other kind. > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html, xcodes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553815 ] Andrew Black commented on STDCXX-683: - Looking at the revised xcodes.html file, I think it will be necessary to add an 'XNOUT' (expected NOUT) and 'XFMAT' (expected FMAT) assertion code. This is because we currently have tests which currently produce NOUT and FMAT status messages, and these messages are expected. Examples include NOUT messages from many regression tests (like 18.limits.stdcxx-436) and FORMAT (FMAT) messages from some driver tests (0.cmdopts, 0.strncmp, and 0.valcmp). > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html, xcodes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553812 ] Martin Sebor commented on STDCXX-683: - Requirements on Markups 1. Location. The markups must be in the form of human-readable and editable text stored in a location that can be easily and intuitively associated with each component (currently, example, locale, or test). One obvious location is the source code for each component itself. There, the markups could take the form of comments that could be easily found by the test harness. Another convenient location is a separate file with the same base name but a different suffix than the component itself. Yet another possibility is storing all markups in a single text file and with the name of the component as the key. The implementation should be such so as to make it easy to switch from one location to the next if it turns out to be convenient. 2. Format. The format of the markups must be easy to read and write and make it possible to easily express precise constraints involving the operating system and its version, the compiler and its version, the library configuration, and the expected status. It must be possible to set more than one constraint for each component, and it must be possible for a single constraint to refer to more than one platform or configuration. One possible format is to use relational operators and boolean logic. For example, to refer to XLC++ 7.0.0.9 on AIX 5.3 and prior, the expression might look something like this: os==AIX && (os_major<5 || os_major==5 && os_minor<=3) && compiler==XLC && compiler_major==7 && compiler_minor==0 && compiler_micro==0 && compiler_patch==9. Another possible format is to adopt a conveniton similar to the GNU cpu-vendor-os triple produced by config.guess (an example of such a triple is i386-redhat-linux or sparc-sun-solaris2.9). In our case, the GNU convention would need to be modified and extended to include the compiler and the library configuration and might look something like this: cpu-vendor-os-compiler-configuration. We could then use shell globbing to implement matching. For example, the following two patterns would have to be used at the same time in order to refer to a 15D configuration of the library build with XLC++ 7.0.0.9 on AIX 5.3 and prior: *-ibm-aix5.[0-3]-xlc7.0.0.9-15D *-ibm-aix[1-4].*-xlc7.0.0.9-15D The leading asterisk indicates no preference for the CPU component. > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html, xcodes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (STDCXX-360) [XLC++ 7.0] ICE compiling 2.smartptr.shared.cpp
[ https://issues.apache.org/jira/browse/STDCXX-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553779 ] Travis Vitek commented on STDCXX-360: - I've tried several different patch levels and I cannot reproduce this problem on VisualAge C++ 7.0.0.0 or later. I have been able to reproduce this with VisualAge C++ 6.0.0.12. > [XLC++ 7.0] ICE compiling 2.smartptr.shared.cpp > --- > > Key: STDCXX-360 > URL: https://issues.apache.org/jira/browse/STDCXX-360 > Project: C++ Standard Library > Issue Type: Bug > Components: Tests >Affects Versions: 4.1.3, 4.2.0 > Environment: IBM XL C/C++ Version: 07.00..0009 >Reporter: Martin Sebor >Assignee: Travis Vitek > Fix For: 4.2.1 > > Attachments: t.cpp.gz, t.cpp.gz.base64 > > > PMR 02337,K78,000 > The attached translation unit (gzipped and encoded in base-64) produced by > preprocessing tests/tr1.util/2.smartptr.shared.cpp) causes an internal > compiler error: > $ xlC -qversion && xlC -q64 -c -g t.cpp > IBM XL C/C++ Enterprise Edition V7.0 > Version: 07.00..0009 > /nfs/packages/mdx/aix/compilers/5.3.0/va70_20070227/root/usr/vacpp/bin/.orig/xlC: > 1501-230 Internal compiler error; please contact your Service Representative > xlCcore_r -c -I/amd/devco/sebor/stdcxx-4.1.3/include/ansi -D_RWSTDDEBUG > -D_RWSTD_USE_CONFIG -I/build/sebor/stdcxx-4.1.3-vacpp-7.0-15D/include > -I/amd/devco/sebor/stdcxx-4.1.3/include > -I/amd/devco/sebor/stdcxx-4.1.3/../rwtest > -I/amd/devco/sebor/stdcxx-4.1.3/../rwtest/include > -I/amd/devco/sebor/stdcxx-4.1.3/tests/include -g -q64 > -qtemplateregistry=2.smartptr.shared.ti > /amd/devco/sebor/stdcxx-4.1.3/tests/tr1.util/2.smartptr.shared.cpp > /nfs/packages/mdx/aix/compilers/5.3.0/va70_20070227/root/usr/vacpp/bin/.orig/xlCcore_r: > 1501-230 Internal compiler error; please contact your Service Representative -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Sebor updated STDCXX-683: Attachment: xcodes.html Attached an updated set of extended status codes to distinguish not only expected failures from ordinary (unexpected) ones, but also unexpected successes from ordinary (expected) ones. > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html, xcodes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Sebor updated STDCXX-683: Attachment: (was: xcodes.html) > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553737 ] Martin Sebor commented on STDCXX-683: - 4) In addition to the requirements listed above, the system itself (as opposed to the user) must indicate when a successful outcome is not expected. I.e., when a component such as a test is expected to fail but succeeds it must be highlighted as such so as to distinguish it from an ordinary success and make it easy to remove the "expected failure" markup. > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (STDCXX-683) implement notion of expected failures in the test suite
[ https://issues.apache.org/jira/browse/STDCXX-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553737 ] sebor edited comment on STDCXX-683 at 12/20/07 10:05 AM: 4. In addition to the requirements listed above, the system itself (as opposed to the user) must indicate when a successful outcome is not expected. I.e., when a component such as a test is expected to fail but succeeds it must be highlighted as such so as to distinguish it from an ordinary success and make it easy to remove the "expected failure" markup. was (Author: sebor): 4) In addition to the requirements listed above, the system itself (as opposed to the user) must indicate when a successful outcome is not expected. I.e., when a component such as a test is expected to fail but succeeds it must be highlighted as such so as to distinguish it from an ordinary success and make it easy to remove the "expected failure" markup. > implement notion of expected failures in the test suite > --- > > Key: STDCXX-683 > URL: https://issues.apache.org/jira/browse/STDCXX-683 > Project: C++ Standard Library > Issue Type: New Feature > Components: Test Driver, Tests >Affects Versions: 4.2.0 >Reporter: Martin Sebor >Assignee: Martin Sebor >Priority: Critical > Attachments: codes.html > > > Tests (or examples) that fail for known reasons that we haven't been able to > deal with should be distinguished from failures that haven't been analyzed > yet. For example, an example program that fails to compile on an older target > platform because of a compiler bug that we can't find a simple/elegant > workaround should be flagged as such in the test results. Similarly, a test > that fails one or more assertions due to compiler or libc bugs on a specific > platform (or a set of platforms) that we are unable to work around should be > reported as such. > This is important in order to reduce the currently fairly large number of > unexpected failures and to be able to make changes without having to worry > about regressions as much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
