Github user nitt10prashant commented on the pull request:
https://github.com/apache/cloudstack/pull/960#issuecomment-150501158
@runseb
"I personally fail to see the benefit of skipped tests versus a failed test"
If i may ,
skip is a type of failure where you know in advance about failure ,
since you know about the failure you just Handle them in order to
differentiate from a general failure(may be product issue).So its all about
categorizing the failure like expected failure and unexpected failure
Its about how you see reports
If you accept skip as known failure then 100% success on BVT ~ 20% of
skipped =80% success with 20% of known failure ,at least not a product issue
which you can say without looking at particular script .
Without skip you have to analyze each and every failure then you may come
up with % of product issue and % of failure due to setup an all
"While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT
even if it includes some tests that are actually not run."
My comment:
1-Not run test cases are the one which are not picked up by runner may be
coz of different tags and those test cases will not appear in your report
2-skipped test case are the one which are picked up by runner but due to
non availability of configurations or some other reasons could not run
completely and they will appear in your report .
3-I want 100% success on BVT without any skip if all the setup config is
available .incase they are not i would prefer to skip them in order to find out
real failure which does not include any failure because of setup/configuration
just by looking at my report .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
