Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Closed #577. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#event-272761787
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-85717259
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
In this PR we'd have to see if tests are retried, and in that case, if the build is still marked as successful. I ran into that too, and was wondering whether one would have to switch all the failed tests to skipped Hmmm How would you do that? In my PR I added a listener that literally *removed* all failed tests that succeeded after being retried. They didn't appear in the TestNG report, and the build succeeded, but Jenkins still marked the build as failed. I'm not sure if that would happen too even if we manually mark the tests as skipped. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59703447
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Hmmm How would you do that? I was planning to change use the ITestResult to change the status, but didn't experiment with Jenkins. I'll add a test here that fails once and then succeeds twice, to see if this actually helps...thanks for suggesting that! --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59736948
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Erm... * http://stackoverflow.com/questions/9539043/success-percentage-not-handled-correctly-by-surefire * https://github.com/apache/maven-surefire/pull/47 * http://jira.codehaus.org/browse/SUREFIRE-654 --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59742410
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Looking at that patch, I doubt there's much we can do using the provided hooks because it seems Surefire simply handles the information incorrectly, whether we then decide to continue or not. I can try with a patched (urg) version of Surefire, just to see what happens. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59742768
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Yes, that would help. If we're trying a new version of Surefire, perhaps we can test my PR too. The difference is that it will only run the tests more than once if they happen to fail. I'll create an account and vote for that issue :) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59750093
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
I'll create an account and vote for that issue Thanks! A bit weird here so far - still can't get the patched version of Surefire to be totally happy. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59834503
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Still can't get the patched version of Surefire to be totally happy. *sigh*: https://jira.codehaus.org/browse/SUREFIRE-654?focusedCommentId=354700page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-354700 --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59871330
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
The RetryAnalyzer approach did not work (see https://github.com/jclouds/jclouds/pull/556). The surefire plugin seems to record the test results on his own and the Jenkins build was marked unstable even if the retried tests suceeded. Let's see if this approach is more Jenkins friendly! --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59642169
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
The RetryAnalyzer approach did not work Ah, I didn't realize you'd been here before! I ran into that too, and was wondering whether one would have to switch all the failed tests to skipped, with some appropriate output message. That was another reason I was tempted to try this approach first ;-) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59661321
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
[jclouds-pull-requests-java-6 #213](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests-java-6/213/) SUCCESS This pull request looks good --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59676492
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
[jclouds-pull-requests #1302](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/1302/) SUCCESS This pull request looks good --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59677716
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
[jclouds » jclouds #1811](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1811/) SUCCESS This pull request looks good [(what's this?)](https://www.cloudbees.com/what-is-buildhive) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59681943
[jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Really only opening a PR for this to gather some thoughts on this approach to dealing with flaky tests. It#39;s easy to configure and makes use of out-of-the-box functionality of TestNG, but _does_ introduce the overhead of running this test three times on each run. An approach an I also considered (similar to what#39;s discussed in [this post](http://stackoverflow.com/questions/7803691/how-to-optimize-testng-and-seleniums-tests)) was to create a custom retry analyzer that would only run the test three times _if the first invocation failed_. Given the relatively small number of consistently flaky tests we have, though, my thought was first to try the out-of-the-box approach, and to go for the custom code if we feel the total overhead imposed by running the flaky tests multiple times on each run gets too big. You can merge this Pull Request by running: git pull https://github.com/jclouds/jclouds add-test-retry Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/577 -- Commit Summary -- * Expecting 2 of 3 or the first flaky test to succeed -- File Changes -- M core/src/test/java/org/jclouds/rest/functions/PresentWhenApiVersionLexicographicallyAtOrAfterSinceApiVersionTest.java (1) -- Patch Links -- https://github.com/jclouds/jclouds/pull/577.patch https://github.com/jclouds/jclouds/pull/577.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
[jclouds-pull-requests-java-6 #209](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests-java-6/209/) SUCCESS This pull request looks good --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59634159
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
[jclouds-pull-requests #1298](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/1298/) SUCCESS This pull request looks good --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59634753
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
[jclouds » jclouds #1806](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1806/) SUCCESS This pull request looks good [(what's this?)](https://www.cloudbees.com/what-is-buildhive) --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-59635377
