+1 on simply deleting flaky tests. As an ATH user myself, I see flaky tests as deeply dangerous for the morale and the trust people have in the tool. At such point that even after the quality had improved dramatically (i.e. lowered flakiness), I have to fight my own first reflex to just ignore any ATH result: "heh, anyway failures are unrelated"...
"Unrelated" failures shouldn't be tolerated indeed. And I tend to think @Ignore is a poor-man's versioning and as the test suite there should only be critical tests, having less tests is probably for the good. @Ulrich about > quality UI test suite for 1500 plugins in an hour." Well I don't know what others think, but for critical usage, I don't think we're aiming for 1500. Now I've no idea what would be the number. How should we base our coverage? Like: I guess that makes no sense to have an ATH for any plugin with less than 5k? 10k installs, more? less? Or what should be the criteria to be at least eligible to enter the testsuite. Or as an intermediate solution, should we create two test suites? Like for instance have two modules inside https://github.com/ jenkinsci/acceptance-test-harness (we could have two repos, but I tend to think having only one with two modules would help having a clearer and more trackable versioning/history when shuffling things around). One would be like *acceptance-test-harness-critical* and we'd *very* aggressive with what would move there. And there would be the module *acceptance-test-harness* (the current one I would say). Any new PR adding a new test should always be against *acceptance-test-harness*, and after a to-be-decided period, if 1) that test is deemed important enough and 2) has *never* failed in the last X weeks/runs and 3) add more criteria here, like test speed, could be *moved to acceptance-test-harness-critical*. My 2 cents 2017-10-03 13:35 GMT+02:00 Andres Rodriguez <[email protected]>: > One thing that could encourage moving certain tests from the ATH to the > JTH would be JENKINS-41827 > <https://issues.jenkins-ci.org/browse/JENKINS-41827>. > > > On Mon, Oct 2, 2017 at 1:52 PM, Oliver Gondža <[email protected]> wrote: > >> A lot of comments and valuable feedback has popped up by the end of the >> week. Let me react here instead of on individual messages. >> >> - No argument on slow test run, high flakiness, fragility. Some of that >> is caused by not enough attention as ATH stands a bit aside the delivery >> pipeline. Bringing it to PR builds of core or plugins would point more >> people to it (hopefully making it better) but it would be annoying at first >> until we make it more mature/stable. The question is if we are willing to >> bite the bullet here. >> >> - With regards to the *kinds* of tests that should be part of ATH, the >> thing is this was never defined so different kinds got in. I tend to agree >> that what can be written against JTH, should. Perhaps it would be >> beneficial to write that down and guard no more such tests are added to >> prevent further bloating. The existing ones can either be deleted, migrated >> or tolerated based on how much effort does it take to maintain them. >> >> - Ad smoke tests: This seems like a good compromise (starting point?) for >> PR integration for several reasons. No question the actual test set should >> be revisited. All (sufficiently distinct) core scenarios + recommended >> plugin set scenarios + some popular scenarios of non-recommended plugins? >> >> - Speaking of deleting flaky tests - I am willing to give this a try as >> the idea of stable build sounds wonderful and almost unheard of in ATH >> land. As a maintainer of downstream plugin update center certified by ATH >> internally, I am less excited. Again, I would like to see some consensus on >> the process. Sometimes it might be enough to check JTH covers same scenario >> or it can be migrated easily. What if it does not and the scenario appears >> fundamental? How to tell? >> >> Please let me know if there are any additional thoughts or I have not >> commented on something. Later this week, I intend to summarize this in a >> doc or two in ATH repo to base our future decisions on (w.r.t. removing >> tests, not accepting them, etc.) and update the thread with PR link. >> >> -- >> oliver >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Jenkins Developers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/jenkinsci-dev/6379b1ca-a85d-9041-1695-e9cd02aa042b%40gmail.com. >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Regards, > > Andres Rodriguez > www.cloudbees.com > > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/jenkinsci-dev/CAM9%3DZ%2B7bxf-Kv68T5F0VPqdB6r18K-aw6Mqhf > nBFrvet_8sD-g%40mail.gmail.com > <https://groups.google.com/d/msgid/jenkinsci-dev/CAM9%3DZ%2B7bxf-Kv68T5F0VPqdB6r18K-aw6MqhfnBFrvet_8sD-g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS78UKCwky-QkCKdmdgA-KUQNgSq8mEuNZH1P9ORsdk_6g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
