> Am 29.09.2017 um 00:45 schrieb Jesse Glick <[email protected]>: > > On Thu, Sep 28, 2017 at 6:31 PM, Ullrich Hafner > <[email protected]> wrote: >> In order to verify core PRs just use the smoke tests. You can select the >> corresponding tests on your own by changing the annotated tests. > > Yes I have seen this, though I think there is no one really keeping > track of which tests are annotated in this way and making sure that > the set is sane. >
Yes, nobody is using or managing this annotation right now. But it would make sense to change this… > The way ATH is structured it is not suitable for running in core PRs I > think: tests could start failing with no relation to the core change > under test. Even if you pinned a specific revision of > `acceptance-test-harness`, you would still be puling some unknown > version of each plugin from the update center. Even if you exclude all > tests with `@WithPlugins` (making running the tests far less useful), > many of them have intrinsic external dependencies that we can expect > to be flaky. > During the four months of our testing course the number of flaky tests was just a handful (<10). We should remove/ignore these as already suggested. > OTOH it makes sense to run a defined set of tests on a regular basis, > having someone designated to watch over the results, and ensuring that > weekly releases with known failures are not published. This would make sense. > >> we have one test for two of our plugins, or we spend 30 seconds for each >> plugin. This is a ridiculously few number of tests! > > If that were the only test coverage, sure. But the vast majority of > tests are functional tests in plugin repositories. ATH contributes > relatively little to our automated test coverage. > >> I use these tests (not all, just the tests for my plugins) before I make a >> plugin release. > > And in fact the tests for the Analysis suite are particularly slow, so > this is a good example of my point. A handful of them test the UI but > most seem that they could have been written more naturally as > functional tests. > Yes, these tests are slow. But converting them to integration tests would not make them much faster. Jenkins rule tests don’t have the same overhead but are also quite slow. (E.g. a pipeline multi-branch tests needs about 30sec on my machine) It would be quite an effort to rewrite these tests just to make them faster. Is is mostly historical that these tests are in ATH, at the creation time the integration test package using the JenkinsRule was not in the state as it is now. I think it makes sense to exclude these tests from a core PR test job, as already mentioned they are only important for a release of my plugins. (Actually a PR to one of my plugins should run these tests!) -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/10922C44-8880-4488-B363-F494FC16767B%40gmail.com. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: Message signed with OpenPGP
