> Am 28.09.2017 um 15:45 schrieb Jesse Glick <[email protected]>: > > After the fiasco of Jenkins 2.80 some of us have been discussing ways > to speed up `acceptance-test-harness` to the point where it would be > practical to run all or part of it against core more frequently, > perhaps even in core PR builds.
This is already supported (since 2014): https://github.com/jenkinsci/acceptance-test-harness/blob/master/docs/JUNIT.md#marking-tests-to-be-member-of-the-smoke-test-group In order to verify core PRs just use the smoke tests. You can select the corresponding tests on your own by changing the annotated tests. > > Part of the problem is that the test infrastructure has a ton of > overhead. One suggestion was to use PhantomJS rather than a live > browser like Firefox during typical test runs. I suggested optimizing > the plugin upload mechanism in `LocalController`, perhaps using a mock > local update center URL to avoid network downloads (as well as > simplifying the way `LOCAL_JARS` and similar options work). There are several thing to do to improve the speed of a single test, however, you typically will not get a good regression test below a minute (maybe 30 seconds). Currently we have 600 tests, resulting in a runtime of 10 hours. This sounds ridiculously slow, however if we compare it to the number of plugins it looks different: i.e. we have one test for two of our plugins, or we spend 30 seconds for each plugin. This is a ridiculously few number of tests! I don’t know any software compony that has so few acceptance tests for their components. If you want to spent the same amount of testing with a QA team you will spend weeks! > > Part of the problem is also that there are just a lot of tests, some > of questionable value. > > https://github.com/jenkinsci/acceptance-test-harness/blob/0e3d03474c28c047b365d1f29b621af32eb7918c/src/test/java/plugins/JobDslPluginTest.java#L1537-L1563 > > is an example of why a full ATH run is so ridiculously slow. On one CI > builder this single test case (there are 61 of them in the test suite > for this one plugin) took 2m45s. To test what? That you can run a DSL > script and it produces a `View` with certain matching jobs? This could > run via `JenkinsRule` in a few seconds and be equally effective in > catching regressions. (Probably more so, since it would be part of the > plugin repo and thus run on every plugin PR, catching mistakes in > plugin code. There is a smaller chance that a fundamental change in > core would break this plugin functionality; that is what > `plugin-compat-tester` is for.) > > We need to just delete stuff like this and make sure ATH is just > testing the UI and critical (“smoke test”) functionality. > We should not delete any of these tests. If you do not like them just do not run them. I use these tests (not all, just the tests for my plugins) before I make a plugin release. This helps me a lot to reduce the manual testing time. They take about an hour to run on my machine, but after running them I am sure that no big issues are part of a release. I think this is good invested time. (Manual testing would require a day at least.) > Think of > test harnesses in a pyramid: > You can have as many unit tests as you want, which test individual > methods or small self-contained components. They should run more or > less instantaneously, and behave essentially the same on any platform. > > You should have a bunch of `JenkinsRule` “functional” tests, which are > slower to run (a few seconds) and a little flakier, but which start up > a real Jenkins server and test your plugin’s functionality in a more > realistic way including extensions and background threads and project > builds and settings and everything. These can even start (open-source > Linux) external services using `docker-fixtures` where necessary, > check basic aspects of HTML page rendering including form submissions > with HtmlUnit, run CLI commands, etc. > > Finally you should have a handful of carefully chosen “acceptance” > tests which are harder to develop, very slow to run, and often flaky, > but provide a convincing demonstration that an entire user-level > feature truly works in a realistic environment including > production-style plugin and class loading (perhaps even contacting > outside services) and driven entirely from the same kind of gestures a > user would make in a web browser. > Yes, I see this also in this way. (At least for the numbers, not for the flakiness, a test should never be flaky). > -- > You received this message because you are subscribed to the Google Groups > "Jenkins Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr0UfKN8M1JF6pp7qdoKjo5SSHVwTcv_XVvbsGycK_tjPw%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/67595E33-A2F7-4A7E-ADF4-31C83217A2BC%40gmail.com. For more options, visit https://groups.google.com/d/optout.
signature.asc
Description: Message signed with OpenPGP
