> > * Autotests > > * Non-significant > > * Not compiled > > What is meant by "not compiled"?
Tests excluded from the parent .pro file (qnetworkproxyfactory used to be commented out, this has been fixed in 5.0 but not 4.8) Also use of contains(QT_CONFIG, private_tests) to exclude test cases in release builds Sometimes it's possible to exclude the individual test function(s) that require autotest exports rather than the entire test. > > * Dependency on the test server > > * eg. QNetworkReply has too many tests they should be broken up > > * Some tests should be recognised as possible to fail, unit tests > should not > > fail. CI can be smarter if it knows which is which. > > * Separate executables for each so they can have differing retry > strategies. > > > > Breaking up tests with a long runtime into smaller test executables > helps in several ways ... it allows each subtest to be retried > individually if necessary, and may allow them to be run in parallel as > well. > It's a good idea. > > I saw some chat on IRC about, I think, grouping the tests into > different directories to classify their expected stability. If you > want to go down that route then I think it would be better to introduce > a new hint to the CI system at http://qt- > project.org/wiki/CI_Autotest_Metadata , rather than making it depend on > the directory layout. Yes, that makes sense. The groupings we discussed were: Tests that don't depend on anything external Tests that depend on test server / DNS servers (may need retry due to network glitches) Slow tests (e.g. the connection timeout tests - these could run in parallel) For UI framework tests, there's a similar split between unit tests vs those that rely on sending mouse/keyboard events to widgets. The latter can be disturbed by system notifications taking focus away for example. Jason discussed the issue in the QtTest session too. ________________________________ Subject to local law, communications with Accenture and its affiliates including telephone calls and emails (including content), may be monitored by our systems for the purposes of security and the assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
