jason.mcdon...@nokia.com said: > > Random question of the day: do you happen to have stats about how > > often those insignificant tests actually fail? That should help to > > figure out which ones are actually working, and therefore should not > > be marked as insignificant. > > I'm glad you asked. I spent some time yesterday and today doing that > analysis for the 40-odd insignificant tests that had no Jira task associated. > > I used the data from the last eleven CI runs for each module (times eight > configs for which autotests are executed). That's not hugely statistically > significant, but it's not that easy to pull large numbers of build logs out > of the CI system manually through the web interface. (Any CI folks who can > tell me how to extract/search all the stored logs?)
I'm using this pattern (which has some minor problems not relevant for any test I've looked at so far): from=02238 to=02538 prj=QtBase_master_Integration cfg=win32-msvc2010_Windows_7 test=tst_QLocale trailing_lines=3 for i in $(seq -w $from $to); do echo "$i:" wget --quiet -O - http://testresults.qt-project.org/ci/$prj/build_$i/$cfg/log.txt.gz | zcat | egrep -A $trailing_lines "Testing $test" || echo "did not run" echo "" done 2>&1 | tee results_${test}.txt ... and scanning at least one hundred logs. Having a transparent caching proxy between self and testresults.qt-project.org helps a lot. _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development