Den 12. mars 2012 23:45, skrev ext Rohan McGovern: > Richard Moore said: >> On 12 March 2012 17:56,<kent.han...@nokia.com> wrote: >>> Besides flaky tests, we also have the general/recurring problem of changes >>> going into qtbase that break qtdeclarative (and possibly/likely other >>> modules). While I realize it's time-consuming for everyone to manually >>> build and run the qtdeclarative tests (or entire qt5.git) when you're >>> working on qtbase, please at least take the time to consider how your >>> change might affect other modules. If you're unsure and/or would like >>> someone else's opinion on the possible impact, feel free to send an email >>> to this list, or ask on IRC. Thanks. >> Do you have any idea of how many of these were due to QtDeclarative >> making assumptions that aren't part of the defined API, vs how many >> were changes in expected behaviour? > Here's the commits which were needed:
Thanks for the overview, Rohan. > http://codereview.qt-project.org/18909 - 16e29f3 Remove unneeded > dependencies to QtWidgets and QtOpenGL > -> I don't think this one resolves any test failures. It does; by switching to using QGuiApplication for those tests, we effectively sidestep a change in QApplication font loading behavior due to http://codereview.qt-project.org/#change,18621, which broke a test on Mac. > http://codereview.qt-project.org/19656 - c787809 Mark presumed unstable > test as insignificant. > -> marks tst_qquickpixmapcache as insignificant, doesn't actually > resolve the problem, so the real issue may not yet be understood Yep, it's still not understood. I added a comment to the change about the symptom, but I don't know how to reproduce it. > http://codereview.qt-project.org/19552 - dda130f Fix MouseArea autotest. > http://codereview.qt-project.org/19534 - 6cf36b2 Fix tst_qquicktextedit. > http://codereview.qt-project.org/19427 - cb1ff7a Fix double click > handler in QQuickItem. > -> all of these were failing due to changes in double-click > semantics, apparently a bug fix: > > commit b371f3f943703840d0dfbe30505018bcca06e260 > Author: Laszlo Agocs<laszlo.p.ag...@nokia.com> > Date: Tue Mar 6 16:09:09 2012 +0200 > > Fix double click handling. > Yes, the mouse event handling change was the most serious source of failures. In addition, there was a change in QtNetwork last week that I reverted because it caused qtdeclarative autotests to crash. That one is being reapplied again in http://codereview.qt-project.org/#change,19591, hopefully with the crashes gone. > http://codereview.qt-project.org/19598 - cbb7f8b Skip test that accesses > deleted QML engine > -> apparently the test reads invalid memory, but doesn't actually > crash on most platforms. It might be that the qtbase changes, > due to changing the layout of a few things in memory, caused it > to start crashing on at least one platform. It turned out that valgrind was already complaining about this on platforms where it didn't crash. Would be good if there were a CI that ran the tests through valgrind, maybe once a week or so. In the qtbase/api_changes branch, there was a change in QString::mid() (the handling of a negative position argument) that caused tst_qquicktextinput::getText to fail. It's difficult to know which type of changes can break stuff in other modules, but things like - QtCore/Gui event/timer handling - Gui text/font changes - Network changes (declarative uses networking extensively) - Changing the semantics of input arguments of public functions might be worth validating against dependent modules. By doing so we can save a lot of time we otherwise have to spend on failed CI rounds and tracking down changes that broke other modules. Regarding flaky tests: This is a nightmare right now since it makes _any_ integration so darn unpredictable. The CI takes ~1.5 hours to build qtdeclarative and run tests on all platforms. Everything might look green and dandy on the seven first machines, but then the last one gets to 99% before failing some test. Then we need to investigate whether that failure is due to a new change in e.g. qtbase, or whether it's flaky, apply some fix (e.g. mark the test as insignificant), and do a whole new round of CI. So please, be careful when adding new tests that rely on timing/event loop specifics. Due to the very nature of QtQuick (interaction tests, networking), I guess it has a higher likelyhood of having flaky tests than other modules. Best regards, Kent _______________________________________________ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
