Hi On Tue, Nov 18, 2014 at 4:52 PM, Matthias Kuhn <matthias.k...@gmx.ch> wrote:
> Hi all, > > You may have noticed that there is a new symbol on the qgis github page > that (hopefully) says build passing. [1] > > The last week I have been busy with fixing tests for integration with > Travis CI. > > Right now, whenever somebody pushes a change to > * a branch in the qgis repository (that affects "master" and > release-x_y branches) > * a pull request for the qgis repository > a service called Travis CI [2] will be activated and compile the > sourcecode and run the test suite there. That has the advantage that we > have a reference platform (ubuntu precise for the moment) where all > tests pass. This defined environment is a big step forward as it makes > test runs comparable and if something goes wrong we know that code is > responsible and not some other variable. > Currently all tests (with a few exceptions which are disabled) are > passing. And it would be excellent if it stays like that! > > What does that mean for developers? > > **Use pull requests** > Whenever you open a pull request first you get the chance to test your > changes without making the nice green symbol on the front page go red. > You can push to these pull requests until the tests go green. For > reviewers this is also nice: you don't need to spend time on pull > requests that don't build or don't pass tests. > > **Write tests** > If you implement a new feature or fix a bug, write a test for it. If > somebody breaks the test, the pull request or the symbol on the github > project page will turn red. This means that it's clear which commit > broke the test. But the test is your feature or your bugfix! > > **Try to avoid rendering tests** > A big portion of fixing has been spent on rendering tests. The problem > with rendering tests is, that a lot of them have slight inconsistencies > across systems. Gradients, anti aliasing, fonts... This all may render > differently without being really a problem. But it produces a (false) > alarm on another system and one needs to take measures. There are such > measures: defining color intolerance, allowed pixel mismatches, known > anomalies, since last week also different reference pictures... But in > the end this means extra work and false alarms once in a while. We do > create a software that has visual products. Therefore rendering tests > are required. But think very about different possibilities before > writing such a test. Maybe you can extract the WKT from a geometry and > compare that against a known WKT string instead? That is less brittle, > faster to test and much easier to maintain than a rendering test! > If you really need to write a rendering test it is likely that it will > fail on travis in the beginning. To make it pass on travis, create a > pull request and wait for it to be tested. The results will be uploaded > to the OTB CDash [3] where you get information what exactly you need to > adjust. How many pixels failed? Visual comparison if adjusting the color > tolerance a bit may help. Is an anomaly ok? Is there a need to add > another reference picture? > > **C++ vs. Python tests** > I made an earlier call for having only either one or the other. That was > based on misinterpretation of test results from my side. Sorry. > My current opinion is: do what you feel comfortable with. If you prefer > to write C++ tests, do so. If you prefer to write Python tests, do so. > If you want both, do so. The important thing is: DO IT! > If you really need advice: go for python tests, they test the same > functionality like C++ functions but also test the python API and > therefore cover a slightly bigger area. > > Which tests are not fixed (on the reference platform)? > > * The atlas tests had interdependencies, that made some tests pass just > because the environment was ok from previous ones. These tests are now > changed to be independent one of another. And there is a PR that should > fix the issues and re-enable the test. [4] > * The symbology tests fail due to problems with sld loading. There is a > pull request that partly reverts a commit that broke things and > re-enables the test. [5] > * The server pal labeling tests produced very different rendering > results. I could imagine that they would be better off if the would also > use the QGIS test font? New reference images could also be introduced. > Dakota cartography's call. > * The server canvas labeling tests crash on exit because of a > not-yet-ended thread. Multithreading issue? Something else? > * Recently the WCS test started to fail. I guess the server it uses to > test does not always respond? We should either fix the server, disable > the test, increase the timeout... > > Where to go from here? > > There are two things I would like to see: > > Many more tests :) > More platforms: Travis can also handle mac. We could ask them to > enable mac testing for us. But we'd first need to fix the tests there. > There is a service called appveyor that runs tests for windows. But we'd > first need to fix the tests there. > > I am very happy that there was such a great interest in the crowdfunding > campaign that made it possible to do that work. THANK YOU to everybody > who helped to get the testing integration to the next level. > > ​Brilliant stuff Matthias! Now we need to agree to 'no new code without tests' as a formal requirement and we will be a long way down the road towards a more stable QGIS. Regards Tim​ > Matthias > > [1] https://github.com/qgis/QGIS > [2] http://travis-ci.org/ > [3] http://dash.orfeo-toolbox.org/index.php?project=QGIS > > _______________________________________________ > Qgis-developer mailing list > Qgis-developer@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/qgis-developer > -- ------------------------------------------------------------------------------------------ Tim Sutton Visit http://kartoza.com to find out about open source: * Desktop GIS programming services * Geospatial web development * GIS Training * Consulting Services Skype: timlinux Irc: timlinux on #qgis at freenode.net Tim is a member of the QGIS Project Steering Committee ------------------------------------------------------------------------------------------- Kartoza is a merger between Linfiniti and Afrispatial
_______________________________________________ Qgis-developer mailing list Qgis-developer@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/qgis-developer