Hi Daniel, As a quick fix it sounds good but doesnt it hide a leak or issue (in test setup or in main code)? Long story short: using a random order can allow to find bugs faster instead of hiding them and discover them randomly adding a new test.
That said, good point to have it configurable with a -D or -P and be able to test quickly this flag. Le 30 janv. 2018 17:33, "Daniel Kulp" <dk...@apache.org> a écrit : > I spent a couple hours this morning trying to figure out why two of the > SQL tests are failing on my machine, but not for Jenkins or for JB. Not > knowing anything about the SQL stuff, it was very hard to debug and it > wouldn’t fail within Eclipse or even if I ran that individual test from the > command line with -Dtest= . Thus, a real pain… > > It turns out, there is an interaction problem between it and a test that > is running before it on my machine, but on Jenkins and JB’s machine, the > tests are run in a different order so the problem doesn’t surface. So > here’s the question: > > Should the surefire configuration specify a “runOrder” so that the tests > would run the same on all of our machines? By default, the runOrder is > “filesystem” so depending on the order that the filesystem returns the test > classes to surefire, the tests would run in different order. It looks > like my APFS Mac returns them in a different order than JB’s Linux. But > that also means if there is a Jenkins test failure or similar, I might not > be able to reproduce it. (Or a Windows person or even a Linux user using > a different fs than Jenkins) For most of the projects I use, we generally > have “<runOrder>alphabetical</runOrder>” to make things completely > predictable. That said, by making things non-deterministic, it can find > issues like this where tests aren’t cleaning themselves up correctly. > Could do a runOrder=hourly to flip back and forth between alphabetical and > reverse-alphabetical. Predictable, but changes to detect issues. > > Thoughts? > > > -- > Daniel Kulp > dk...@apache.org - http://dankulp.com/blog > Talend Community Coder - http://coders.talend.com > >
