Matt, This sounds like a good idea to me too.
My ITK valgrind it taking too long. Just some test are just too long. How would I disable these tests on a dashboard script? Also any idea what the change is coverage is? Thanks, Brad On Sep 20, 2013, at 6:40 PM, Matt McCormick <[email protected]> wrote: > I did a visualization on test times [1] and it turns out that less > than 1% of the tests (0.97%) take of half (48.7%) of the test times! > > It would be good to add a "RUNS_LONG" label [2] to these tests so that > developers can run the test suite relatively quickly with > > ctest -LE RUNS_LONG > > and the Visual Studio Debug Nightly builds can exclude them with > > set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG) > > These tests are mostly integration tests that test complicated > algorithms on semi-realistic data. They have value, but leaving them > out in these situations should not do much harm relative to their time > overhead. > > Thanks, > Matt > > [1] http://review.source.kitware.com/#/c/12733/ > [2] http://review.source.kitware.com/#/c/12734/ > > > On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <[email protected]> > wrote: >> In general, we do want to make sure the tests produce "correct" results and >> make sure there are no regressions. By having the Release build run a proper >> set of iterations, the developer can verify the correctness. >> >> That said, using a lower resolution input dataset could probably help most >> the these tests and the release/debug times would both be reduced. >> >> >> >> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <[email protected]> wrote: >>> >>> >>>> I think it is OK to drop the iterations in the Debug tests. We are still >>>> exercising the code. If we >>>> need alternate baselines, we will still catch exceptions. This will also >>>> allow more valgrind tests >>>> to run to completion. >>> >>> >>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast” in >>> the test name... Perhaps we should at the very least consider a rename. 😊 >>> >>> If it’s ok to drop the iterations in the Debug tests, then why not drop >>> the iterations in the Release tests, too? There is something to be said for >>> the “same” code being run for the test in both Debug and Release. If it’s >>> not the same, then I should definitely be able to tell just by glancing at >>> the source code for a test that there are Debug/non-Debug differences... >>> >>> I would say each individual test should be able to run in *seconds*, not >>> minutes or hours. Ideally, less than a second, but I realize that’s overly >>> ambitious with some ITK code. >>> >>> However, for the main purpose I have in mind here, namely reducing the >>> time burden on my volunteer machine, I would be absolutely *ecstatic* if all >>> of the 20 tests I listed in the original email were able to run in under 4 >>> or 5 minutes each. >>> >>> Let me know if I can help out in some additional way (beyond just >>> continuing to submit the build to CDash...) >>> >>> >>> Thanks for the discussion, I appreciate it >>> D >>> >> >> >> >> >> -- >> Unpaid intern in BillsBasement at noware dot com >> >> _______________________________________________ >> Powered by www.kitware.com >> >> Visit other Kitware open-source projects at >> http://www.kitware.com/opensource/opensource.html >> >> Kitware offers ITK Training Courses, for more information visit: >> http://kitware.com/products/protraining.php >> >> Please keep messages on-topic and check the ITK FAQ at: >> http://www.itk.org/Wiki/ITK_FAQ >> >> Follow this link to subscribe/unsubscribe: >> http://www.itk.org/mailman/listinfo/insight-developers >> > _______________________________________________ > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Kitware offers ITK Training Courses, for more information visit: > http://kitware.com/products/protraining.php > > Please keep messages on-topic and check the ITK FAQ at: > http://www.itk.org/Wiki/ITK_FAQ > > Follow this link to subscribe/unsubscribe: > http://www.itk.org/mailman/listinfo/insight-developers _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Kitware offers ITK Training Courses, for more information visit: http://kitware.com/products/protraining.php Please keep messages on-topic and check the ITK FAQ at: http://www.itk.org/Wiki/ITK_FAQ Follow this link to subscribe/unsubscribe: http://www.itk.org/mailman/listinfo/insight-developers
