To Arjen and Hazen: I have now enabled the convenient submission of a dashboard (defined as the collection of essentially all results from a run of ctest) to our cdash dartboard server, my.cdash.org, with results displayed at <http://my.cdash.org/index.php?project=PLplot_git>. This step was completely straightforward, i.e, it only required introduction of the 15-line file, CTestConfig.cmake that is actually provided by my.cmake.org when I created the PLplot_git project there. See the commit message for d7192bb for details.
See also the commit messages for 9adf35e and e187a6c which implement opt-in dashboard submission for each ctest that is run by our comprehensive test script and which eliminates an annoying truncation problem that was occurring for my previously submitted dashboards. Will you both please try submitting dashboards to prove this process works well for you on the various platforms accessible to you? The whole submission process is straightforward and should work for any Unix platform (Cygwin, MinGW-w64/MSYS2, Mac OS X, and Linux should all qualify). Either you run scripts/comprehensive_test.sh with the "do_submit_dashboard yes" option or else you can configure PLplot with cmake as normal (using the -DBUILD_TEST=ON option), build the "all" target (as you likely do normally as well), and the only extra step required _after_ the "all" target is built is to execute ctest -D Experimental That command (also used in our comprehensive test script when "--do_submit_dashboard yes" runs all the ctest tests that are enabled by our build system as normal PLUS with that -D option, ctest collects the dashboard of those results and sends it off to my.cash.org as configured by CTestConfig.cmake. And you can then review those results (and my previous ones) at <http://my.cdash.org/index.php?project=PLplot_git>. At first these results seem extremely terse (organized as one dashboard per line), but you soon realize that virtually every field for a given dashboard is a clickable link starting a hierarchy of further clickable links to make complete information on a given ctest run accessible in a very nicely organized way. The only slight concern I have with regard to these dashboard results is the false positive count of 14 warnings for the cmake run. Apparently, my use of "WARNING:" in many of the messages emitted by our build system is triggering this. I think that is likely a ctest bug, and instead the dashboard should report a warning only when there is an official cmake warning message. Anyhow, the number of configuration warnings these dashboard summaries show should be taken with a large grain of salt. In sum, simply remember to configure, build, and then execute ctest -D Experimental whenever there is an interesting git commit you would like to test or whenever you would like to test your own work before a commit. Then visit <http://my.cdash.org/index.php?project=PLplot_git> to look at those results in detail and to compare them with ctest results submitted by others. Now I have implemented this nice new facility for submitting dashboards, I have put together a table that compares this facility with the travis-ci capability recently implemented by Hazen. (N.B. "WLA" means whatever locally available.) Characteristic dashboard travis-ci Nicely formatted? yes no Git server allowed? any limited to GitHub Where tests are run? locally travis-ci server Compilers? WLA Many gcc suite versions, clang, etc. Platforms? WLA 2.5 year-old Linux, Mac OS X The nicely formatted results (e.g., those at <http://my.cdash.org/index.php?project=PLplot_git> of the dashboard approach are a clear advantage for that approach, but I presume the travis-ci developers will get on top of this issue with their raw results at some point. The fact is lots of developers use GitHub so the extra freedom in this regard for the dashboard approach should be considered just a modest advantage for the dashboard approach. The biggest disadvantage of the dashboard approach from the user's perspective is ctest chews up lots of local cpu cycles while the advantage for the travis-ci approach from that same perspective is it chews up lots of cpu cycles on the travis-ci server. The variety of compilers and compiler versions supported by travis-ci is an advantage for that approach from the single-users perspective, but this advantage decreases considerably if you get many users of PLplot reporting dashboards for the wide variety of compilers and compiler versions available to them. The local platforms where this dashboard approach should work include essentially any platform that supports bash and other Unix commands which are needed by our tests. Thus, our ctest results (and presumably dashboard submission as well) are known to work on Cygwin, MinGW-w64/MSYS2, Mac OS X, and Linux and should work on most other free and commercial Unices. Nobody has yet got ctest to work on "bare" Windows (e.g., MSVC) yet, although theoretically it should work if you used nmake and MSVC while putting e.g., the unix tools such as bash on the PATH from say, your MinGW-w64/MSYS2 platform. Anyhow, even if this issue with ctest on "bare" Windows persists, I think there is a clear advantage with platform support here for the dashboard approach. Note, I have been in contact with the travis-ci support staff about that 2.5 year-old Linux limitation (which I consider to be a modest disadvantage now, but it is on the cusp of becoming a major disadvantage for our free software needs since we typically do not support extremely old versions of Linux libraries). From the reply I got it appears the travis-ci developers have recently cleared up a bottleneck in their software which was making it difficult to update their Linux platforms available on their server so it is expected that limitation will be gone reasonably soon although there is no ETA yet. In sum, I think it is fair to say the two methods complement each other with advantages and disavantages for each as summarized above. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ ------------------------------------------------------------------------------ _______________________________________________ Plplot-devel mailing list Plplot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plplot-devel
