On 2016-09-13 10:44+0100 Phil Rosenberg wrote: > On 13 September 2016 at 03:27, Alan W. Irwin <ir...@beluga.phys.uvic.ca> > wrote: >> There is a substantial testing requirement for fill changes. The >> reason for that is we have had years of mostly good user experience >> with the present fill algorithm so I feel it is important that anybody >> that changes the fill algorithm in any further way needs to >> demonstrate with substantial testing that their change does not >> introduce fill regressions. >> >> I have alluded previously to automated fill regression testing (for >> all our standard examples that directly or indirectly use fill and all >> our file devices). The core of the idea is to compare old and updated >> plot file results using imagemagick image differences. Do you agree >> this general approach (supplemented by actually looking at the "fill" >> subset of our examples for each of our interactive devices) is the way >> forward to insure that our further fill changes do not introduce more >> issues than they fix? > > I think this is a good idea in general to avoid all sorts of > regressions.
Good. > Would it be easier to use the svg output and diff as the > svg examples can be built on all systems? Presumably you are aware of this already, but it should be mentioned that one issue with diff results is you cannot reliably compare results from two different floating-point (FP) platforms because FP errors propagate to rounded results on two such platforms in different ways. So it is essential to calculate both the old and revised fill result with exactly the same FP platform (i.e., you cannot store plot results from one FP platform to compare with plot results from other FP platforms). Another issue with using diff even for the same FP platform is any significant change in the fill code is likely to cause real differences, e.g., the effective boundaries of the filled regions will be slightly different. In fact, this is a problem for all fully automated methods I can think of to test for fill regressions; there are going to be large local changes (as boundaries change slightly) in results, but so long as the final result looks OK (i.e., no obvious holes in any fill region and no significantly overfilled regions) then the change does not count as a significant fill regression. As a result of this concern, I am now thinking along the lines of an interactive script (i.e., not completely automated so it will be tiring to use unless we are careful with the selection of what images to compare). At the heart of the script, imagemagick would calculate the normalized difference of the old and modified image results, and then simultanously display old, modified, and difference image results to help evaluate by visual means (as opposed to fully automated means) whether any significant regression has occurred. (Of course, for those cases where the normalized difference image is zero, the script could simply skip the visual comparison to speed up the comparison). I have mentioned imagemagick here because I am reasonably familiar with it, and it is one of the primary collections of software that is used on Unix to manipulate and display images. However, you should also note that imagemagick is available as a binary download on Windows (see <http://www.imagemagick.org/script/binary-releases.php>). Furthermore, even though it appears not to be specifically packaged for Cygwin such packaging has occurred for the MinGW-w64/MSYS2 platform so if the generic windows binary download does not work, there is that other Windows platform alternative to try as well. Alan __________________________ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); the Time Ephemerides project (timeephem.sf.net); PLplot scientific plotting software package (plplot.sf.net); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __________________________ Linux-powered Science __________________________ ------------------------------------------------------------------------------ _______________________________________________ Plplot-devel mailing list Plplot-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plplot-devel
