I have also developed some image-based unit tests to compare MPL output,
and I completely agree that getting something like this into MPL is a
very good idea. As Ted writes, the hard part is defining "close" for
images. Minor changes to, for example, agg pixel alignment, cause the
tests to fail when they have low tolerances. So, such tests are a bit
more interactive than plain old unit tests when something changes, but I
think that's worth the pain.
Prabhu Ramachandran has also done similar things for mayavi2 using VTK's
image comparison (see compare_image_with_saved_image in
https://svn.enthought.com/enthought/browser/Mayavi/trunk/integrationtests/mayavi/common.py
).
I'm attaching the code I use to compare images as a starting point -- it
currently uses scipy to load the images, but this could surely be worked
around.
-Andrew
Drain, Theodore R wrote:
> John,
>
> One thing that would help w/ a rapid delivery/response cycle would be
> more comprehensive tests. They would let other developers try out
> various ideas and see what breaks before you release it.
>
>
>
> We’ve implemented an automated approach where we run an MPL script using
> Agg, save the output image and then compare it against a “good” image
> that someone looked at. We use PIL to do the compare and if it’s close
> (that’s the hard part), then the test passes. If it’s not, someone
> looks at the two images to see if the difference is benign. Something
> similar to this could be done (if you’re not already) for the MPL
> examples to make sure that changes don’t cause plotting problems in
> other areas.
>
>
>
> Having this kind of system is also a great driver for people to expand
> it. For example – we really care about unit processing support
> everywhere. Every once in awhile, we find a change that someone submits
> that breaks unit support. So once of the tasks we‘re going to work on
> next year is to build a set of automated test cases that try and hit
> every plot function with units which can then run on every release. If
> there were a simple to use MPL standard test system like this, other
> people might contribute more tests as a way of insuring that the things
> they care about stay working through various changes.
>
>
>
> It would also be nice to have a test system for unit testing of
> components. A lot of the code that does different transformations,
> symbol and color mapping, etc etc could be unit tested without the need
> for actually drawing anything. If there was a standard location, style,
> and system, people could slowly add to the tests over time. You can
> also consider requiring some level of unit test for newly submitted code
> where ever it’s possible.
>
>
>
> Just some thoughts…
>
> Ted
>
>
>
> *From:* John Hunter [mailto:jdh2...@gmail.com]
> *Sent:* Wednesday, December 10, 2008 8:10 PM
> *To:* Darren Dale
> *Cc:* matplotlib-devel@lists.sourceforge.net
> *Subject:* Re: [matplotlib-devel] requesting permission to remove traits
> and configobj
>
>
>
>
>
> On Wed, Dec 10, 2008 at 9:20 PM, Darren Dale <dsdal...@gmail.com
> <mailto:dsdal...@gmail.com>> wrote:
>
> There has been a report at the bugtracker complaining that matplotlib is
> overwriting an existing installation of configobj. I had a look at the
> code and thought the bug report must be a mistake or windows specific,
> but I just saw similar behavior on my linux system.
>
>
> Ignoring for a minute the question of whether we can/should flush
> configobj/traits, it sounds like the real problem is that setup.cfg is
> not working like we expect it to. And that is something that should be
> fixed if is broken. If mpl is installing configobj or traits even if we
> are telling it not to via setup.cfg, then we have a problem. This is
> worth knowing, since the last mpl release was broken vis-a-vis the
> default backend on win32, which *could* be explained by a broken setup.cfg.
>
> I would like to simply remove configobj and traits from our
> repository. They are only used by the long-neglected experimental
> traited config package, which is only of interest to developers who
> can easily install them as external dependencies. Is it ok to remove
> them? If so, should it be done on all the branches?
>
> How long are we going to continue to maintain the different
> branches? It was so much easier back when we only had to worry about
> the trunk...
>
>
> You can remove them from the trunk. They should remain on the 0.91
> branch as is (with any known bugs fixed and merged) since that is the
> point of the branch (stability for those who cannot upgrade -- in
> principle someone might be depending on the traited config, in practice
> unlikely). As for the 0.98 branch, it is slated for destruction so no
> worries. I share your visceral reaction against branches, but my head
> is starting to override this bodily reaction, as I see the need for them
> in practice. If we carefully document the best practices and
> motivations in the developerr's guide, we can use them advantageously.
>
> We have a lot of people contributing to mpl, and approaching or just
> after release time we need some mechanism for stabilizing the tested
> feature set of the release candidate while allowing other development
> to proceed, and branches are the natural mechanism for that. That we
> are starting to use them is a reflection of the fact that we have many
> more active developers than we ever had before (12 developers
> contributed between 98.3 and 98.4, it used to be just 3 or 4 at a
> time). I wouldn't be advocating branches otherwise, because I am an
> advocate of doing things as simply as possible: "Make everything as
> simple as possible, but not simpler.".
>
> In general, I am in favor of the wildest, wooliest, development process
> we can afford. I would like to have everyone on the trunk, making
> releases as often as possible (nightly if we can), with an attitude of
> "if you break it, just fix it an rerelease it". This model worked fine
> for us for years, and I think it would continue to work if we have a
> hyper-active developer or an automated build bot. In the old days, I
> would release any time I added a new feaure, and if I broke something I
> would have a new release out in hours. I no longer have the time for
> that, and we are lucky to have Charlie buildng OS X and win32 binaties
> and eggs for multiple python versions. When we release broken code,
> Charlie has to go through the entire test/upload/release cycle again,
> building for multiple OSs and python versions while taking care of his
> wife and two babies, so we want to minimize that. At the same time, we
> have lots of developers pushing code into the mainline. We need some
> mechanism of balancing the desire of developers to get new code in and
> the need for the packagers and release manangers to get stable code out.
>
> I think the right balance for mpl before a release is to test the HEAD,
> sign off on it, branch it, let development proceed on the HEAD, and put
> any release critical bugs and fixes into the branch. When the next
> release comes up, delete the old branch, and wash-rinse-repeat. We will
> have in perpetuity one active release branch at a time, which gets
> important bug fixes and nothing else, and in addition (for a while) a
> legacy branch (0.91) which is updated once a month or so. I am happy to
> get input on this.
>
> JDH
>
> No virus found in this incoming message.
> Checked by AVG - http://www.avg.com
> Version: 8.0.176 / Virus Database: 270.9.16/1841 - Release Date:
> 12/10/2008 6:53 PM
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
> SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
> The future of the web can't happen without you. Join us at MIX09 to help
> pave the way to the Next Web now. Learn more and register at
> http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Matplotlib-devel mailing list
> Matplotlib-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
import scipy.misc
import numpy as np
def get_fraction_similar( im1_filename, im2_filename,
pix_diff_threshold=0.1):
"""Get fraction of pixels that are similar
If all pixels are identical, return 1.0. If not, convert to
grayscale (by taking the mean of the color and alpha channels, if
present), count the number of pixels more different than
pix_diff_threshold, and return the fraction of similar pixels.
"""
im1 = scipy.misc.pilutil.imread(im1_filename)
im2 = scipy.misc.pilutil.imread(im2_filename)
if im1.ndim != im2.ndim:
raise ValueError('images have different ndim')
if im1.shape != im2.shape:
raise ValueError('images have different shape')
if np.allclose( im1, im2 ):
# identical -- no more testing needed
return True
# maybe-3D absolute difference image
di = abs(im1.astype(np.float)-im2.astype(np.float) )
# ensure 2D
if di.ndim==3:
# flatten
di2d = np.mean(di,axis=2)
else:
di2d = di
# Count number of pixels more different than pix_diff_threshold.
n_diff = np.sum(di2d > pix_diff_threshold)
# Find fraction of all pixels
n_total = di2d.shape[0]* di2d.shape[1]
fraction_different = n_diff/float(n_total)
fraction_same = 1.0-fraction_different
return fraction_same
def are_images_close( im1_filename, im2_filename,
pix_diff_threshold=0.1,
ok_fraction_threshold=0.99,
):
"""Compare whether two image files are very similar
Calls get_fraction_similar() and returns True if the result is
greater than or equal to ok_fraction_threshold, False otherwise.
"""
fraction_similar = get_fraction_similar( im1_filename, im2_filename,
pix_diff_threshold=pix_diff_threshold)
# Compare with ok_fraction_threshold
return fraction_similar>=ok_fraction_threshold
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
