Michael Droettboom wrote: > On 09/08/2009 10:24 AM, John Hunter wrote: > >> On Tue, Sep 8, 2009 at 8:54 AM, Michael Droettboom<md...@stsci.edu> wrote: >> >> >>> I've been only skimming the surface of the discussion about the new test >>> framework up until now. >>> >>> Just got around to trying it, and every comparison failed because it was >>> selecting a different font than that used in the baseline images. (My >>> matplotlibrc customizes the fonts). >>> >>> It seems we should probably force "font.family" to "Bitstream Vera Sans" >>> when running the tests. Adding "rcParam['font.family'] = 'Bitstream Vera >>> Sans'" to the "test" function seems to do the trick, but I'll let Andrew >>> make the final call about whether that's the right change. Perhaps we >>> should (as with the documentation build) provide a stock matplotlibrc >>> specifically for testing, since there will be other things like this? Of >>> course, all of these options cause matplotlib.test() to have rcParam >>> side-effects. Probably not worth addressing now, but perhaps worth noting. >>> >>> >> We do have a matplotlibrc file in the "test" dir (the dir that lives >> next to setup.py, not lib/matplotlib/tests. This is where we run the >> buildbot tests from. It might be a good idea to set the font >> explicitly in the test code itself so people can run the tests from >> any dir, but I'll leave it to Andrew to weigh in on that. >> >> > Sure. If we *don't* decide to set it in the code, we should perhaps add > a line suggesting to "run the tests from lib/matplotlib/tests" in the > documentation. An even better solution might be to forcibly load the > matplotlibrc in that directory (even if it's an install directory) when > the tests are run. > While the default test usage should probably set as much as possible to ensure things are identical, we also want to be able to test other code paths, so I think I'll add some kind of kwarg to matplotlib.test() to handle non-testing-default rcParams. I think setting lots of things, including the font, explicitly in the default case is a good idea.
Question for the rcParams experts: Can we save a copy of it so that we can restore its state after matplotlib.test() is done? (It's just a dictionary, right?) >> >> >>> I am also still getting 6 image comparison failures due to hinting >>> differences (I've attached one of the diffs as an example). Since I haven't >>> been following closely, what's the status on that? Should we be seeing >>> these as failures? What type of hinting are the baseline images produced >>> with? >>> >>> >> We ended up deciding to do identical source builds of freetype to make >> sure there were no version differences or freetype configuration >> differences. We are using freetype 2.3.5 with the default >> configuration. We have seen other versions, eg 2.3.7, even in the >> default configuration, give rise to different font renderings, as you >> are seeing. This will make testing hard for plain-ol-users, since it >> is a lot to ask them to install a special version of freetype for >> testing. The alternative, which we discussed before, is to expose the >> unhinted option to the frontend, and do all testing with unhinted >> text. >> >> > I just committed a change to add a "text.hinting" rcParam (which is > currently only followed by the Agg backend, though it might make sense > for Cairo and macosx to also obey it). This param is then forcibly set > to False when the tests are run. > > Doing so, my results are even *less* in agreement with the baseline, but > the real question is whether my results are in agreement with those on > the buildbot machines with this change to forcibly turn hinting off. I > should no pretty quickly when the buildbots start complaining in a few > minutes and I can look at the results ;) > I think we compiled freetype with no hinting as a configuration option, so I don't anticipate a failure. Of course, now I look at the waterfall display, see a bunch of green, think "this looks suspicious" (what does that say about my personality?), click the log of the stdio of the "test" components and see a whole bunch of errors. It seems when I switched over to the matplotlib.test() call for running the tests, I forgot to set the exit code. Let me do that right now. Expect a flood of buildbot errors in the near future... > Hopefully we can find a way for Joe Developer to run these tests without > a custom build of freetype. > Yes, I completely agree. In the matplotlib.testing.image_comparison() decorator, we right now have only a single image comparison algorithm based on RMS error. Perhaps we could try the perceptual difference code you linked to? Also, maybe we could simply turn font rendering off completely for a majority of the tests? Or maybe the tests should be run with and without text drawn, with much lower error tolerances when there's no text? The nice thing about our test infrastructure now is that it's pretty small, lightweight, and flexible. The image comparison stuff is just done in a single decorator function, and the only nose plugin is the "known failure" plugin. We can continue writing tests which I hope will be mostly indepdendent from improving the infrastructure. ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
