Hi Tilman
Thanks for your thoughts, I think that your concerns are already covered by my
original proposal, I’ll try to explain why and how:
> Of course I agree with the need for regression tests, however it isn't easy:
> besides the problems of the different JDKs (I use JDK7 Windows 64 bit), there
> is the problem that some enhancements create slight changes in rendering that
> are not errors, i.e. both the "before" and the "after" files look OK by
> itself. This has happened when we changed the text rendering recently, and
> has happened again when the clipping was improved. The cause are probably
> slight changes in color or in boundaries.
If a rendering has changed then the regression test should fail. When a failure
occurs the developer needs to manually inspect the differences (we could
generate a visual diff which highlights what changed to make this easier) and
if ok then they can replace the known-good PNG with the ones just rendered.
Indeed this will be the basic workflow for working with regression tests.
> Copyrights is a problem: I'm testing mostly with JIRA attachments that I've
> downloaded over the years. While uploading such files to JIRA might count as
> fair use, I doubt that this would still be true if they are included in a
> distribution. Instead, they should be stored somewhere on Apache servers
> where only committers and build software ("Travis", "Jenkins", ...) can
> access then. The public PDFs that Maruan mentions don't possibly have all the
> Problem cases that we solved before. However I have started working with
> these files and there are at least 5 recent issues that deals with them.
The PDFs won’t be in a distribution. They will just happen to be stored in an
SVN repo but not our source code repo, in the same way that the website is
stored in the “cmssite” branch of SVN or indeed, are on JIRA. The law doesn’t
distinguish between JIRA and SVN, both are publicly available via HTTP, so
using SVN will simply be a continuation of what we’re already doing with JIRA.
The crucial factor is that we’re only storing publicly available PDFs, because
we have the right to do so, just like Google’s cache, and like we currently do
with JIRA.
Additionally, the PDFs need to be version controlled otherwise we won’t be able
to reliably recreate previous builds, so storing the files on a web server
won’t be practical. Also committers will frequently be updating the renderings
as bugs are fixed and we’ll need to version-control the rendered PNG files for
the same reason. Finally, having committers-only files doesn’t fit well with
the Apache goal of open development and would be unnecessary anyway given that
all the PDFs are to be taken from public sources only.
In summary, I’m proposing that we just keep doing what we’re currently doing
with JIRA but we move it into its own SVN repo along with some pre-rendered
PNGs.
> Re preflight: the default mode should be to have the Isartor tests on.
> Individuals could still disable them locally, but the central build software
> should always use them.
Yes - does anybody know why this isn’t the default?
-- John
