Am 04.07.2014 19:39, schrieb John Hewson:
Hi Tilman
Thanks for your thoughts, I think that your concerns are already covered by my
original proposal, I’ll try to explain why and how:
Of course I agree with the need for regression tests, however it isn't easy: besides the problems
of the different JDKs (I use JDK7 Windows 64 bit), there is the problem that some enhancements
create slight changes in rendering that are not errors, i.e. both the "before" and the
"after" files look OK by itself. This has happened when we changed the text rendering
recently, and has happened again when the clipping was improved. The cause are probably slight
changes in color or in boundaries.
If a rendering has changed then the regression test should fail. When a failure
occurs the developer needs to manually inspect the differences (we could
generate a visual diff which highlights what changed to make this easier) and
if ok then they can replace the known-good PNG with the ones just rendered.
Indeed this will be the basic workflow for working with regression tests.
Thats exactly what I do now, I generate a visual diff and I make a
decision whether it is relevant or not. If I think not, then I replace
the PNG.
Copyrights is a problem: I'm testing mostly with JIRA attachments that I've downloaded over the
years. While uploading such files to JIRA might count as fair use, I doubt that this would still be
true if they are included in a distribution. Instead, they should be stored somewhere on Apache
servers where only committers and build software ("Travis", "Jenkins", ...) can
access then. The public PDFs that Maruan mentions don't possibly have all the Problem cases that we
solved before. However I have started working with these files and there are at least 5 recent
issues that deals with them.
The PDFs won’t be in a distribution. They will just happen to be stored in an
SVN repo but not our source code repo, in the same way that the website is
stored in the “cmssite” branch of SVN or indeed, are on JIRA. The law doesn’t
distinguish between JIRA and SVN, both are publicly available via HTTP, so
using SVN will simply be a continuation of what we’re already doing with JIRA.
The crucial factor is that we’re only storing publicly available PDFs, because
we have the right to do so, just like Google’s cache, and like we currently do
with JIRA.
Yes but many PDFs we got aren't really "public". If this svn repo is
only accessible to committers, and if the publicly available build
scripts won't break because of this, then it is OK.
Note that even if something is "publicly available", it may still be
copyrighted. Other risks can be that some people upload PDFs that
include personal data. One really good test PDF was apparently a loan
application. I remember that the user insisted that 1. it was test data,
and 2. that it be removed.
Tilman
Additionally, the PDFs need to be version controlled otherwise we won’t be able
to reliably recreate previous builds, so storing the files on a web server
won’t be practical. Also committers will frequently be updating the renderings
as bugs are fixed and we’ll need to version-control the rendered PNG files for
the same reason. Finally, having committers-only files doesn’t fit well with
the Apache goal of open development and would be unnecessary anyway given that
all the PDFs are to be taken from public sources only.
In summary, I’m proposing that we just keep doing what we’re currently doing
with JIRA but we move it into its own SVN repo along with some pre-rendered
PNGs.
Re preflight: the default mode should be to have the Isartor tests on.
Individuals could still disable them locally, but the central build software
should always use them.
Yes - does anybody know why this isn’t the default?
-- John
