Interesting, it certainly looks pretty similar. -- John
On 14 Jul 2014, at 23:15, Tilman Hausherr <thaush...@t-online.de> wrote: > As yet another proof that IT people always solve things in similar ways, see > this interesting blog post by one of our "competitors": > http://blog.idrsolutions.com/2013/06/save-time-test/ > > Tilman > > Am 04.07.2014 23:05, schrieb Petr Slabý: >> Hi, >> following is a description of what we are doing in our company. >> >> With our software, we run regression tests after each nightly build and >> sometimes it is a tough fight. If there is a regression, it is not so easy >> to find which commit caused it, because there are potentially many between >> the nightly builds. Then, the decision whether the change is wanted and >> expected is in some cases also difficult (this part might be easier with PDF >> where there is the "golden standard" rendering in Acrobat). If the change is >> expected and the new rendering "better" then one has to commit the new >> reference. This means that the files produced on the nightly build machine >> must be available somehow - it is almost impossible to produce them locally >> as the rendering results are slightly different with different versions of >> java and many other reasons. All this has to be done before the next >> regression test is run to avoid that new regressions are hidden by earlier >> ones. Our complete build with all tests runs several hours... >> >> To improve this workflow, we now use the following schema in addition: >> - there is a smaller set of regression tests which runs relatively fast >> - these tests are triggered by each commit in formatting and rendering >> related projects >> - before running the test itself, the modified project(s) are compiled >> locally, w/o publishing the result to maven >> - the reference rendering files are stored in SVN >> - if a test finds a regression, it immediately stores the new result as a >> new reference into SVN. This makes sure that a) the test renderings do not >> get lost and b) that each regression exactly points to the commit that has >> caused it - the one that triggered the test. The failed test creates a new >> issue in JIRA with a pointer to SVN to the before and after rendering and a >> bitmap of the differencies. The issue is then processed. If we find the >> change to be expected then the issue is simply closed, otherwise we take >> actions to fix the problem. The only annoying thing about this scheme is >> that, after commiting the correction, the test runs again and reports a >> regression because it now compares to the faulty version of the rendering. >> >> Best regards, >> Petr. >> >> -----Původní zpráva----- From: John Hewson >> Sent: Friday, July 04, 2014 7:39 PM >> To: dev@pdfbox.apache.org >> Subject: Re: Regression Testing >> >> Hi Tilman >> >> Thanks for your thoughts, I think that your concerns are already covered by >> my original proposal, I’ll try to explain why and how: >> >>> Of course I agree with the need for regression tests, however it isn't >>> easy: besides the problems of the different JDKs (I use JDK7 Windows 64 >>> bit), there is the problem that some enhancements create slight changes in >>> rendering that are not errors, i.e. both the "before" and the "after" files >>> look OK by itself. This has happened when we changed the text rendering >>> recently, and has happened again when the clipping was improved. The cause >>> are probably slight changes in color or in boundaries. >> >> If a rendering has changed then the regression test should fail. When a >> failure occurs the developer needs to manually inspect the differences (we >> could generate a visual diff which highlights what changed to make this >> easier) and if ok then they can replace the known-good PNG with the ones >> just rendered. Indeed this will be the basic workflow for working with >> regression tests. >> >>> Copyrights is a problem: I'm testing mostly with JIRA attachments that I've >>> downloaded over the years. While uploading such files to JIRA might count >>> as fair use, I doubt that this would still be true if they are included in >>> a distribution. Instead, they should be stored somewhere on Apache servers >>> where only committers and build software ("Travis", "Jenkins", ...) can >>> access then. The public PDFs that Maruan mentions don't possibly have all >>> the Problem cases that we solved before. However I have started working >>> with these files and there are at least 5 recent issues that deals with >>> them. >> >> The PDFs won’t be in a distribution. They will just happen to be stored in >> an SVN repo but not our source code repo, in the same way that the website >> is stored in the “cmssite” branch of SVN or indeed, are on JIRA. The law >> doesn’t distinguish between JIRA and SVN, both are publicly available via >> HTTP, so using SVN will simply be a continuation of what we’re already doing >> with JIRA. >> >> The crucial factor is that we’re only storing publicly available PDFs, >> because we have the right to do so, just like Google’s cache, and like we >> currently do with JIRA. >> >> Additionally, the PDFs need to be version controlled otherwise we won’t be >> able to reliably recreate previous builds, so storing the files on a web >> server won’t be practical. Also committers will frequently be updating the >> renderings as bugs are fixed and we’ll need to version-control the rendered >> PNG files for the same reason. Finally, having committers-only files doesn’t >> fit well with the Apache goal of open development and would be unnecessary >> anyway given that all the PDFs are to be taken from public sources only. >> >> In summary, I’m proposing that we just keep doing what we’re currently doing >> with JIRA but we move it into its own SVN repo along with some pre-rendered >> PNGs. >> >>> Re preflight: the default mode should be to have the Isartor tests on. >>> Individuals could still disable them locally, but the central build >>> software should always use them. >> >> Yes - does anybody know why this isn’t the default? >> >> -- John >
