Hi, I do not think that there is a way to algorithmically define what an acceptable difference is. Here are a few cases where it's critical to detect small differences:
- color management, e.g. testing different code paths that should match precisely; - finding uninitialized memory use bugs in rendering pipeline, which do cause minor pixel noise; - testing antialiasing and scaling behavior (e.g. that we should revert to high quality scaling after an animation is done); - testing text rendering, where the difference can easily be small, e.g one accent over one letter on a mostly blank test result. Talking from past experience that we had with pixel test tolerance and with retrying failing tests, I believe that leeway in reporting failures quickly causes significant deterioration in infrastructure quality, up to a point where we can't tell what's going on with many tests. - Alexey > 18 нояб. 2015 г., в 4:36, Carlos Alberto Lopez Perez <clo...@igalia.com> > написал(а): > > Hi, > > Some reference tests give a 1-pixel or very few pixel differences [1]. > > I'm not sure if this really indicates a problem in the WebKit code, or > it indicates that we are just too strict not allowing even a very small > percentage in pixel differences for this kind of tests. > > Should we tolerate a few pixel differences for reftests ? > > I have done some tests, and the test in [1] passes for any value of > tolerance >= 0.00001% (with the GTK port). > > I'm inclined to allow a very small value, for example a 0.001% (that > would be 100 times stricter than the tolerance value we use for the > other tests) > > > Regards > ------- > > [1] > > For example, this is happening in the GTK port: > https://build.webkit.org/results/GTK%20Linux%2064-bit%20Release%20%28Tests%29/r192415%20%2812243%29/imported/blink/fast/canvas/canvas-clip-stack-persistence-diffs.html > The diff image normalized (so you can see where is the diff): > http://people.igalia.com/clopez/wkbug/151261/canvas-clip-stack-persistence-diff_normalized.png > > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev