Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
On Thu, Jan 22, 2015 at 1:56 PM, Darin Adler da...@apple.com wrote: On Jan 22, 2015, at 12:58 PM, Dirk Pranke dpra...@chromium.org wrote: Of course, Chromium was more willing to incur the cost of keeping all of the pixel tests up to date when trivial things changed the output This comment, and much of the discussion, seems to be about pixel tests. But what does this have to do with reference tests? The message has nothing to do with whether the test is a pixel test or a ref test. My reply was intended to explain why we had the hashes didn't match but diff passed message, and the discrepancy around fuzzy matching; on the Chromium ports, if run-webkit-tests had printed this message, it would've served as a warning to the user/developer that there was something weird going on. If you all decide you wanted to keep fuzzy matching enabled (and hence that message would happen regularly and could be safely ignored), I would either delete the message outright or at least make it be a debug message. You could also decide you wanted to do fuzzy matching only on ref tests, or only on pixel tests, and change the logic accordingly, of course. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Abandoned Tests
On Jan 22, 2015, at 14:02, Benjamin Poulain benja...@webkit.org wrote: On 1/21/15 8:48 PM, Brent Fulgham wrote: I’ve been reviewing the various Windows layout tests, and have run across a few tests that seem to be leftovers from the Chromium project. I think all ports currently skip these tests: 1. fast/dom/title-directionality-removeChild.html 2. fast/dom/title-directionality.html They come from http://trac.webkit.org/changeset/84199 That's very much useless. The test was for chromium public API. Let's remove! 3. fast/events/drag-image-filename.html Looks useful. Probably worth fixing if possible. 4. fast/autoresize That's from https://bugs.webkit.org/show_bug.cgi?id=73631 Let's remove. [from the right address so may be it will go to the list] No, I think we should fix these (or reimplement them). We use the FrameView autosizing code! And testing it would be awesome. 5. fast/overflow/scrollbar-restored-and-then-locked.html Useless too. If no one is using these tests, or plans to, I would like to remove them and the disconnected bits of code in DRT/WKTR that were originally meant to drive this code. If I don’t hear anything by this time next week, I’ll remove the code. Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
On Jan 22, 2015, at 11:30 AM, Simon Fraser simon.fra...@apple.com wrote: For example, some of the tests in question render a green box either via CALayers, or by painting, and there’s a slight color difference between the two code paths. What about the test I cited? svg/css/svg-resource-fragment-identifier-img-src.html I don’t think the layer vs. painting explains the difference. Is there any reason to expect that one to have a “close but not identical” result? — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
On Jan 22, 2015, at 8:44 AM, Darin Adler da...@apple.com wrote: I noticed that when running tests I get messages like this: svg/css/svg-resource-fragment-identifier-img-src.html - ref test hashes didn't match but diff passed I understand that we can have hash mismatches in actual checked in images (expected.png files), and there, at least, I believe the fix might be to regenerate the image. But for a ref test I am really unclear on this. And further, I think it’s distracting to get messages like this unless you are specifically working on the testing infrastructure. Why should the average WebKit engineer care about this? Does this reflect a real performance problem when running tests? If so, who would fix it and what would that person do? Alexey, Tim and I looked at this yesterday. This happens when the expected and actual images are very close, but not identical. ImageDiff has some built-in rounding that effectively acts as a small tolerance, so the hashes are different, but ImageDiff incorrectly says the images are the same. For example, some of the tests in question render a green box either via CALayers, or by painting, and there’s a slight color difference between the two code paths. My preference for how to fix this would be to fix ImageDiff to remove its slight built-in tolerance, and then to expose testRunner API to allow a test to set an explicit tolerance. There are many cases where we’d like to use ref tests, but are unable to because of slight, justifiable rendering differences, and having an explicit tolerance would permit the use of ref tests in these cases. Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
22 янв. 2015 г., в 11:30, Simon Fraser simon.fra...@apple.com написал(а): This happens when the expected and actual images are very close, but not identical. ImageDiff has some built-in rounding that effectively acts as a small tolerance, so the hashes are different, but ImageDiff incorrectly says the images are the same. For example, some of the tests in question render a green box either via CALayers, or by painting, and there’s a slight color difference between the two code paths. My preference for how to fix this would be to fix ImageDiff to remove its slight built-in tolerance, and then to expose testRunner API to allow a test to set an explicit tolerance. There are many cases where we’d like to use ref tests, but are unable to because of slight, justifiable rendering differences, and having an explicit tolerance would permit the use of ref tests in these cases. One thing about tolerance is that it is super confusing - are we talking about the number of pixels that are different, or about how different the pixels are? Also, a lot of failures only cause small differences in pixel results. Even a 100x100 box that becomes red instead of green is only a small portion of the 800x600 image, and it's even more the case for tests that check e.g. text rendering. It is not currently known what the root causes are for the tests that say ref test hashes didn't match but diff passed. Given that the differences are very tiny, one guess is that even though compositing and non-compositing code paths are mathematically equivalent, there are different rendering steps taken, and rounding at each step adds up to slight differences. Another theory is that we have actual bugs, such as with color management. If it's just rounding differences, then the right thing to do is probably to silence the console output, keeping behavior the same otherwise. - Alexey ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
[webkit-dev] run-webkit-tests question; hashes when comparing ref test output
I noticed that when running tests I get messages like this: svg/css/svg-resource-fragment-identifier-img-src.html - ref test hashes didn't match but diff passed I understand that we can have hash mismatches in actual checked in images (expected.png files), and there, at least, I believe the fix might be to regenerate the image. But for a ref test I am really unclear on this. And further, I think it’s distracting to get messages like this unless you are specifically working on the testing infrastructure. Why should the average WebKit engineer care about this? Does this reflect a real performance problem when running tests? If so, who would fix it and what would that person do? — Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Abandoned Tests
Oh! Sorry. All of these tests fail. They fail because none of the ports actually implement the features they test: 1. fast/dom/title-directionalty and fast/dom/title-directionality-removeChild.html: These test that title supports a ‘dir’ property. We don’t, so they all fail. 2. fast/events/drag-image-filename.html: These expect DRT/WKTR to support ‘dumpFilenameBeingDragged’, which might be a useful debugging method. 3. fast/autoresize: These expect the system to support something called “enableAutoResizeMode(x, y, width, height)”. I’m not sure exactly what it’s supposed to do. 4. fast/overflow/scrollbar-restored-and-then-locked.html These expect DRT/WKTR to support “setScrollbarPolicy”, which seems to allow you to force a scrollbar (horizontal or vertical) to be on or off. Perhaps this would be a useful thing to have for testing. Thanks, -Brent On Jan 22, 2015, at 6:41 AM, Myles C. Maxfield mmaxfi...@apple.com wrote: That's what I mean. What happens when we actually run them? On Jan 21, 2015, at 9:29 PM, Brent Fulgham bfulg...@apple.com wrote: Hi Myles, These tests don’t fail — they are all skipped on mac, win, gtk, and efl. -Brent On Jan 21, 2015, at 9:24 PM, Myles C. Maxfield mmaxfi...@apple.com wrote: On which platforms do they fail? —Myles On Jan 21, 2015, at 8:48 PM, Brent Fulgham bfulg...@apple.com wrote: I’ve been reviewing the various Windows layout tests, and have run across a few tests that seem to be leftovers from the Chromium project. I think all ports currently skip these tests: 1. fast/dom/title-directionality-removeChild.html 2. fast/dom/title-directionality.html 3. fast/events/drag-image-filename.html 4. fast/autoresize 5. fast/overflow/scrollbar-restored-and-then-locked.html If no one is using these tests, or plans to, I would like to remove them and the disconnected bits of code in DRT/WKTR that were originally meant to drive this code. If I don’t hear anything by this time next week, I’ll remove the code. Thanks, -Brent ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] run-webkit-tests question; hashes when comparing ref test output
22 янв. 2015 г., в 17:57, Darin Adler da...@apple.com написал(а): What about the test I cited? svg/css/svg-resource-fragment-identifier-img-src.html This particular test is buggy - it is a hidpi test, so it dumps results as a 1600x1200 image, but its -expected.html is not hidpi, and is dumped as 800x600, so hashes are obviously different. I now filed https://bugs.webkit.org/show_bug.cgi?id=140815 about this test. When we compare pixels, we draw both images into bitmaps of the same size, so they become similar enough for ImageDiff to consider them identical. Earlier today, Simon and I agreed that we should just silence the error message, because it only tells us about minor color differences that are inevitable when comparing compositing vs. non-compositing. Looks like it tells us about more actionable issues, too. Also, I just found https://bugs.webkit.org/show_bug.cgi?id=69444, and I think that its rationale applies in this case, too. So we should probably keep this error message. I'm not sure whether we should make it a hard error though. - Alexey ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
