Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 6:42 PM, Maciej Stachowiak wrote: The challenge with reftests is coming up with different markup that's supposed to render exactly the same, but for unrelated reasons. While it's possible, for example, to compare a table layout to absolute positioned boxes, for some things you get down to primitives that can't be tested independently. For example you can test that b, strong and font-weight: bold have the same results, but none of that tells you whether bold in fact works. You could have a regression which caused bold to be completely ignored and would not be able to catch it with a reftest. Apparently you can also write tests that assert that the two renderings are *not* identical. So you could have one test that has bold text and another that has non-bold test, and assert that they are different. See https://developer.mozilla.org/en/Creating_reftest-based_unit_tests. -Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Apr 13, 2010 at 1:49 PM, Simon Fraser simon.fra...@apple.comwrote: dumpAsText also turns off pixel results, which seems like a loss of useful testing. What's the plan there? The hope is that the dumpRenderTree tests will be sufficient testing for pixel results. On Tue, Apr 13, 2010 at 2:55 PM, Dirk Pranke dpra...@chromium.org wrote: I worry that making dumpAsText the default may lead to somewhat counter-intuitive tests being written. To echo what Maciej seemed to say earlier, you can write a rendertree test or a pixel test using vanilla HTML and CSS, but a text-only test almost by definition requires some amount of JS in the test. At which point, inserting a dumpAsText() call is no big deal and clearly indicates the intent. However, having to add DontDumpAsText() as the only script in the test seems a little silly. I don't see why this is silly. Adding the following to an otherwise HTML/CSS-only a test seems totally reasonable to me: script if (window.layoutTestController) window.layoutTestController.dumpRenderTree(); /script If we did this, we'd probably whitelist some whole directories to default to dumpRenderTree though. Specifically, I'm thinking of external tests that we'd like to modify as little as possible (e.g. CSS2.1). It would also allow us to avoid modifying some external tests that we currently have modified to be dumpAsText (e.g. the dom tests). Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Mon, Mar 1, 2010 at 7:53 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2010, at 6:46 PM, Ojan Vafai wrote: On Mon, Mar 1, 2010 at 6:40 PM, Maciej Stachowiak m...@apple.com wrote: On the one hand, it's good to shift the default. On the other hand, many render tree dumping tests do not currently need to run any JavaScript script, while many text-only tests do. I would almost suggest having a per-directory default but that might be too confusing and mysterious. Yeah, I thought about having per-directory defaults too. There are some nice things about that. For example, the editing tests mostly want something in between dumpAsText and a render dump. It would be great if we could default those to dumpAsMarkup ( https://bugs.webkit.org/show_bug.cgi?id=26501). 3 dumpAsMarkup In the webkit-meeting discussion about reducing platform-specific tests, Adam brought up the idea that we could have a file that lists all the tests that default to dumpAsRenderTree, move the default over to dumpAsText and then gradually mark tests appropriately as dumpAsRenderTree and remove them from the list. We might want to leave some directories as defaulting to dumpAsRenderTree as they're tests that we imported from elsewhere (e.g. the CSS2.1 tests). We can annotate the file appropriately listing just the directories. So, eventually, that file will be a few lines. I think moving to just having dumpAsText be the default is better than having per-directory defaults. How does that sound? Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
There are also a fair number of directories that test rendering and layout, although they are usually obvious by their names... many of the subdirectories in fast cannot be converted to pure text tests (clipping, margin collapsing, backgrounds, etc.). dave On Apr 13, 2010, at 1:54 PM, Ojan Vafai wrote: On Mon, Mar 1, 2010 at 7:53 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2010, at 6:46 PM, Ojan Vafai wrote: On Mon, Mar 1, 2010 at 6:40 PM, Maciej Stachowiak m...@apple.com wrote: On the one hand, it's good to shift the default. On the other hand, many render tree dumping tests do not currently need to run any JavaScript script, while many text-only tests do. I would almost suggest having a per-directory default but that might be too confusing and mysterious. Yeah, I thought about having per-directory defaults too. There are some nice things about that. For example, the editing tests mostly want something in between dumpAsText and a render dump. It would be great if we could default those to dumpAsMarkup (https://bugs.webkit.org/show_bug.cgi?id=26501). 3 dumpAsMarkup In the webkit-meeting discussion about reducing platform-specific tests, Adam brought up the idea that we could have a file that lists all the tests that default to dumpAsRenderTree, move the default over to dumpAsText and then gradually mark tests appropriately as dumpAsRenderTree and remove them from the list. We might want to leave some directories as defaulting to dumpAsRenderTree as they're tests that we imported from elsewhere (e.g. the CSS2.1 tests). We can annotate the file appropriately listing just the directories. So, eventually, that file will be a few lines. I think moving to just having dumpAsText be the default is better than having per-directory defaults. How does that sound? Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Apr 13, 2010, at 11:54 AM, Ojan Vafai wrote: On Mon, Mar 1, 2010 at 7:53 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2010, at 6:46 PM, Ojan Vafai wrote: On Mon, Mar 1, 2010 at 6:40 PM, Maciej Stachowiak m...@apple.com wrote: On the one hand, it's good to shift the default. On the other hand, many render tree dumping tests do not currently need to run any JavaScript script, while many text-only tests do. I would almost suggest having a per-directory default but that might be too confusing and mysterious. Yeah, I thought about having per-directory defaults too. There are some nice things about that. For example, the editing tests mostly want something in between dumpAsText and a render dump. It would be great if we could default those to dumpAsMarkup (https://bugs.webkit.org/show_bug.cgi?id=26501). 3 dumpAsMarkup In the webkit-meeting discussion about reducing platform-specific tests, Adam brought up the idea that we could have a file that lists all the tests that default to dumpAsRenderTree, move the default over to dumpAsText and then gradually mark tests appropriately as dumpAsRenderTree and remove them from the list. We might want to leave some directories as defaulting to dumpAsRenderTree as they're tests that we imported from elsewhere (e.g. the CSS2.1 tests). We can annotate the file appropriately listing just the directories. So, eventually, that file will be a few lines. I think moving to just having dumpAsText be the default is better than having per-directory defaults. dumpAsText also turns off pixel results, which seems like a loss of useful testing. What's the plan there? Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Apr 13, 2010 at 11:54 AM, Ojan Vafai o...@chromium.org wrote: On Mon, Mar 1, 2010 at 7:53 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2010, at 6:46 PM, Ojan Vafai wrote: On Mon, Mar 1, 2010 at 6:40 PM, Maciej Stachowiak m...@apple.com wrote: On the one hand, it's good to shift the default. On the other hand, many render tree dumping tests do not currently need to run any JavaScript script, while many text-only tests do. I would almost suggest having a per-directory default but that might be too confusing and mysterious. Yeah, I thought about having per-directory defaults too. There are some nice things about that. For example, the editing tests mostly want something in between dumpAsText and a render dump. It would be great if we could default those to dumpAsMarkup (https://bugs.webkit.org/show_bug.cgi?id=26501). 3 dumpAsMarkup In the webkit-meeting discussion about reducing platform-specific tests, Adam brought up the idea that we could have a file that lists all the tests that default to dumpAsRenderTree, move the default over to dumpAsText and then gradually mark tests appropriately as dumpAsRenderTree and remove them from the list. We might want to leave some directories as defaulting to dumpAsRenderTree as they're tests that we imported from elsewhere (e.g. the CSS2.1 tests). We can annotate the file appropriately listing just the directories. So, eventually, that file will be a few lines. I think moving to just having dumpAsText be the default is better than having per-directory defaults. How does that sound? I worry that making dumpAsText the default may lead to somewhat counter-intuitive tests being written. To echo what Maciej seemed to say earlier, you can write a rendertree test or a pixel test using vanilla HTML and CSS, but a text-only test almost by definition requires some amount of JS in the test. At which point, inserting a dumpAsText() call is no big deal and clearly indicates the intent. However, having to add DontDumpAsText() as the only script in the test seems a little silly. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
Mozilla has been using this technique for years. Perhaps we can pick their brains for some good tricks. Or, dare I say it, share some tests. Hi, hope I'm not crashing the party, and sorry I'm late :-). Let me just say a few things about reftests... Maciej mentioned that a reftest can assert that b and style=font-weight:bold are equivalent, but would not catch a bug that completely disabled font-weight. That is true. However, in our reftest framework you can also assert that two pages render differently, so you can test that font-weight:normal renders differently from font-weight:bold, which would catch a bug like that. If you look in the Mozilla reftests, there are many tests with 'sanity' in the name ensuring that pages that should render differently, do. You could still miss a weird bug like font-weight:bold making the text italic instead. But in practice, we hardly ever encounter bugs that cause some reftests to render incorrectly and still all tests pass. Over time we have learned a number of tricks for writing reftests. Here are a few: -- In general we require tests to match pixel-perfectly. It's always tempting to allow a fudge factor, but tiny differences can indicate significant bugs. -- We have nifty SVG filter tricks to help find those tiny differences, e.g. http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-analyzer.xhtml -- If some pixels of a test are unpredictable or platform-dependent, but not relevant to the test, you can censor them out by placing an opaque element over them in the test and reference. -- In other tests with unpredictable pixel values, e.g. video, we use SVG filters to threshold pixel channel values. -- The Web usually has More Than One Way To Do It. E.g. for CSS gradients, a lot of our reference pages use canvas to render a reference gradient. -- Many Web features have particular cases that are easy to reduce to reftests even when general cases aren't. For example, box-shadow and text-shadow with no blur are trivial to test with reftests. You can create gradients using repeated stops to achieve abrupt transitions and test them against colored boxes. You could use a similar trick to test patterns in SVG text. -- Aggressive subpixel antialiasing is a real pain. For example, wrapping text in an overflow:hidden container isn't always a no-op even if the container is sized to its contents. Using sans-serif fonts helps, using CSS padding helps more. -- For speed, conciseness and overall expediency, you can often pack many variations of a test into a single reftest page, and in practice there's no significant downside to that. One might argue that although reftests are a good way to test what they test, they don't provide enough broad functional testing, e.g. to make sure that blurs look right. I don't have good data to refute or confirm that. However, if we want to, we can get an image test just by making the reference page a PNG ... but I can only see two cases out of over 4000 where we felt we needed to do that. Overall, I'm extremely enthusiastic about reftests! As others mentioned, we'd like to get official CSS and SVG test suites using reftests; they're the best approach I know of for writing robust automated layout and rendering tests across platforms and browsers. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6] ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Mar 9, 2010 at 9:19 AM, Robert O'Callahan rob...@ocallahan.orgwrote: Mozilla has been using this technique for years. Perhaps we can pick their brains for some good tricks. Or, dare I say it, share some tests. Hi, hope I'm not crashing the party, and sorry I'm late :-). Let me just say a few things about reftests... Maciej mentioned that a reftest can assert that b and style=font-weight:bold are equivalent, but would not catch a bug that completely disabled font-weight. That is true. However, in our reftest framework you can also assert that two pages render differently, so you can test that font-weight:normal renders differently from font-weight:bold, which would catch a bug like that. If you look in the Mozilla reftests, there are many tests with 'sanity' in the name ensuring that pages that should render differently, do. You could still miss a weird bug like font-weight:bold making the text italic instead. But in practice, we hardly ever encounter bugs that cause some reftests to render incorrectly and still all tests pass. Over time we have learned a number of tricks for writing reftests. Here are a few: -- In general we require tests to match pixel-perfectly. It's always tempting to allow a fudge factor, but tiny differences can indicate significant bugs. -- We have nifty SVG filter tricks to help find those tiny differences, e.g. http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-analyzer.xhtml -- If some pixels of a test are unpredictable or platform-dependent, but not relevant to the test, you can censor them out by placing an opaque element over them in the test and reference. This and several of the other tips in here seem like they can apply nicely to our existing layout tests as well. -- In other tests with unpredictable pixel values, e.g. video, we use SVG filters to threshold pixel channel values. -- The Web usually has More Than One Way To Do It. E.g. for CSS gradients, a lot of our reference pages use canvas to render a reference gradient. -- Many Web features have particular cases that are easy to reduce to reftests even when general cases aren't. For example, box-shadow and text-shadow with no blur are trivial to test with reftests. You can create gradients using repeated stops to achieve abrupt transitions and test them against colored boxes. You could use a similar trick to test patterns in SVG text. -- Aggressive subpixel antialiasing is a real pain. For example, wrapping text in an overflow:hidden container isn't always a no-op even if the container is sized to its contents. Using sans-serif fonts helps, using CSS padding helps more. -- For speed, conciseness and overall expediency, you can often pack many variations of a test into a single reftest page, and in practice there's no significant downside to that. One might argue that although reftests are a good way to test what they test, they don't provide enough broad functional testing, e.g. to make sure that blurs look right. I don't have good data to refute or confirm that. However, if we want to, we can get an image test just by making the reference page a PNG ... but I can only see two cases out of over 4000 where we felt we needed to do that. Overall, I'm extremely enthusiastic about reftests! As others mentioned, we'd like to get official CSS and SVG test suites using reftests; they're the best approach I know of for writing robust automated layout and rendering tests across platforms and browsers. Rob -- He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all. [Isaiah 53:5-6] ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Feb 23, 2010 at 3:26 PM, Eric Seidel e...@webkit.org wrote: On Tue, Feb 23, 2010 at 2:19 PM, Darin Adler da...@apple.com wrote: But in practice pixel results are often ignored entirely. I think that reftest-style tests if done right could be a great addition. Also, there are zillions of render-tree-dump tests which could easily be converted to plain-text tests. No one has yet written a tool to visualize how many tests we have with platform-specific results, but I would expect many of those platform-specific results could be removed by converting more existing tests to dump as text. The vast majority of plain-text tests don't end up needing platform-specific results. Also, they run faster. In the past there was some resistance to converting older tests to dumpAsText out of fear of losing test coverage. Could we instead try to make it so that the default is dumpAsText? That would go a long way in making new tests be dumpAsText. The difficulty there, of course, is actually doing it since there are so many tests that would need to be modified (i.e. current render dump tests would need something like a dumpRenderTree() call). Implementation difficulties aside, does that sound like a good idea? Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Mar 1, 2010, at 5:49 PM, Ojan Vafai wrote: On Tue, Feb 23, 2010 at 3:26 PM, Eric Seidel e...@webkit.org wrote: On Tue, Feb 23, 2010 at 2:19 PM, Darin Adler da...@apple.com wrote: But in practice pixel results are often ignored entirely. I think that reftest-style tests if done right could be a great addition. Also, there are zillions of render-tree-dump tests which could easily be converted to plain-text tests. No one has yet written a tool to visualize how many tests we have with platform-specific results, but I would expect many of those platform-specific results could be removed by converting more existing tests to dump as text. The vast majority of plain-text tests don't end up needing platform- specific results. Also, they run faster. In the past there was some resistance to converting older tests to dumpAsText out of fear of losing test coverage. Could we instead try to make it so that the default is dumpAsText? That would go a long way in making new tests be dumpAsText. The difficulty there, of course, is actually doing it since there are so many tests that would need to be modified (i.e. current render dump tests would need something like a dumpRenderTree() call). Implementation difficulties aside, does that sound like a good idea? On the one hand, it's good to shift the default. On the other hand, many render tree dumping tests do not currently need to run any JavaScript script, while many text-only tests do. I would almost suggest having a per-directory default but that might be too confusing and mysterious. Question: are text-only tests really faster, in non-pixel mode? I would be surprised if there was a significant different, and it may point to something we can optimize. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Mon, Mar 1, 2010 at 6:40 PM, Maciej Stachowiak m...@apple.com wrote: On Mar 1, 2010, at 5:49 PM, Ojan Vafai wrote: On Tue, Feb 23, 2010 at 3:26 PM, Eric Seidel e...@webkit.org wrote: On Tue, Feb 23, 2010 at 2:19 PM, Darin Adler da...@apple.com wrote: But in practice pixel results are often ignored entirely. I think that reftest-style tests if done right could be a great addition. Also, there are zillions of render-tree-dump tests which could easily be converted to plain-text tests. No one has yet written a tool to visualize how many tests we have with platform-specific results, but I would expect many of those platform-specific results could be removed by converting more existing tests to dump as text. The vast majority of plain-text tests don't end up needing platform-specific results. Also, they run faster. In the past there was some resistance to converting older tests to dumpAsText out of fear of losing test coverage. Could we instead try to make it so that the default is dumpAsText? That would go a long way in making new tests be dumpAsText. The difficulty there, of course, is actually doing it since there are so many tests that would need to be modified (i.e. current render dump tests would need something like a dumpRenderTree() call). Implementation difficulties aside, does that sound like a good idea? On the one hand, it's good to shift the default. On the other hand, many render tree dumping tests do not currently need to run any JavaScript script, while many text-only tests do. I would almost suggest having a per-directory default but that might be too confusing and mysterious. Yeah, I thought about having per-directory defaults too. There are some nice things about that. For example, the editing tests mostly want something in between dumpAsText and a render dump. It would be great if we could default those to dumpAsMarkup (https://bugs.webkit.org/show_bug.cgi?id=26501). But, I agree, it's probably too confusing. Question: are text-only tests really faster, in non-pixel mode? I would be surprised if there was a significant different, and it may point to something we can optimize. I just meant that they are faster assuming you're running pixel tests. Ideally we'd head in the direction of running pixel tests by default one day. :) Ojan ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Feb 23, 2010 at 2:00 PM, Tor Arne Vestbø tor.arne.ves...@nokia.com wrote: Lately we've been playing with the idea of using SVG fonts for the Qt port to get the same set of expected results for qt-mac, qt-linux and qt-win, by injecting new @font-face rules using a user-stylesheet and preventing platform-fonts from being loaded, but this approach/hack has proven to be quite fragile, and we will also miss out on those tests that actually test font loading/rendering. For Chromium on Linux, we try to insulate ourselves from the platform settings by injecting a custom fontconfig file and font list that makes things match Windows behavior. (Matching Windows is important because many sites will hardcode a pixel width on a div then fill it with text and expect the text not to wrap.) This makes our tests sadly kind of font-set and platform-dependent but it at least makes the test behave the same regardless of which system it's run on. See this code for a discussion: http://src.chromium.org/cgi-bin/gitweb.cgi?p=chromium.git;a=blob;f=webkit/tools/test_shell/test_shell_gtk.cc;h=e6673ffc1b6dfe756c5f95e59679c8e857af530d;hb=HEAD#l173 ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On 23/2/10 14:15 , Evan Martin wrote: On Tue, Feb 23, 2010 at 2:00 PM, Tor Arne Vestbø tor.arne.ves...@nokia.com wrote: Lately we've been playing with the idea of using SVG fonts for the Qt port to get the same set of expected results for qt-mac, qt-linux and qt-win, by injecting new @font-face rules using a user-stylesheet and preventing platform-fonts from being loaded, but this approach/hack has proven to be quite fragile, and we will also miss out on those tests that actually test font loading/rendering. For Chromium on Linux, we try to insulate ourselves from the platform settings by injecting a custom fontconfig file and font list that makes things match Windows behavior. (Matching Windows is important because many sites will hardcode a pixel width on a div then fill it with text and expect the text not to wrap.) We do the same thing for the Qt DRT on Linux, but I was hoping to get something that would work for Windows and Mac OS X too, since we don't use FreeType on those platforms, etc. That's where the idea of using SVG fonts spawned, since if we force the Qt DRT to use the raster paint engine those SVG fonts should be rendered the same way regardless of platform. But then we would have the problem of not being able to test actual font loading (without adding lots of exceptions) and the trick would not work for other ports, just the Qt port. That's why I was playing with the idea of using hard-coded metrics for fonts and themes on a global basis in WebKit (if running under the DRT), and letting those tests that actually test font loading/rendering or theming override this using the layoutTestController. Does that sound achievable/desirable at all? :) tor arne ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 5:00 AM, Tor Arne Vestbø wrote: Hey all, A reoccurring problem when trying to maintain layout-test results is differences in font and theme metrics for tests that dump the render tree. Often a test does not actually test font loading/rendering or theming, but has a piece of text or an input element somewhere in the test which causes the metrics in the render tree to be different, and hence the test failing. I think the correct longterm solution to this problem is to use reftests. A reftest consists of two files; the test file, and a reference file that should give the same on-screen rendering. When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. Isn't it our existing pixel test infrastructure? Or do you mean something else? -- Ariya Hidayat http://www.linkedin.com/in/ariyahidayat ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 8:08 AM, Ariya Hidayat wrote: When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. Isn't it our existing pixel test infrastructure? Or do you mean something else? In pixel tests, the expected file is a checked-in image. In addition, we still have a render tree dump; it’s not just pixels. In reference tests, the expected file is HTML; different than the test file, typically simple markup. Both files are rendered and the resulting pixels are compared. And I presume there would be no render tree dump. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On 23/2/10 17:02 , Simon Fraser wrote: On Feb 23, 2010, at 5:00 AM, Tor Arne Vestbø wrote: Hey all, A reoccurring problem when trying to maintain layout-test results is differences in font and theme metrics for tests that dump the render tree. Often a test does not actually test font loading/rendering or theming, but has a piece of text or an input element somewhere in the test which causes the metrics in the render tree to be different, and hence the test failing. I think the correct longterm solution to this problem is to use reftests. A reftest consists of two files; the test file, and a reference file that should give the same on-screen rendering. When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. You mean for example that the reference file of a css-border test that has some header-text describing the test would contain the same header-text but then a image to represent the reference of the css-part? tor arne ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 8:21 AM, Tor Arne Vestbø wrote: On 23/2/10 17:02 , Simon Fraser wrote: On Feb 23, 2010, at 5:00 AM, Tor Arne Vestbø wrote: Hey all, A reoccurring problem when trying to maintain layout-test results is differences in font and theme metrics for tests that dump the render tree. Often a test does not actually test font loading/rendering or theming, but has a piece of text or an input element somewhere in the test which causes the metrics in the render tree to be different, and hence the test failing. I think the correct longterm solution to this problem is to use reftests. A reftest consists of two files; the test file, and a reference file that should give the same on-screen rendering. When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. You mean for example that the reference file of a css-border test that has some header-text describing the test would contain the same header-text but then a image to represent the reference of the css-part? It could be an image, or it could be a configuration of div elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Simon ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On 23/2/10 17:34 , Simon Fraser wrote: On Feb 23, 2010, at 8:21 AM, Tor Arne Vestbø wrote: On 23/2/10 17:02 , Simon Fraser wrote: I think the correct longterm solution to this problem is to use reftests. A reftest consists of two files; the test file, and a reference file that should give the same on-screen rendering. When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. You mean for example that the reference file of a css-border test that has some header-text describing the test would contain the same header-text but then a image to represent the reference of the css-part? It could be an image, or it could be a configuration ofdiv elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Right. Seems to me we have all the pieces to the puzzle? DRT can ouput a PNG with --pixel-tests, we can add foo-reference.html to accompany foo-expected.txt, and run-webkit-tests can be taught to pick up on these reference-files and run DRT with pixel-tests on both and then use ImageDiff to compare them? Then we can gradually migrate tests to use references rather than expected render-trees? Tor Arne ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
Am Dienstag, den 23.02.2010, 08:34 -0800 schrieb Simon Fraser: It could be an image, or it could be a configuration of div elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Simon I like the idea of reftests. But how do we write tests for box-shadows, gradients or patterns on a text (like in SVG)? All basic graphical elements would still need pixel-tests, or do you have another idea? Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 9:58 AM, Dirk Schulze wrote: But how do we write tests for box-shadows, gradients or patterns on a text (like in SVG)? Possibly with canvas. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Feb 23, 2010 at 12:58 PM, Dirk Schulze vb...@gmx.de wrote: Am Dienstag, den 23.02.2010, 08:34 -0800 schrieb Simon Fraser: It could be an image, or it could be a configuration of div elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Simon I like the idea of reftests. But how do we write tests for box-shadows, gradients or patterns on a text (like in SVG)? All basic graphical elements would still need pixel-tests, or do you have another idea? Mozilla has been using this technique for years. Perhaps we can pick their brains for some good tricks. Or, dare I say it, share some tests. -Sam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
Hi, I like this concept of reftests because it seems easier to maintain and scales better on multiple ports than the current form of testing. On the other hand it has to be noted that it would apparently be much work to create reference tests for the current tests which doesn't appear to be a trivial task, furthermore the testing process would be much slower than now becasue, as I experienced it, running the pixel tests is really slow (at least on the Qt port) and we have to generate two pictures per test if we introduce reftests. BTW, does the python based layout testing framework, which was checked in recently, support pixel tests? AFAIK it can run multiple DRT's in parallel and speed up the testing process which wouldn't be bad a bad thing in this case either. Regards, Andras 2010-02-23 19:04 keltezéssel, Sam Weinig írta: On Tue, Feb 23, 2010 at 12:58 PM, Dirk Schulze vb...@gmx.de mailto:vb...@gmx.de wrote: Am Dienstag, den 23.02.2010, 08:34 -0800 schrieb Simon Fraser: It could be an image, or it could be a configuration of div elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Simon I like the idea of reftests. But how do we write tests for box-shadows, gradients or patterns on a text (like in SVG)? All basic graphical elements would still need pixel-tests, or do you have another idea? Mozilla has been using this technique for years. Perhaps we can pick their brains for some good tricks. Or, dare I say it, share some tests. -Sam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Feb 23, 2010 at 11:06 AM, Andras Becsi abe...@inf.u-szeged.huwrote: BTW, does the python based layout testing framework, which was checked in recently, support pixel tests? AFAIK it can run multiple DRT's in parallel and speed up the testing process which wouldn't be bad a bad thing in this case either. Offtopic, but yes it does. It doesn't support everything that run-webkit-test does (yet and I don't know where they are in the work). Regards, Andras ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
I agree. We should organize cross-browser-test-fest and pick each other's brains. I like the idea of introducing ref tests and I think they are teh hot. My only concern would be to the amount effort of converting existing tests to that, so I think we should start small. :DG On Tue, Feb 23, 2010 at 10:04 AM, Sam Weinig sam.wei...@gmail.com wrote: On Tue, Feb 23, 2010 at 12:58 PM, Dirk Schulze vb...@gmx.de wrote: Am Dienstag, den 23.02.2010, 08:34 -0800 schrieb Simon Fraser: It could be an image, or it could be a configuration of div elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Simon I like the idea of reftests. But how do we write tests for box-shadows, gradients or patterns on a text (like in SVG)? All basic graphical elements would still need pixel-tests, or do you have another idea? Mozilla has been using this technique for years. Perhaps we can pick their brains for some good tricks. Or, dare I say it, share some tests. -Sam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
The first step is probably to get the infrastructure in place so that new tests can be written as ref tests when possible. -Darin On Tue, Feb 23, 2010 at 11:24 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: I agree. We should organize cross-browser-test-fest and pick each other's brains. I like the idea of introducing ref tests and I think they are teh hot. My only concern would be to the amount effort of converting existing tests to that, so I think we should start small. :DG On Tue, Feb 23, 2010 at 10:04 AM, Sam Weinig sam.wei...@gmail.com wrote: On Tue, Feb 23, 2010 at 12:58 PM, Dirk Schulze vb...@gmx.de wrote: Am Dienstag, den 23.02.2010, 08:34 -0800 schrieb Simon Fraser: It could be an image, or it could be a configuration of div elements, or a table, or something else that can be configured to look exactly the same as the CSS border property being tested. Simon I like the idea of reftests. But how do we write tests for box-shadows, gradients or patterns on a text (like in SVG)? All basic graphical elements would still need pixel-tests, or do you have another idea? Mozilla has been using this technique for years. Perhaps we can pick their brains for some good tricks. Or, dare I say it, share some tests. -Sam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 11:06 AM, Andras Becsi wrote: I like this concept of reftests because it seems easier to maintain and scales better on multiple ports than the current form of testing. A reftest-style test seems like a good alternative to a pure text test. It can handle testing a lot more things but has a similar level of cross-platform portability. It’s great that reftest-style tests are not as dependent on font metrics as render tree dumps and not as dependent on rendering differences as pixel tests are. But depending on the reftest machinery is implemented these may only be good for simple tests that fit on one screen, a limitation that currently affects only pixel test results, and these kinds of tests can miss breakages that pixel tests or render tree dumps could catch. If we convert too many tests we’ll lose coverage. It’s probably common for both the reference rendering and the test case to both render wrong for the same reason. And in quite a few cases we can detect this type of regression even with just a render tree dump. Today we have two types of tests: 1) Plain text result only. 2) Render tree test plus pixel result. But in practice pixel results are often ignored entirely. I think that reftest-style tests if done right could be a great addition. -- Darin PS: Someone, please, rename DumpRenderTree! ;-) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
Dimitri Glazkov: I agree. We should organize cross-browser-test-fest and pick each other's brains. I believe some Mozilla people are interested in working on a cross browser test suite framework, based in part on reftests. Jonathan Watt and fantasai are people to coordinate with on this, IIRC. -- Cameron McCormack ≝ http://mcc.id.au/ ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Tue, Feb 23, 2010 at 2:19 PM, Darin Adler da...@apple.com wrote: But in practice pixel results are often ignored entirely. I think that reftest-style tests if done right could be a great addition. Pixel tests are run for every build by chromium, and regressions tracked there. :) Also, there are zillions of render-tree-dump tests which could easily be converted to plain-text tests. No one has yet written a tool to visualize how many tests we have with platform-specific results, but I would expect many of those platform-specific results could be removed by converting more existing tests to dump as text. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Increasing the number of cross-platform/port expected results
On Feb 23, 2010, at 8:14 AM, Darin Adler wrote: On Feb 23, 2010, at 8:08 AM, Ariya Hidayat wrote: When the test is run, the browser loads both files, takes snapshots, and does a pixel comparison. Thus font differences between platforms become less of an issue. Isn't it our existing pixel test infrastructure? Or do you mean something else? In pixel tests, the expected file is a checked-in image. In addition, we still have a render tree dump; it’s not just pixels. In reference tests, the expected file is HTML; different than the test file, typically simple markup. Both files are rendered and the resulting pixels are compared. And I presume there would be no render tree dump. The challenge with reftests is coming up with different markup that's supposed to render exactly the same, but for unrelated reasons. While it's possible, for example, to compare a table layout to absolute positioned boxes, for some things you get down to primitives that can't be tested independently. For example you can test that b, strong and font-weight: bold have the same results, but none of that tells you whether bold in fact works. You could have a regression which caused bold to be completely ignored and would not be able to catch it with a reftest. Even in cases where there are truly independent ways to get the same visual result, it can be tricky to design both. That being said, it does do a good job of factoring out font differences. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
