Re: [webkit-dev] Baselines ignoring metrics

2012-09-19 Thread Dirk Pranke
On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen k...@carewolf.com wrote: On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here. If we have tests that don't depend on checking the metrics, can they just

Re: [webkit-dev] Baselines ignoring metrics

2012-09-19 Thread Dirk Pranke
On Wed, Sep 19, 2012 at 2:00 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Sep 19, 2012 at 1:54 PM, Allan Sandfeld Jensen k...@carewolf.com wrote: On Wednesday 19 September 2012, Dirk Pranke wrote: After some limited amount of thought, I'm inclined to agree with Ryosuke and Ossy here

Re: [webkit-dev] the new TestExpectations syntax is landing soon

2012-09-19 Thread Dirk Pranke
for the inconvenience. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates

[webkit-dev] the TestExpectations syntax has changed

2012-09-19 Thread Dirk Pranke
Assuming my changes stick, all of the TestExpectations files in the repo have been converted to the new syntax. The old syntax is still supported as well, but please don't use it :). http://trac.webkit.org/wiki/TestExpectations At the moment I'm chasing down some minor issues but I haven't

Re: [webkit-dev] the new TestExpectations syntax is landing soon

2012-09-14 Thread Dirk Pranke
On Fri, Sep 14, 2012 at 9:47 AM, Sergio Villar Senin svil...@igalia.com wrote: En 14/09/12 18:34, Sergio Villar Senin escribiu: En 13/09/12 01:29, Dirk Pranke escribiu: Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days

Re: [webkit-dev] Removing the prefix from webkitPostMessage

2012-09-13 Thread Dirk Pranke
On Wed, Sep 12, 2012 at 11:00 PM, Maciej Stachowiak m...@apple.com wrote: On Sep 12, 2012, at 10:36 PM, Ojan Vafai o...@chromium.org wrote: On Wed, Sep 12, 2012 at 6:40 PM, Maciej Stachowiak m...@apple.com wrote: - Is this approach substantially less time and effort than adding a

Re: [webkit-dev] Should we support multiple bug URLs in TestExpectations?

2012-09-13 Thread Dirk Pranke
On Thu, Sep 13, 2012 at 12:32 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi, This discussion came out of https://bugs.webkit.org/show_bug.cgi?id=96569. Should we allow URLs other than webkit.org/b/ to be used in TestExpectations? I vote for yes, and in fact, adding new URLs should be easy.

[webkit-dev] the new TestExpectations syntax is landing soon

2012-09-12 Thread Dirk Pranke
Hi all, The new format of the much-debated TestExpectations syntax will be landing soon (hopefully in the next couple days). For those of who have forgotten / repressed the earlier debates, the new syntax looks something like: webkit.org/b/12345 [ Mac Vista] fast/html/keygen.html [

Re: [webkit-dev] the new TestExpectations syntax is landing soon

2012-09-12 Thread Dirk Pranke
Also, for anyone wondering, the check in failing test baselines feature will be landing shortly after this sticks. I'll send out another note with details on that when we get there. -- Dirk On Wed, Sep 12, 2012 at 4:29 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, The new format

Re: [webkit-dev] Subpixel layout - requesting help for big rebaseline

2012-08-22 Thread Dirk Pranke
The Chromium canaries now exit after 5000 failures or 1000 crashes/timeouts. Ideally we'd hold off on this change until we can get some sort of a fix or workaround to https://bugs.webkit.org/show_bug.cgi?id=94665 though (and I'm working on this today), or life might be annoyingly painful for us.

Re: [webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling failing layout tests and TestExpectations

2012-08-22 Thread Dirk Pranke
On Tue, Aug 21, 2012 at 4:16 PM, Maciej Stachowiak m...@apple.com wrote: On Aug 21, 2012, at 3:23 PM, Ojan Vafai o...@chromium.org wrote: On Mon, Aug 20, 2012 at 6:03 PM, Maciej Stachowiak m...@apple.com wrote: Here's how I imagine the workflow when a sheriff or just innocent bystander

Re: [webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling failing layout tests and TestExpectations

2012-08-20 Thread Dirk Pranke
On Sat, Aug 18, 2012 at 8:31 PM, Filip Pizlo fpi...@apple.com wrote: On Aug 18, 2012, at 5:55 PM, Maciej Stachowiak m...@apple.com wrote: On Aug 18, 2012, at 5:11 PM, Filip Pizlo fpi...@apple.com wrote: Maybe at this point we can agree to let Dirk land some variant of this with whatever

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-17 Thread Dirk Pranke
On Fri, Aug 17, 2012 at 8:07 AM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Aug 16, 2012 at 6:11 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, Aug 16, 2012 at 5:41 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Aug 16, 2012 at 5:18 PM, Dirk Pranke dpra...@chromium.org wrote

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-17 Thread Dirk Pranke
On Fri, Aug 17, 2012 at 11:29 AM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Aug 17, 2012 at 11:06 AM, Dirk Pranke dpra...@chromium.org wrote: On the other hand, the pixel test output that's correct to one expert may not be correct to another expert. For example, I might think that one

Re: [webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling failing layout tests and TestExpectations

2012-08-17 Thread Dirk Pranke
On Fri, Aug 17, 2012 at 5:01 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Aug 17, 2012 at 4:55 PM, Ojan Vafai o...@chromium.org wrote: Asserting a test case is 100% correct is nearly impossible for a large percentage of tests. The main advantage it gives us is the ability to have

Re: [webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling failing layout tests and TestExpectations

2012-08-17 Thread Dirk Pranke
All non-flaky failures, yes. Flaky tests would still require entries in the TestExpectations files at this time; discussion of how to treat them is a separate topic. -- Dirk On Fri, Aug 17, 2012 at 5:35 PM, Filip Pizlo fpi...@apple.com wrote: +1, contingent upon the following: are we agreeing

Re: [webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling failing layout tests and TestExpectations

2012-08-17 Thread Dirk Pranke
On Fri, Aug 17, 2012 at 5:43 PM, Filip Pizlo fpi...@apple.com wrote: Then I am on board. We still do need to revisit the handling of flaky tests. The current approach is an absolute disaster. (I normally love exaggerating, but in this case, I feel no satisfaction in doing so because it is

Re: [webkit-dev] A simpler proposal for handling failing tests WAS: A proposal for handling failing layout tests and TestExpectations

2012-08-17 Thread Dirk Pranke
I'm not sure if I like this idea or not. A couple of observations/questions ... 1) I wouldn't want to call it '-correct' unless we were sure it was correct; '-previous' is better in that regard 2) the issue with keeping a '-correct' in the tree is that it's quite possible for a previous correct

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-16 Thread Dirk Pranke
On Wed, Aug 15, 2012 at 5:19 PM, Ryosuke Niwa rn...@webkit.org wrote: I have a concern that a lot of people wouldn't know what the correct output is for a given test. For a lot of pixel tests, deciding whether a given output is correct or not is really hard. e.g. some seemingly insignificant

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-16 Thread Dirk Pranke
On Wed, Aug 15, 2012 at 6:02 PM, Filip Pizlo fpi...@apple.com wrote: 2) Possibility of the sheriff getting it wrong. (2) concerns me most. We're talking about using filenames to serve as a kind of unchecked comment. We already know that comments are usually bad because there is no

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-16 Thread Dirk Pranke
On Thu, Aug 16, 2012 at 5:41 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Aug 16, 2012 at 5:18 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, Aug 16, 2012 at 3:50 PM, Stephen Chenney schen...@chromium.org wrote: I agree with the priorities above, at least. I also agree

[webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-15 Thread Dirk Pranke
Hi all, As many of you know, we normally treat the -expected files as regression tests rather than correctness tests; they are intended to capture the current behavior of the tree. As such, they historically have not distinguished between a correct failure and an incorrect failure. The chromium

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-15 Thread Dirk Pranke
On Wed, Aug 15, 2012 at 12:27 PM, Filip Pizlo fpi...@apple.com wrote: This sounds like it's adding even more complication to an already complicated system. In some ways, yes. In other ways, perhaps it will allow us to simplify things; e.g., if we are checking in failing tests, there is much

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-15 Thread Dirk Pranke
. The solution I've described is the least intrusive mechanism we can try that I've yet come up with. -- Dirk On Wed, Aug 15, 2012 at 12:22 PM, Dirk Pranke dpra...@chromium.org wrote: Hi all, As many of you know, we normally treat the -expected files as regression tests rather than correctness tests

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-15 Thread Dirk Pranke
On Wed, Aug 15, 2012 at 1:36 PM, Michael Saboff msab...@apple.com wrote: It seems to me that there are two issues here. One is Chromium specific about process conformity. It seems to me that should stay a Chromium issue without making the mechanism more complex for all ports. The other

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-15 Thread Dirk Pranke
On Wed, Aug 15, 2012 at 3:06 PM, Filip Pizlo fpi...@apple.com wrote: Apparently I was somewhat unclear. Let me restate. We have the following mechanisms available when a test fails: 1) Check in a new -expected.* file. 2) Modify the test. 3) Modify a TestExpectations file. 4) Add the

Re: [webkit-dev] A proposal for handling failing layout tests and TestExpectations

2012-08-15 Thread Dirk Pranke
On Wed, Aug 15, 2012 at 5:00 PM, Filip Pizlo fpi...@apple.com wrote: On Aug 15, 2012, at 4:02 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Aug 15, 2012 at 3:06 PM, Filip Pizlo fpi...@apple.com wrote: Apparently I was somewhat unclear. Let me restate. We have the following

Re: [webkit-dev] Best way to skip whole directories of layout tests

2012-08-09 Thread Dirk Pranke
On Thu, Aug 9, 2012 at 3:40 PM, Xianzhu Wang wangxian...@chromium.org wrote: Hi, I want to skip several whole directories for chromium-android because of the related features are not available. For example, plugin/. 1. Add the following line in platform/chromium/TestExpectations: WONTFIX

Re: [webkit-dev] DRT/WTR should clear the cache at the beginning of each test?

2012-08-08 Thread Dirk Pranke
On Wed, Aug 8, 2012 at 10:47 AM, Ojan Vafai o...@chromium.org wrote: See https://bugs.webkit.org/show_bug.cgi?id=93195. media/W3C/video/networkState/networkState_during_progress.html and media/video-poster-blocked-by-willsendrequest.html are flaky on all platforms because they behave

Re: [webkit-dev] impending changes to how new-run-webkit-tests prints/logs stuff

2012-08-01 Thread Dirk Pranke
For anyone bikeshedding along at home, my current plan of record (and changes to what has been written earlier in the thread) ... 1) '--verbose --verbose' to increase verbosity is right out 2) we can't use '--debug' to get debug-level output, since that is already used to run the debug build 3)

Re: [webkit-dev] impending changes to how new-run-webkit-tests prints/logs stuff

2012-08-01 Thread Dirk Pranke
On Wed, Aug 1, 2012 at 12:48 PM, Dirk Pranke dpra...@chromium.org wrote: For anyone bikeshedding along at home, my current plan of record (and changes to what has been written earlier in the thread) ... 1) '--verbose --verbose' to increase verbosity is right out 2) we can't use '--debug

[webkit-dev] impending changes to how new-run-webkit-tests prints/logs stuff

2012-07-31 Thread Dirk Pranke
Hi all, I'm finally getting around to cleaning up the byzantine mass of options in new-run-webkit-tests that controls what gets printed to stderr and stdout during a test run. The patch is posted in https://bugs.webkit.org/show_bug.cgi?id=92432. To quote the changelog: [All of the --print X,Y,Z

Re: [webkit-dev] impending changes to how new-run-webkit-tests prints/logs stuff

2012-07-31 Thread Dirk Pranke
On Tue, Jul 31, 2012 at 6:36 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Jul 31, 2012 at 6:29 PM, Dirk Pranke dpra...@chromium.org wrote: I'm finally getting around to cleaning up the byzantine mass of options in new-run-webkit-tests that controls what gets printed to stderr and stdout

Re: [webkit-dev] impending changes to how new-run-webkit-tests prints/logs stuff

2012-07-31 Thread Dirk Pranke
On Tue, Jul 31, 2012 at 7:02 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Jul 31, 2012 at 6:59 PM, Dirk Pranke dpra...@chromium.org wrote: On Tue, Jul 31, 2012 at 6:36 PM, Ryosuke Niwa rn...@webkit.org wrote: On Tue, Jul 31, 2012 at 6:29 PM, Dirk Pranke dpra...@chromium.org wrote

Re: [webkit-dev] Removing --pixel-tests from DRT/WTR

2012-07-26 Thread Dirk Pranke
On Thu, Jul 26, 2012 at 11:00 AM, Balazs Kelemen kbal...@webkit.org wrote: Hi webkittens! I am going to upload a patch to https://bugs.webkit.org/show_bug.cgi?id=92398 that will remove the --pixel-tests option from test drivers. Don't worry, I don't want to kill pixel testing, I want to

Re: [webkit-dev] Comments in the code (Was Please include function-level comments in change log entries)

2012-07-12 Thread Dirk Pranke
On Thu, Jul 12, 2012 at 10:53 AM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Jul 12, 2012 at 10:43 AM, Stephen Chenney schen...@chromium.org wrote: As several people have shown, it is quite easy to come up with a formula that shows the cost of maintaining comments is much lower than the

Re: [webkit-dev] PSA: rebaseline tooling

2012-07-12 Thread Dirk Pranke
At the top of the garden-o-matic page there is a line like Latest revision processed by every bot: 122499 (trunk is at 122524). I think that does what you want? On Thu, Jul 12, 2012 at 3:56 PM, Peter Kasting pkast...@google.com wrote: On Thu, Jul 12, 2012 at 3:17 PM, Ojan Vafai o...@chromium.org

[webkit-dev] is DNS for webkit.org down?

2012-07-01 Thread Dirk Pranke
Hi all, It seems like DNS for webkit.org is down ... I can still get to build.webkit.org, but everything else is timing out? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] is DNS for webkit.org down?

2012-07-01 Thread Dirk Pranke
Seems fine now. Leap second? -- Dirk On Sun, Jul 1, 2012 at 7:34 AM, William Siegrist wsiegr...@apple.com wrote: There was a network issue that has since been resolved. If you're still having trouble, please let me know. -Bill On Jun 30, 2012, at 8:17 PM, Dirk Pranke wrote: Hi all

Re: [webkit-dev] Time out issue (30s) of WebKit layout test [Mac OS]

2012-06-29 Thread Dirk Pranke
On Fri, Jun 29, 2012 at 9:00 AM, Balazs Kelemen kbal...@webkit.org wrote: I think this is just the default, WebKitTestRunner has a --timeout that should control this if given. If that's not the case than it seems like a bug for me. On the other hand, I don't think run-webkit-tests supports

[webkit-dev] new bugzilla keyword: 'NRWT'

2012-06-19 Thread Dirk Pranke
Hi all, A new keyword 'NRWT' for new-run-webkit-tests-related issues has been added to bugzilla (thanks Eric!). I've updated all of the open bugs I have found that are NRWT-related, and will try to keep things up to date in the future, but if you felt like adding the keywords when filing new bugs

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-14 Thread Dirk Pranke
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org wrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote: Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? IMAGE : allow image failure;

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-14 Thread Dirk Pranke
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth aba...@webkit.org wrote: I too would like to see us remove TEXT+IMAGE.  It's really confusing to non-experts, and it doesn't scale as we introduce new kinds of failures (like Audio).  Do we really need TEXT+IMAGE+AUDIO, TEXT+AUDIO, and IMAGE+AUDIO?

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-14 Thread Dirk Pranke
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting pkast...@chromium.org wrote: On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger epo...@chromium.org wrote:

Re: [webkit-dev] IMAGE+TEXT WAS: TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-14 Thread Dirk Pranke
On Thu, Jun 14, 2012 at 10:37 PM, Ojan Vafai o...@chromium.org wrote: On Thu, Jun 14, 2012 at 9:20 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 14, 2012, at 9:06 PM, Adam Barth aba...@webkit.org wrote: On Thu, Jun 14, 2012 at 9:02 PM, Ojan Vafai o...@chromium.org wrote: Seems

Re: [webkit-dev] are fuzzer tests appropriate layout tests?

2012-06-13 Thread Dirk Pranke
On Wed, Jun 13, 2012 at 12:05 PM, Darin Adler da...@apple.com wrote: On Jun 12, 2012, at 5:17 PM, Ojan Vafai o...@chromium.org wrote: It's great to use a fuzzer in order to find cases where we're broken and then make reduced layout tests from those. Generally we do require a test each time

Re: [webkit-dev] are fuzzer tests appropriate layout tests?

2012-06-13 Thread Dirk Pranke
recall a time when the test saved them from making a broken commit, or when it helped a bot watcher identify a genuinely broken changeset, then we should probably get rid of it. -F On Jun 13, 2012, at 11:58 AM, Dirk Pranke wrote: I agree that the fuzzer should be used to create dedicated

[webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
Hi all, Because I have infinite patience for bikeshedding, I thought I would send out Yet Another note on the proposed changes to the expectation syntax. Based on the last thread, I'm planning to change ORWT so that it will recognize the syntax in the TestExpectations files and treat any

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
On Wed, Jun 13, 2012 at 3:58 PM, Darin Adler da...@apple.com wrote: On Jun 13, 2012, at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: * we use \ (backslash) as a delimiter instead of : and = Seems worse to me. When I see a backslash I assume it’s a line continuation character or a C

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.org wrote: On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: webkit.org/12345 WIN MAC DEBUG \ animations/stop-animation

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
On Wed, Jun 13, 2012 at 4:38 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke dpra...@chromium.org wrote: On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain benja...@webkit.org wrote

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
:). -- Dirk On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek t...@codeaurora.org wrote: As long as we're considering TitleCase for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout --Tom On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: There's still

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
On Wed, Jun 13, 2012 at 5:05 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek t...@codeaurora.org wrote: As long as we're considering TitleCase for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout For

Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...

2012-06-13 Thread Dirk Pranke
On Wed, Jun 13, 2012 at 5:42 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 13, 2012, at 3:58 PM, Darin Adler da...@apple.com wrote: On Jun 13, 2012, at 3:53 PM, Dirk Pranke dpra...@chromium.org wrote: * we use \ (backslash) as a delimiter instead of : and = Seems worse to me. When I

Re: [webkit-dev] can we stop using Skipped files?

2012-06-12 Thread Dirk Pranke
On Mon, Jun 11, 2012 at 5:46 PM, Maciej Stachowiak m...@apple.com wrote: On Jun 10, 2012, at 9:26 AM, Ojan Vafai o...@chromium.org wrote: On Sun, Jun 10, 2012 at 4:54 AM, Balazs Kelemen kbal...@webkit.org wrote: So the unit tests are superfluous.  In particular, if I had to pick between

[webkit-dev] No (was Re: can we stop using Skipped files?)

2012-06-12 Thread Dirk Pranke
On Thu, Jun 7, 2012 at 4:47 PM, Dirk Pranke dpra...@chromium.org wrote: I believe most if not all of the ports have started using either TestExpectations files or a combination of TestExpectations files (except for the Apple Win port). Can we explicitly switch to the TestExpectations files

Re: [webkit-dev] No (was Re: can we stop using Skipped files?)

2012-06-12 Thread Dirk Pranke
On Mon, Jun 11, 2012 at 11:45 PM, Ryosuke Niwa rn...@webkit.org wrote: On Mon, Jun 11, 2012 at 11:37 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, Jun 7, 2012 at 4:47 PM, Dirk Pranke dpra...@chromium.org wrote: I believe most if not all of the ports have started using either

Re: [webkit-dev] Can we distinguish imported tests in LayoutTests/css3 ?

2012-06-11 Thread Dirk Pranke
On Mon, Jun 11, 2012 at 2:24 PM, Ryosuke Niwa rn...@webkit.org wrote: On Mon, Jun 11, 2012 at 2:21 PM, Jacob Goldstein jac...@adobe.com wrote: Can we just create an imported-w3c folder at the same level as LayoutTests? You mean at trunk? I don't think that makes sense, and our testing

Re: [webkit-dev] can we stop using Skipped files?

2012-06-09 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 8:58 PM, Zoltan Herczeg zherc...@webkit.org wrote: Hi Dirk, At any rate, I believe we are definitely open to adding new features; feel free to suggest them or work on them! I am happy to hear that. https://bugs.webkit.org/show_bug.cgi?id=88680 This is definitely a

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
Hi Ossy, Thanks for your reply ... On Fri, Jun 8, 2012 at 12:46 AM, Osztrogonac Csaba o...@inf.u-szeged.hu wrote: Hi, Dirk Pranke írta: I believe most if not all of the ports have started using either TestExpectations files or a combination of TestExpectations files (except for the Apple

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 8:21 AM, Filip Pizlo fpi...@apple.com wrote: On Jun 8, 2012, at 4:38 AM, Balazs Kelemen kbal...@webkit.org wrote: On 06/08/2012 09:46 AM, Osztrogonac Csaba wrote: Hi, Dirk Pranke írta: I believe most if not all of the ports have started using either TestExpectations

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 10:14 AM, Zoltan Herczeg zherc...@webkit.org wrote: Hi, I don't see why it would make sense to keep two parallel tools for this once all the workflow bugs people have are addressed. The reason is easy. In the past when people tried to add new features to NRWT, they

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 10:56 AM, Filip Pizlo fpi...@apple.com wrote: It's a lot harder to dive into, a lot more cumbersome to improve, and not any easier to maintain. I definitely agree that NRWT is more complicated than it seems like it should be; it got contorted as we added all the

Re: [webkit-dev] Defaulting --exit-after-n-failures to 0

2012-06-08 Thread Dirk Pranke
I have no objection either to increasing the defaults for either of these numbers or making it possible to have different defaults per port. Do you want to suggest different defaults? Should we use ORWT's (infinite failures and infinite crashes by default)? -- Dirk On Fri, Jun 8, 2012 at 12:31

Re: [webkit-dev] Defaulting --exit-after-n-failures to 0

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 12:44 PM, Ryosuke Niwa rn...@webkit.org wrote: On Fri, Jun 8, 2012 at 12:37 PM, Dirk Pranke dpra...@chromium.org wrote: I have no objection either to increasing the defaults for either of these numbers or making it possible to have different defaults per port. Do you

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 12:23 PM, Filip Pizlo fpi...@apple.com wrote: On Jun 8, 2012, at 12:19 PM, Dirk Pranke wrote: On Fri, Jun 8, 2012 at 8:21 AM, Filip Pizlo fpi...@apple.com wrote: On Jun 8, 2012, at 4:38 AM, Balazs Kelemen kbal...@webkit.org wrote: On 06/08/2012 09:46 AM, Osztrogonac

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 12:50 PM, Filip Pizlo fpi...@apple.com wrote: On Jun 8, 2012, at 12:31 PM, Dirk Pranke wrote: On Fri, Jun 8, 2012 at 10:56 AM, Filip Pizlo fpi...@apple.com wrote: It's a lot harder to dive into, a lot more cumbersome to improve, and not any easier to maintain. I

Re: [webkit-dev] can we stop using Skipped files?

2012-06-08 Thread Dirk Pranke
On Fri, Jun 8, 2012 at 1:16 PM, Ryosuke Niwa rn...@webkit.org wrote: I don't think that's true from my experience working on webkitpy so far. The root of problem is that we support way too many configurations platforms, and Chromium port has had a completely different test runner program

Re: [webkit-dev] PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT

2012-06-07 Thread Dirk Pranke
The reason for this (which is debatable) is that CRASH and TIMEOUT are deemed to be more serious and shouldn't be suppressed as lightly. MISSING, on the other hand, just indicates that there's something wrong (tests should never have missing results for very long). -- Dirk On Wed, Jun 6, 2012 at

Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)

2012-06-07 Thread Dirk Pranke
On Wed, Jun 6, 2012 at 10:36 PM, Peter Kasting pkast...@chromium.org wrote: On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa rn...@webkit.org wrote: Now that everyone knows the problem, I propose to rename FAIL to DIFF. FAIL should mean that the test fails, not that it fails with image, text,

[webkit-dev] can we stop using Skipped files?

2012-06-07 Thread Dirk Pranke
I believe most if not all of the ports have started using either TestExpectations files or a combination of TestExpectations files (except for the Apple Win port). Can we explicitly switch to the TestExpectations files at this point and drop support for Skipped files on the other ports (and

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 11:56 AM, Ryosuke Niwa rn...@webkit.org wrote: As I have said in the past, we should just import all tests, and treat non-text, non-ref tests as pixel tests. If we wanted to reduce the number of pixel tests we import, then we should submit those patches to W3C instead of

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 1:41 PM, Ryosuke Niwa rn...@webkit.org wrote: The only sane argument I've heard so far to gate pixel tests is that the correctness of such tests need to be manually inspected, which requires a lot of manual labor and is very error prone. I'm assuming the above

Re: [webkit-dev] Importing W3C tests to webkit

2012-05-23 Thread Dirk Pranke
On Wed, May 23, 2012 at 2:59 PM, Jacob Goldstein jac...@adobe.com wrote: As a side note to this discussion, there is talk in the W3C community regarding their test approval process.  At the recent working group meetings in Germany the idea was floated to simply approve all tests that are

Re: [webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-18 Thread Dirk Pranke
On Thu, May 17, 2012 at 11:03 PM, Ojan Vafai o...@chromium.org wrote: On Thu, May 17, 2012 at 10:37 PM, Maciej Stachowiak m...@apple.com wrote: On May 17, 2012, at 7:27 PM, Ojan Vafai o...@chromium.org wrote: On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak m...@apple.com wrote: On May

Re: [webkit-dev] Reconsidering test expectations (PASS, TEXT, IMAGE, TEXT+IMAGE, TIMEOUT, CRASH, etc...)

2012-05-18 Thread Dirk Pranke
On Fri, May 18, 2012 at 12:43 AM, Maciej Stachowiak m...@apple.com wrote: I guess we do. I think there is no point to saying PASS, because if a test crashes, hangs or is skipped it's meaningless. And if it does not crash and none of those other things apply, then it shouldn't be listed. But I

[webkit-dev] summarizing the current proposals for changing the test_expectations syntax (and semantics)

2012-05-18 Thread Dirk Pranke
Per Maciej's request, I will attempt to summarize both the decisions reached in the prior mega-thread, and the discussion that has so far occurred subsequently occurred on webkit.org/b/86749. This is largely requoting one of Ojan's earlier messages: Changes mostly agreed to: * WONTFIX will imply

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
There's lot of good discussion going on in this thread ... I'm going to attempt to reply to various threads in one message, hixie-style :) On Thu, May 17, 2012 at 1:06 AM, Maciej Stachowiak m...@apple.com wrote: SKIP and WONTFIX seem parallel to PASS to me. [and] On Thu, May 17, 2012 at 11:07

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 1:06 AM, Maciej Stachowiak m...@apple.com wrote: What does the build configuration info do? Does it apply the line to only those configurations? If that is the case, it does seem potentially different in kind, though maybe also better expressed by being able to combine

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 12:48 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 17, 2012 at 1:06 AM, Maciej Stachowiak m...@apple.com wrote: What does the build configuration info do? Does it apply the line to only those configurations? If that is the case, it does seem potentially

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 12:47 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, May 17, 2012 at 12:36 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 17, 2012 at 4:30 AM, Ojan Vafai o...@chromium.org wrote: -Make everything but the test name case-insensitive. I don't think I like

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 2:28 PM, Peter Kasting pkast...@chromium.org wrote: On Thu, May 17, 2012 at 2:11 PM, Ojan Vafai o...@chromium.org wrote: Oh, I supposed I misread Peter's earlier email as being opposed to this. You didn't misread me.  I have the same opinion as you: this would be a

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 2:39 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, May 17, 2012 at 2:36 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 17, 2012 at 2:28 PM, Peter Kasting pkast...@chromium.org wrote: On Thu, May 17, 2012 at 2:11 PM, Ojan Vafai o...@chromium.org wrote

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 3:21 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, May 17, 2012 at 3:01 PM, Dirk Pranke dpra...@chromium.org wrote: As I said before, I believe they increase the readability of the file. I see them as pure noise. Clearly, different people can have different

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 4:00 PM, Maciej Stachowiak m...@apple.com wrote: On May 17, 2012, at 12:53 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 17, 2012 at 12:47 PM, Ryosuke Niwa rn...@webkit.org wrote: I find either all-lowercase or all-caps to be much harder to read than

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 4:16 PM, Maciej Stachowiak m...@apple.com wrote: On May 17, 2012, at 3:37 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 17, 2012 at 3:21 PM, Ryosuke Niwa rn...@webkit.org wrote: On Thu, May 17, 2012 at 3:01 PM, Dirk Pranke dpra...@chromium.org wrote: As I

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 4:50 PM, Maciej Stachowiak m...@apple.com wrote: On May 17, 2012, at 4:40 PM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 17, 2012 at 4:16 PM, Maciej Stachowiak m...@apple.com wrote: Let's take an example. TEXT next to a test name apparently means

Re: [webkit-dev] Merging Skipped and test_expectations.txt formats WAS: Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-17 Thread Dirk Pranke
On Thu, May 17, 2012 at 7:27 PM, Ojan Vafai o...@chromium.org wrote: On Thu, May 17, 2012 at 4:29 PM, Maciej Stachowiak m...@apple.com wrote: On May 17, 2012, at 1:42 PM, Ojan Vafai o...@chromium.org wrote: On Thu, May 17, 2012 at 1:37 PM, Peter Kasting pkast...@chromium.org wrote: On

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-16 Thread Dirk Pranke
On Wed, May 16, 2012 at 9:08 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi, There has been some complaints / discussions about how syntax in test_expectations.txt is confusing (and I agree with you) on webkit-dev and at contributors' meeting. So I have a patch on 

Re: [webkit-dev] Simplifying syntax in test_expectations.txt (bug 86691)

2012-05-16 Thread Dirk Pranke
On Wed, May 16, 2012 at 10:30 PM, Maciej Stachowiak m...@apple.com wrote: On May 16, 2012, at 9:08 PM, Ryosuke Niwa rn...@webkit.org wrote: Hi, There has been some complaints / discussions about how syntax in test_expectations.txt is confusing (and I agree with you) on webkit-dev and at

Re: [webkit-dev] EFL Debug Buildbot Green

2012-05-10 Thread Dirk Pranke
+1 :). On Thu, May 10, 2012 at 12:01 PM, Ojan Vafai o...@chromium.org wrote: That's a great milestone. Congratulations! On Thu, May 10, 2012 at 8:43 AM, Dominik Röttsches dominik.rottsc...@intel.com wrote: Hi all, We're happy to share with you that yesterday the EFL Linux Debug Buildbot

Re: [webkit-dev] Process for importing third party tests

2012-05-08 Thread Dirk Pranke
Thanks for the comments, Ryosuke. My replies are inline ... On Tue, May 8, 2012 at 12:42 PM, Ryosuke Niwa cont...@rniwa.com wrote: On Wed, Apr 25, 2012 at 2:18 PM, Dirk Pranke dpra...@chromium.org wrote: 1b. Run suite locally in WebKit directory   * Ref Tests     * Pass - good, submit

Re: [webkit-dev] LayoutTestHelper

2012-05-03 Thread Dirk Pranke
On Thu, May 3, 2012 at 7:35 AM, Maciej Stachowiak m...@apple.com wrote: Better naming/documentation may help. But I think the root problem here was that the Chromium Android LayoutTestHelper does not exist in the WebKit repository, as far as I can tell. So there was no reasonable way for

Re: [webkit-dev] LayoutTestHelper

2012-05-03 Thread Dirk Pranke
On Thu, May 3, 2012 at 9:16 AM, Maciej Stachowiak m...@apple.com wrote: On May 3, 2012, at 8:54 AM, Dirk Pranke dpra...@chromium.org wrote: On Thu, May 3, 2012 at 7:35 AM, Maciej Stachowiak m...@apple.com wrote: Better naming/documentation may help. But I think the root problem here

Re: [webkit-dev] LayoutTestHelper

2012-04-30 Thread Dirk Pranke
On Sun, Apr 29, 2012 at 6:20 PM, Maciej Stachowiak m...@apple.com wrote: On Apr 29, 2012, at 5:49 PM, Maciej Stachowiak m...@apple.com wrote: Hi folks, new-run-webkit-tests seems to mess with the system color profile on Mac, even when not running pixel tests. Historically, I believe we

Re: [webkit-dev] Deprecating features guideline wiki

2012-04-27 Thread Dirk Pranke
Hi Julien, Thanks for the writeup! I definitely wanted to be in that discussion but it conflicted with another discussion I wanted to be in. It looks like I more or less agree with the writeup, though :). That said, what sort of officialness does this writeup/guideline/policy have? Do all the

Re: [webkit-dev] Deprecating features guideline wiki

2012-04-27 Thread Dirk Pranke
On Fri, Apr 27, 2012 at 5:17 PM, Maciej Stachowiak m...@apple.com wrote: I think the concern is that, due to lack of clear standards, we end up not knowing when we can remove things. Thus, we end up with a lot of inconclusive and frustrating conversations, and people may shy away from even

Re: [webkit-dev] Process for importing third party tests

2012-04-25 Thread Dirk Pranke
On Mon, Apr 23, 2012 at 4:13 PM, Jacob Goldstein jac...@adobe.com wrote: At the recent WebKit Contributors Meeting, a process was drafted for importing third party tests into WebKit. I created a wiki page that captures the process that we came up with here:

Re: [webkit-dev] Bot watching and Apple bots

2012-04-25 Thread Dirk Pranke
On Wed, Apr 25, 2012 at 2:40 PM, Ryosuke Niwa rn...@webkit.org wrote: On Wed, Apr 25, 2012 at 2:34 PM, Adrienne Walker e...@google.com wrote: Also, when you say pay attention, what level of involvement should we expect for one committer making the bots go red on a port that they are not

Re: [webkit-dev] Bot watching and Apple bots

2012-04-25 Thread Dirk Pranke
On Wed, Apr 25, 2012 at 3:50 PM, Ojan Vafai o...@chromium.org wrote: Good to know. I had stopped paying attention to many of the Apple bots for the reason you mention. It would be really helpful if someone could make the webkit-patch tooling works correctly for the non-Chromium bots.

<    1   2   3   4   >