Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 13, 2012, at 3:53 PM, Dirk Pranke wrote: > * we use "\" (backslash) as a delimiter instead of ":" and "=" Seems worse to me. When I see a backslash I assume it’s a line continuation character or a C escape sequence. On the other hand, neither the ":" nor the "=" meant anything to me either, and I suggested dropping the delimiter entirely, so I’m not sure my feedback will get listened to! > * As of now the keywords remain all UPPERCASE. I'm happy to change them to > all lowercase or mixed case if there's a consensus that such a change is > desired. I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT TO CHANGE. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke wrote: > webkit.org/12345 WIN MAC DEBUG \ > animations/stop-animation-on-suspend.html \ CRASH TEXT PASS My bikeshedding: -In my opinion, the backslash is not any better than the old delimiter. It looks like we are escaping something. I liked better the proposal to have platform and results grouped instead of the delimiter. e.g: webkit.org/12345 (WIN MAC DEBUG) animations/stop-animation-on-suspend.html (CRASH TEXT PASS) -I prefer lowercase keywords, possibly with the first character uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all uppercase keywords somewhere else in the project? I like the other changes. :) Cheers, Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 3:58 PM, Darin Adler wrote: > On Jun 13, 2012, at 3:53 PM, Dirk Pranke wrote: > > * we use "\" (backslash) as a delimiter instead of ":" and "=" > > Seems worse to me. When I see a backslash I assume it’s a line > continuation character or a C escape sequence. > Gotta admit this one mystifies me too. I must have missed where someone suggested this in the last thread. > * As of now the keywords remain all UPPERCASE. I'm happy to change them > to all lowercase or mixed case if there's a consensus that such a change is > desired. > Given the last thread it seems clear there will not be consensus on any outcome of this particular question. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain wrote: > On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke wrote: > > webkit.org/12345 WIN MAC DEBUG \ > > animations/stop-animation-on-suspend.html \ CRASH TEXT PASS > > My bikeshedding: > -In my opinion, the backslash is not any better than the old > delimiter. It looks like we are escaping something. > > I liked better the proposal to have platform and results grouped > instead of the delimiter. > e.g: webkit.org/12345 (WIN MAC DEBUG) > animations/stop-animation-on-suspend.html (CRASH TEXT PASS) > > -I prefer lowercase keywords, possibly with the first character > uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all > uppercase keywords somewhere else in the project? > So something like webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? I like that! - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 3:58 PM, Darin Adler wrote: > On Jun 13, 2012, at 3:53 PM, Dirk Pranke wrote: > >> * we use "\" (backslash) as a delimiter instead of ":" and "=" > > Seems worse to me. When I see a backslash I assume it’s a line continuation > character or a C escape sequence. > > On the other hand, neither the ":" nor the "=" meant anything to me either, > and I suggested dropping the delimiter entirely, so I’m not sure my feedback > will get listened to! > That's a bit harsh. "listened to" does not necessarily mean "will do what you ask" :). Plenty of other people have said that they liked delimiters, so I think you just got outvoted. Are there other delimiters you might prefer to ':' and '=' ? Others suggestions have included bracketing the test name , and or uniformly using one of : , = - ; , or as Benjamin or Ryoskue have suggested, bracketing the keywords ... For the record, I didn't like bracketing the test name since brackets make it look optional. I could be open to bracketing the modifiers and keywords, since the modifiers are optional, and it would seem reasonable to make the absence of keywords equivalent to SKIP ... >> * As of now the keywords remain all UPPERCASE. I'm happy to change them to >> all lowercase or mixed case if there's a consensus that such a change is >> desired. > > I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT TO > CHANGE. > Would you prefer lowercase, lowercase_with_underscores, Initialcaps, MixedCase ... ? On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain wrote: > On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke wrote: >> webkit.org/12345 WIN MAC DEBUG \ >> animations/stop-animation-on-suspend.html \ CRASH TEXT PASS > > My bikeshedding: > -In my opinion, the backslash is not any better than the old > delimiter. It looks like we are escaping something. > > I liked better the proposal to have platform and results grouped > instead of the delimiter. > e.g: webkit.org/12345 (WIN MAC DEBUG) > animations/stop-animation-on-suspend.html (CRASH TEXT PASS) > > -I prefer lowercase keywords, possibly with the first character > uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all > uppercase keywords somewhere else in the project? C++ macros are the only place I know of. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa wrote: > On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain > wrote: >> >> On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke wrote: >> > webkit.org/12345 WIN MAC DEBUG \ >> > animations/stop-animation-on-suspend.html \ CRASH TEXT PASS >> >> My bikeshedding: >> -In my opinion, the backslash is not any better than the old >> delimiter. It looks like we are escaping something. >> >> I liked better the proposal to have platform and results grouped >> instead of the delimiter. >> e.g: webkit.org/12345 (WIN MAC DEBUG) >> animations/stop-animation-on-suspend.html (CRASH TEXT PASS) >> >> -I prefer lowercase keywords, possibly with the first character >> uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all >> uppercase keywords somewhere else in the project? > > > So something like > webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html > [Crash Text Pass] > ? I like that! This seems quite readable to me as well. As mentioned above, I'd be inclined to make the absence of [] on the right hand side mean "SKIP". ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke wrote: > On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa wrote: > > On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain > > wrote: > >> On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke > wrote: > >> > webkit.org/12345 WIN MAC DEBUG \ > >> > animations/stop-animation-on-suspend.html \ CRASH TEXT PASS > >> > >> My bikeshedding: > >> -In my opinion, the backslash is not any better than the old > >> delimiter. It looks like we are escaping something. > >> > >> I liked better the proposal to have platform and results grouped > >> instead of the delimiter. > >> e.g: webkit.org/12345 (WIN MAC DEBUG) > >> animations/stop-animation-on-suspend.html (CRASH TEXT PASS) > >> > >> -I prefer lowercase keywords, possibly with the first character > >> uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all > >> uppercase keywords somewhere else in the project? > > > > So something like > > webkit.org/12345 [Win Mac > Debug] animations/stop-animation-on-suspend.html > > [Crash Text Pass] > > ? I like that! > > This seems quite readable to me as well. As mentioned above, I'd be > inclined to make the absence of [] on the right hand side mean "SKIP". > That sounds very reasonable. In fact, I'd argue that it should be the only way to skip tests. It makes no sense to have any other test expectation when a test is skipped because the types of failures of skipped tests (when actually ran) will very likely change over time, and there is no way to track them. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:38 PM, Ryosuke Niwa wrote: > On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke wrote: >> >> On Wed, Jun 13, 2012 at 4:26 PM, Ryosuke Niwa wrote: >> > On Wed, Jun 13, 2012 at 4:12 PM, Benjamin Poulain >> > wrote: >> >> On Wed, Jun 13, 2012 at 3:53 PM, Dirk Pranke >> >> wrote: >> >> > webkit.org/12345 WIN MAC DEBUG \ >> >> > animations/stop-animation-on-suspend.html \ CRASH TEXT PASS >> >> >> >> My bikeshedding: >> >> -In my opinion, the backslash is not any better than the old >> >> delimiter. It looks like we are escaping something. >> >> >> >> I liked better the proposal to have platform and results grouped >> >> instead of the delimiter. >> >> e.g: webkit.org/12345 (WIN MAC DEBUG) >> >> animations/stop-animation-on-suspend.html (CRASH TEXT PASS) >> >> >> >> -I prefer lowercase keywords, possibly with the first character >> >> uppercase (Crash, Text, Pass, Fail, Win, Mac, Debug). Do we use all >> >> uppercase keywords somewhere else in the project? >> > >> > So something like >> > webkit.org/12345 [Win Mac >> > Debug] animations/stop-animation-on-suspend.html >> > [Crash Text Pass] >> > ? I like that! >> >> This seems quite readable to me as well. As mentioned above, I'd be >> inclined to make the absence of [] on the right hand side mean "SKIP". > > > That sounds very reasonable. > > In fact, I'd argue that it should be the only way to skip tests. It makes no > sense to have any other test expectation when a test is skipped because the > types of failures of skipped tests (when actually ran) will very likely > change over time, and there is no way to track them. > There's still a distinction between SKIP (temporary) and WONTFIX (permanent until some plan changes) that we should preserve, I think. It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but I agree that the type of failure will probably change over time. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:32 PM, Dirk Pranke wrote: >> So something like >> webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html >> [Crash Text Pass] >> ? I like that! > > This seems quite readable to me as well. As mentioned above, I'd be > inclined to make the absence of [] on the right hand side mean "SKIP". I like that. Maybe TestExpectations is not the right name because of Skip? TestExceptions would be better? I don't care about the name of the file, this is just a random idea. Benjamin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
As long as we're considering "TitleCase" for the keywords, could we use it to keep all of them as single words? WontFix, SkipCrash, SkipTimeout --Tom On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: > > There's still a distinction between SKIP (temporary) and WONTFIX > (permanent until some plan changes) that we should preserve, I think. > It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but > I agree that the type of failure will probably change over time. > > -- Dirk > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev On Jun 13, 2012 at 4:32 PM, Dirk Pranke wrote: >> So something like >> webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html >> [Crash Text Pass] >> ? I like that! > > This seems quite readable to me as well. As mentioned above, I'd be > inclined to make the absence of [] on the right hand side mean "SKIP". ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
"Skip" and "WontFix" have a different purpose than "Crash" and "timeout"; the first two (like "slow") tell you what to do with the test, and the latter tell you what you expect the test to result in. I don't want to combine them into single tokens because that would cause an explosion of tokens :). -- Dirk On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek wrote: > As long as we're considering "TitleCase" for the keywords, could we use it > to keep all of them as single words? > > WontFix, SkipCrash, SkipTimeout > > --Tom > > On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: > > > There's still a distinction between SKIP (temporary) and WONTFIX > (permanent until some plan changes) that we should preserve, I think. > It could be useful to indicate SKIP CRASH or SKIP TIMEOUT as well, but > I agree that the type of failure will probably change over time. > > -- Dirk > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > > On Jun 13, 2012 at 4:32 PM, Dirk Pranke wrote: > > So something like > > webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html > > [Crash Text Pass] > > ? I like that! > > > This seems quite readable to me as well. As mentioned above, I'd be > > inclined to make the absence of [] on the right hand side mean "SKIP". ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek wrote: > As long as we're considering "TitleCase" for the keywords, could we use it > to keep all of them as single words? > > WontFix, SkipCrash, SkipTimeout > For skipped tests, it doesn't make sense to have crash, timeout, etc... because we wouldn't know even if failure types changed from text to crash, or from timeout to image. It's just a pure noise as far as I'm concerned. On Jun 13, 2012, at 4:46 PM, Dirk Pranke wrote: > There's still a distinction between SKIP (temporary) and WONTFIX > (permanent until some plan changes) that we should preserve, I think. The way I see it, WontFix and NotImplemented, like bug URL, are orthogonal to other test expectations. They tell us "why" we have a given test expectation but doesn't define the expectation itself. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 5:05 PM, Ryosuke Niwa wrote: > On Wed, Jun 13, 2012 at 4:55 PM, Tom Zakrajsek wrote: >> >> As long as we're considering "TitleCase" for the keywords, could we use it >> to keep all of them as single words? >> >> WontFix, SkipCrash, SkipTimeout > > > For skipped tests, it doesn't make sense to have crash, timeout, etc... > because we wouldn't know even if failure types changed from text to crash, > or from timeout to image. It's just a pure noise as far as I'm concerned. > Well, when you add the skip rule, you presumably know what happens when you run the test. Since you can actually run tests that are skipped using --force or --skipped=ignore/only (which I'm adding to NRWT as we speak), it's not quite noise, but I agree that it's pretty close. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 13, 2012, at 3:58 PM, Darin Adler wrote: > On Jun 13, 2012, at 3:53 PM, Dirk Pranke wrote: > >> * we use "\" (backslash) as a delimiter instead of ":" and "=" > > Seems worse to me. When I see a backslash I assume it’s a line continuation > character or a C escape sequence. > > On the other hand, neither the ":" nor the "=" meant anything to me either, > and I suggested dropping the delimiter entirely, so I’m not sure my feedback > will get listened to! > >> * As of now the keywords remain all UPPERCASE. I'm happy to change them to >> all lowercase or mixed case if there's a consensus that such a change is >> desired. > > I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT TO > CHANGE. I also hate all-caps keywords and would prefer mixed case or all-lowercase, but I'm willing to discuss that for a future round of changes if we can't get agreement now. Regards, Maciej ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 5:42 PM, Maciej Stachowiak wrote: > > On Jun 13, 2012, at 3:58 PM, Darin Adler wrote: > >> On Jun 13, 2012, at 3:53 PM, Dirk Pranke wrote: >> >>> * we use "\" (backslash) as a delimiter instead of ":" and "=" >> >> Seems worse to me. When I see a backslash I assume it’s a line continuation >> character or a C escape sequence. >> >> On the other hand, neither the ":" nor the "=" meant anything to me either, >> and I suggested dropping the delimiter entirely, so I’m not sure my feedback >> will get listened to! >> >>> * As of now the keywords remain all UPPERCASE. I'm happy to change them to >>> all lowercase or mixed case if there's a consensus that such a change is >>> desired. >> >> I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT >> TO CHANGE. > > I also hate all-caps keywords and would prefer mixed case or all-lowercase, > but I'm willing to discuss that for a future round of changes if we can't get > agreement now. > I think most of the people who liked all-caps mostly wanted there to be *some* sort of delimiter. Anyone not okay with: webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html [Crash Text Pass] ? -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 5:49 PM, Dirk Pranke wrote: > On Wed, Jun 13, 2012 at 5:42 PM, Maciej Stachowiak wrote: >> On Jun 13, 2012, at 3:58 PM, Darin Adler wrote: >>> On Jun 13, 2012, at 3:53 PM, Dirk Pranke wrote: >>> * we use "\" (backslash) as a delimiter instead of ":" and "=" >>> >>> Seems worse to me. When I see a backslash I assume it’s a line continuation >>> character or a C escape sequence. >>> >>> On the other hand, neither the ":" nor the "=" meant anything to me either, >>> and I suggested dropping the delimiter entirely, so I’m not sure my >>> feedback will get listened to! >>> * As of now the keywords remain all UPPERCASE. I'm happy to change them to all lowercase or mixed case if there's a consensus that such a change is desired. >>> >>> I DON'T LIKE READING THINGS IN ALL UPPERCASE AND WOULD VERY MUCH LIKE THAT >>> TO CHANGE. >> >> I also hate all-caps keywords and would prefer mixed case or all-lowercase, >> but I'm willing to discuss that for a future round of changes if we can't >> get agreement now. >> > > I think most of the people who liked all-caps mostly wanted there to > be *some* sort of delimiter. > > Anyone not okay with: > > webkit.org/12345 [Win Mac Debug] > animations/stop-animation-on-suspend.html [Crash Text Pass] > > ? I think you need a /b/ in there if you want the link to be copy/pastable: webkit.org/b/12345 Adam P.S., Please feel free to ignore this suggestion, but this is my favorite shade of purple: animations/stop-animation-on-suspend.html Crash Text Pass (Win Mac Debug) ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 5:49 PM, Dirk Pranke wrote: > Anyone not okay with: > > webkit.org/12345 [Win Mac Debug] > animations/stop-animation-on-suspend.html [Crash Text Pass] > I withdraw my objections to this in the hopes of getting _some_ kind of consensus and moving on with our lives :) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 13, 2012, at 4:26 PM, Ryosuke Niwa wrote: > So something like > > webkit.org/12345 [Win Mac Debug] animations/stop-animation-on-suspend.html > [Crash Text Pass] > > ? I like that! Yes, looks good to me too. -- Darin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Here are some examples of old and new format: BUGCR24182 SLOW MAC DEBUG : fast/css/large-list-of-rules-crash.html = PASS BUGCR24182 SLOW LINUX MAC DEBUG : fast/dom/Window/window-postmessage-clone-really-deep-array.html = PASS BUGCR24182 SLOW MAC DEBUG : fast/forms/select-set-length-with-mutation-remove.html = PASS crbug.com/24182 [Mac Debug] fast/css/large-list-of-rules-crash.html [Slow] crbug.com/24182 [Linux Mac Debug] fast/dom/Window/window-postmessage-clone-really-deep-array.html [Slow] crbug.com/24182 [Mac Debug] fast/dom/Window/window-postmessage-clone-really-deep-array.html [Slow] BUGWK87364 DEBUG : fast/dom/shadow/drop-event-for-input-in-shadow.html = TEXT PASS BUGWK87364 DEBUG : fast/dom/shadow/drop-event-in-shadow.html = TEXT PASS webkit.org/b/87364 [Debug] fast/dom/shadow/drop-event-for-input-in-shadow.html [Text Pass] webkit.org/b/87364 [Debug] fast/dom/shadow/drop-event-in-shadow.html [Text Pass] BUG_EAE SKIP : fast/repaint/reflection-repaint-test.html = PASS BUG_EAE SKIP : fast/repaint/transform-layout-repaint.html = PASS Bug(eae) fast/repaint/reflection-repaint-test.html [Skip] Bug(eae) fast/repaint/transform-layout-repaint.html [Skip] I've started to think that we might want to keep WontFix at the beginning of each line as supposed to placing them at the end of line as initially proposed: Beginning of line (current proposal with expectations): WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT media/media-captions.html [WontFix] webkit.org/b/43459 WontFix http/tests/appcache/origin-quota.html [WontFix Missing Text Image+Text] End of line: WONTFIX SKIP : media/media-captions.html = TIMEOUT BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT IMAGE+TEXT WontFix media/media-captions.html WontFix webkit.org/b/43459 http/tests/appcache/origin-quota.html [Missing Text Image+Text] In fact, there are very few entries of WontFix with a bug number at least in Chromium port's test expectations file. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
Oops, some really bad typos. On Thu, Jun 14, 2012 at 3:10 AM, Ryosuke Niwa wrote: > > Beginning of line (current proposal with expectations): > End of line with expectations (current proposal) WONTFIX SKIP : media/media-captions.html = TIMEOUT > BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT > IMAGE+TEXT > > media/media-captions.html [WontFix] > webkit.org/b/43459 WontFix http/tests/appcache/origin-quota.html [WontFix > Missing Text Image+Text] > > End of line: Beginning of line WONTFIX SKIP : media/media-captions.html = TIMEOUT > BUGWK43459 WONTFIX : http/tests/appcache/origin-quota.html = MISSING TEXT > IMAGE+TEXT > > WontFix media/media-captions.html > WontFix webkit.org/b/43459 http/tests/appcache/origin-quota.html [Missing > Text Image+Text] > > In fact, there are very few entries of WontFix with a bug number at least > in Chromium port's test expectations file. > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Wed, Jun 13, 2012 at 6:53 PM, Dirk Pranke wrote: > > * We'll probably rename IMAGE+TEXT to IMAGE_AND_TEXT. > > Can someone please remind me why IMAGE+TEXT even exists? Wouldn't it be simpler to just mark a test as follows? - IMAGE : allow image failure; go red if there is a text failure - TEXT: allow text failure; go red if there is an image failure - IMAGE TEXT: allow text and/or image failure ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote: > Can someone please remind me why IMAGE+TEXT even exists? > > Wouldn't it be simpler to just mark a test as follows? > >- IMAGE : allow image failure; go red if there is a text failure >- TEXT: allow text failure; go red if there is an image failure >- IMAGE TEXT: allow text and/or image failure > > The distinction is that IMAGE TEXT will allow image, text, or both to fail, thus making transitions among the three generate no events. IMAGE+TEXT says specifically that we expect both to fail and that if one starts passing, someone should do something. (For example, maybe someone checks in a partial rebaseline where they miss the image expectations.) PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting wrote: > On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote: > >> Can someone please remind me why IMAGE+TEXT even exists? >> >> Wouldn't it be simpler to just mark a test as follows? >> >>- IMAGE : allow image failure; go red if there is a text failure >>- TEXT: allow text failure; go red if there is an image failure >>- IMAGE TEXT: allow text and/or image failure >> >> The distinction is that IMAGE TEXT will allow image, text, or both to > fail, thus making transitions among the three generate no events. > IMAGE+TEXT says specifically that we expect both to fail and that if one > starts passing, someone should do something. (For example, maybe someone > checks in a partial rebaseline where they miss the image expectations.) > Not to bike-shed on anything, but I think we should rename Text and Image to TextOnly and ImageOnly. Every single person I know, including myself, had never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone explaining it to him/her . - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:44 PM, Peter Kasting wrote: > On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote: > >> Can someone please remind me why IMAGE+TEXT even exists? >> >> Wouldn't it be simpler to just mark a test as follows? >> >>- IMAGE : allow image failure; go red if there is a text failure >>- TEXT: allow text failure; go red if there is an image failure >>- IMAGE TEXT: allow text and/or image failure >> >> The distinction is that IMAGE TEXT will allow image, text, or both to > fail, thus making transitions among the three generate no events. > IMAGE+TEXT says specifically that we expect both to fail and that if one > starts passing, someone should do something. (For example, maybe someone > checks in a partial rebaseline where they miss the image expectations.) > Thanks for the explanation. That makes sense, although it seems to me that the problem of no-events-generated-by-changes-in-actual-images-while-IMAGE-failure-is-expected is about 100x worse for us. But that's not a reason to hide these particular transitions! :-) > > PK > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting wrote: > On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote: >> >> Can someone please remind me why IMAGE+TEXT even exists? >> >> Wouldn't it be simpler to just mark a test as follows? >> >> IMAGE : allow image failure; go red if there is a text failure >> TEXT: allow text failure; go red if there is an image failure >> IMAGE TEXT: allow text and/or image failure > > The distinction is that IMAGE TEXT will allow image, text, or both to fail, > thus making transitions among the three generate no events. IMAGE+TEXT says > specifically that we expect both to fail and that if one starts passing, > someone should do something. (For example, maybe someone checks in a > partial rebaseline where they miss the image expectations.) > Also, unlike the now-deleted FAIL keyword, IMAGE TEXT indicates that the test is expected to be flaky. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa wrote:
>
> On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting wrote:
> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote:
> Can someone please remind me why IMAGE+TEXT even exists?
>
> Wouldn't it be simpler to just mark a test as follows?
> IMAGE : allow image failure; go red if there is a text failure
> TEXT: allow text failure; go red if there is an image failure
> IMAGE TEXT: allow text and/or image failure
> The distinction is that IMAGE TEXT will allow image, text, or both to fail,
> thus making transitions among the three generate no events. IMAGE+TEXT says
> specifically that we expect both to fail and that if one starts passing,
> someone should do something. (For example, maybe someone checks in a partial
> rebaseline where they miss the image expectations.)
>
> Not to bike-shed on anything, but I think we should rename Text and Image to
> TextOnly and ImageOnly. Every single person I know, including myself, had
> never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
> explaining it to him/her .
I think IMAGE+TEXT is not a very useful distinction from TEXT either. I checked
for uses of TEXT that is not IMAGE+TEXT in the Chromium TextExpectations, and
it seems that nearly all instances fall into one of the two following
categories:
1) text-only test, so IMAGE+TEXT would not have different semantics from TEXT
(the vast majority)
2) Flaky test that may actually pass, so distinguishing what happens with the
image result is of limited utility (most of these are also text-only tests;
only a small subset even have an image result)
Thus, I think Fail and ImageOnlyFail would be more useful and understandable
categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
semantic that a text failure is expected, and image result if any can either
pass or fail.
Regards,
Maciej
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak wrote:
> On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa wrote:
> On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting
> wrote:
>> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote:
>>> Can someone please remind me why IMAGE+TEXT even exists?
>>>
>>> Wouldn't it be simpler to just mark a test as follows?
>>>
>>> IMAGE : allow image failure; go red if there is a text failure
>>> TEXT: allow text failure; go red if there is an image failure
>>> IMAGE TEXT: allow text and/or image failure
>>
>> The distinction is that IMAGE TEXT will allow image, text, or both to
>> fail, thus making transitions among the three generate no events.
>> IMAGE+TEXT says specifically that we expect both to fail and that if one
>> starts passing, someone should do something. (For example, maybe someone
>> checks in a partial rebaseline where they miss the image expectations.)
>
> Not to bike-shed on anything, but I think we should rename Text and Image to
> TextOnly and ImageOnly. Every single person I know, including myself, had
> never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
> explaining it to him/her .
>
> I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
> checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
> TextExpectations, and it seems that nearly all instances fall into one of
> the two following categories:
>
> 1) text-only test, so IMAGE+TEXT would not have different semantics from
> TEXT (the vast majority)
> 2) Flaky test that may actually pass, so distinguishing what happens with
> the image result is of limited utility (most of these are also text-only
> tests; only a small subset even have an image result)
>
> Thus, I think Fail and ImageOnlyFail would be more useful and understandable
> categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
> semantic that a text failure is expected, and image result if any can either
> pass or fail.
I too would like to see us remove TEXT+IMAGE. It's really confusing
to non-experts, and it doesn't scale as we introduce new kinds of
failures (like Audio). Do we really need TEXT+IMAGE+AUDIO,
TEXT+AUDIO, and IMAGE+AUDIO?
Adam
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth wrote: > I too would like to see us remove TEXT+IMAGE. It's really confusing > to non-experts, and it doesn't scale as we introduce new kinds of > failures (like Audio). Do we really need TEXT+IMAGE+AUDIO, > TEXT+AUDIO, and IMAGE+AUDIO? AUDIO tests can only produce audio output, not text or images, so, no. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak wrote:
>
> On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa wrote:
>
>
> On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting
> wrote:
>>
>> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger wrote:
>>>
>>> Can someone please remind me why IMAGE+TEXT even exists?
>>>
>>> Wouldn't it be simpler to just mark a test as follows?
>>>
>>> IMAGE : allow image failure; go red if there is a text failure
>>> TEXT: allow text failure; go red if there is an image failure
>>> IMAGE TEXT: allow text and/or image failure
>>
>> The distinction is that IMAGE TEXT will allow image, text, or both to
>> fail, thus making transitions among the three generate no events.
>> IMAGE+TEXT says specifically that we expect both to fail and that if one
>> starts passing, someone should do something. (For example, maybe someone
>> checks in a partial rebaseline where they miss the image expectations.)
>
>
> Not to bike-shed on anything, but I think we should rename Text and Image to
> TextOnly and ImageOnly. Every single person I know, including myself, had
> never got the distinction between IMAGE TEXT and IMAGE+TEXT without someone
> explaining it to him/her .
>
>
> I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
> checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
> TextExpectations, and it seems that nearly all instances fall into one of
> the two following categories:
>
> 1) text-only test, so IMAGE+TEXT would not have different semantics from
> TEXT (the vast majority)
> 2) Flaky test that may actually pass, so distinguishing what happens with
> the image result is of limited utility (most of these are also text-only
> tests; only a small subset even have an image result)
>
> Thus, I think Fail and ImageOnlyFail would be more useful and understandable
> categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have the
> semantic that a text failure is expected, and image result if any can either
> pass or fail.
This is perhaps true, but if it's okay I would like to treat that
feature request separately from the other syntactic changes we've been
discussing. So far the rest of the changes have not really implied any
changes to how we actually track which changes fail and how (note that
FAIL is different and we've fixed that separately from these changes
as well).
-- Dirk
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] TestExpectations syntax changes, last call (for a while, at least) ...
On Thu, Jun 14, 2012 at 4:34 PM, Adam Barth wrote:
> On Thu, Jun 14, 2012 at 4:22 PM, Maciej Stachowiak wrote:
> > On Jun 14, 2012, at 1:47 PM, Ryosuke Niwa wrote:
> > On Thu, Jun 14, 2012 at 1:44 PM, Peter Kasting
> > wrote:
> >> On Thu, Jun 14, 2012 at 1:39 PM, Elliot Poger
> wrote:
> >>> Can someone please remind me why IMAGE+TEXT even exists?
> >>>
> >>> Wouldn't it be simpler to just mark a test as follows?
> >>>
> >>> IMAGE : allow image failure; go red if there is a text failure
> >>> TEXT: allow text failure; go red if there is an image failure
> >>> IMAGE TEXT: allow text and/or image failure
> >>
> >> The distinction is that IMAGE TEXT will allow image, text, or both to
> >> fail, thus making transitions among the three generate no events.
> >> IMAGE+TEXT says specifically that we expect both to fail and that if
> one
> >> starts passing, someone should do something. (For example, maybe
> someone
> >> checks in a partial rebaseline where they miss the image expectations.)
> >
> > Not to bike-shed on anything, but I think we should rename Text and
> Image to
> > TextOnly and ImageOnly. Every single person I know, including myself, had
> > never got the distinction between IMAGE TEXT and IMAGE+TEXT without
> someone
> > explaining it to him/her .
> >
> > I think IMAGE+TEXT is not a very useful distinction from TEXT either. I
> > checked for uses of TEXT that is not IMAGE+TEXT in the Chromium
> > TextExpectations, and it seems that nearly all instances fall into one of
> > the two following categories:
> >
> > 1) text-only test, so IMAGE+TEXT would not have different semantics from
> > TEXT (the vast majority)
> > 2) Flaky test that may actually pass, so distinguishing what happens with
> > the image result is of limited utility (most of these are also text-only
> > tests; only a small subset even have an image result)
> >
> > Thus, I think Fail and ImageOnlyFail would be more useful and
> understandable
> > categories than {TEXT, IMAGE, TEXT+IMAGE, TEXT IMAGE}. Fail would have
> the
> > semantic that a text failure is expected, and image result if any can
> either
> > pass or fail.
>
> I too would like to see us remove TEXT+IMAGE. It's really confusing
> to non-experts, and it doesn't scale as we introduce new kinds of
> failures (like Audio). Do we really need TEXT+IMAGE+AUDIO,
> TEXT+AUDIO, and IMAGE+AUDIO?
>
+1 to that. Also, I can never remember whether it's IMAGE+TEXT or
TEXT+IMAGE (it's IMAGE+TEXT).
But I agree with Dirk that we should probably discuss about this on a
separate thread.
- Ryosuke
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
