[webkit-dev] can we stop using Skipped files?
I believe most if not all of the ports have started using either TestExpectations files or a combination of TestExpectations files (except for the Apple Win port). Can we explicitly switch to the TestExpectations files at this point and drop support for Skipped files on the other ports (and perhaps disable old-run-webkit-tests for all but apple win)? If not, what needs to happen so that we can do that? Do we need to land more of the discussed changes to the syntax of the TestExpectations file? Anything else? It would be nice to get rid of the complexity (both in the code and in our heads) if we can. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Thu, Jun 7, 2012 at 4:41 PM, Brendan Eich wrote: > Adam Barth wrote: >> >> On Thu, Jun 7, 2012 at 1:00 PM, Ryosuke Niwa wrote: >>> >>> What is the rationale for adding this property on the navigator object >>> >>> instead of the chrome object we also expose if your'e not expecting this >>> property to be ever standarized? >> >> >> Generally, we avoid implementing web visible features via the "chrome" >> object because that makes them Chrome-proprietary. In this case, it >> seems entirely reasonable for other browsers (e.g., Firefox) to want >> to implement this feature. By putting it on navigator, we invite them >> to implement it as well. > > > No, that's not how you invite someone to do something, in conventional > human-to-human interactions :-|. It's a bit aggro -- if it makes a de-facto > standard, others will follow whether they like it or not. If it has > undesirable unintended consequences, too bad. As Dean pointed out, it also > contradicts the pitch in Annie's first message. > > As Annie noted, we don't feel the need since our UA string has some [ab] > cruft in it at the end, and a number after the letter that's important. > > But UA groveling sucks. Could we instead discuss navigator.buildType in some > appropriate W3C or WHATWG list? Thanks everybody for your feedback! It sounds like we need to think a bit more about the use case and whether there's a better approach we can take. We'll send an updated proposal to the whatwg list after we've thought about it a bit more. -Annie ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
Adam Barth wrote: On Thu, Jun 7, 2012 at 1:00 PM, Ryosuke Niwa wrote: What is the rationale for adding this property on the navigator object instead of the chrome object we also expose if your'e not expecting this property to be ever standarized? Generally, we avoid implementing web visible features via the "chrome" object because that makes them Chrome-proprietary. In this case, it seems entirely reasonable for other browsers (e.g., Firefox) to want to implement this feature. By putting it on navigator, we invite them to implement it as well. No, that's not how you invite someone to do something, in conventional human-to-human interactions :-|. It's a bit aggro -- if it makes a de-facto standard, others will follow whether they like it or not. If it has undesirable unintended consequences, too bad. As Dean pointed out, it also contradicts the pitch in Annie's first message. As Annie noted, we don't feel the need since our UA string has some [ab] cruft in it at the end, and a number after the letter that's important. But UA groveling sucks. Could we instead discuss navigator.buildType in some appropriate W3C or WHATWG list? /be ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On 07/06/2012, at 1:10 PM, Adam Barth wrote: > On Thu, Jun 7, 2012 at 1:00 PM, Ryosuke Niwa wrote: >> On Wed, Jun 6, 2012 at 1:51 PM, Annie Sullivan >> wrote: >>> >>> I wanted to let you know that I plan to add support for >>> navigator.buildType (e.g., "nightly", "beta", "final") to WebKit. This >>> feature isn't on the standards track (but neither are a bunch of other >>> similar properties on Navigator). This feature will be behind the >>> ENABLE(NAVIGATOR_BUILDTYPE) feature define. See: >>> https://bugs.webkit.org/show_bug.cgi?id=88358 >>> http://html.spec.whatwg.org/#navigator >> >> What is the rationale for adding this property on the navigator object >> instead of the chrome object we also expose if your'e not expecting this >> property to be ever standarized? > > Generally, we avoid implementing web visible features via the "chrome" > object because that makes them Chrome-proprietary. In this case, it > seems entirely reasonable for other browsers (e.g., Firefox) to want > to implement this feature. By putting it on navigator, we invite them > to implement it as well. But the original message said: > I wanted to let you know that I plan to add support for navigator.buildType > (e.g., "nightly", "beta", "final") to WebKit. This feature isn't on the > standards track (but neither are a bunch of other similar properties on > Navigator) If you don't want it to be Chrome or WebKit-proprietary, and you're inviting other browsers to implement it, then you should probably speak to them and the rest of the community up front. It definitely would be much nicer if User Agent was exposed as a bunch of neatly organised JavaScript properties on the navigator object. Of course then we'd eventually have issues similar to what are now parsing errors. Who says a browser won't want to add "alpha" or "release-candidate" to the list above. This is going to be flakey no matter what. I should stop replying now because I really don't care anywhere near as much about this as I'm suggesting by all my email :) Dean > >> The feedback the WebKit community at large has given us so far appears to be >> strictly negative about adding this to the navigator object. > > The mechanism for implementing the feature doesn't really affect the > arguments that folks are making on this thread. If we decide that the > feature is worth implementing, we should implement it on navigator. > If the feature is not worth implementing, we shouldn't implement it on > the "chrome" object either. > > Adam > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On 07/06/2012, at 12:53 PM, Annie Sullivan wrote: > On Thu, Jun 7, 2012 at 3:43 PM, Dean Jackson wrote: >> >> On 07/06/2012, at 12:05 PM, Annie Sullivan wrote: >> >>> In many browsers in the past, it's been >>> pretty easy to determine from "a" and "b" characters in the user agent >>> of many browsers which builds are "alpha" and "beta", and I haven't >>> heard of bugs caused specifically by checking for build type there. >> >> So why not just do that then? > > While it's nice that web developers don't seem to be using the build > type info in the user agent string in their code, user agent parsing > code is still very brittle. Some browsers, like Firefox, have had > buildtype characters in the user agent string for many years, so > parsing code can handle things like "Firefox/14.0a2". But Chrome > hasn't ever changed its version format, so we're worried about > breaking user agent parsers. Right, I understand that issue. But I don't think replacing something flakey and problematic with something that could be equally flakey and problematic is a big win. Your original problem was: > We'd love for these sites to be able to report regressions they see in > Chrome's performance as early as possible. But it turns out users on > different channels actually show different performance characteristics. Beta > users seem to have faster machines, for example. So in order to compare two > versions of Chrome, we need to compare data from users on the same release > channel. So we'd like sites who collect performance information to be able to > collect the build type in order to do that comparison. That sounds exactly like User Agent detection to me. You want to detect build version and type. I still think there are similarities to prefixing. The Web community (not just WebKit) is making a lot of noise about how being able to detect browsers might seem like a good idea at first but ends up causing longer-term headaches. By the way, I don't feel strongly about this. I'm just pointing out that I don't see any benefit and that what looks like a small change in metadata has just as important consequences as a significant technical change. Dean ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Thu, Jun 7, 2012 at 1:00 PM, Ryosuke Niwa wrote: > On Wed, Jun 6, 2012 at 1:51 PM, Annie Sullivan > wrote: >> >> I wanted to let you know that I plan to add support for >> navigator.buildType (e.g., "nightly", "beta", "final") to WebKit. This >> feature isn't on the standards track (but neither are a bunch of other >> similar properties on Navigator). This feature will be behind the >> ENABLE(NAVIGATOR_BUILDTYPE) feature define. See: >> https://bugs.webkit.org/show_bug.cgi?id=88358 >> http://html.spec.whatwg.org/#navigator > > What is the rationale for adding this property on the navigator object > instead of the chrome object we also expose if your'e not expecting this > property to be ever standarized? Generally, we avoid implementing web visible features via the "chrome" object because that makes them Chrome-proprietary. In this case, it seems entirely reasonable for other browsers (e.g., Firefox) to want to implement this feature. By putting it on navigator, we invite them to implement it as well. > The feedback the WebKit community at large has given us so far appears to be > strictly negative about adding this to the navigator object. The mechanism for implementing the feature doesn't really affect the arguments that folks are making on this thread. If we decide that the feature is worth implementing, we should implement it on navigator. If the feature is not worth implementing, we shouldn't implement it on the "chrome" object either. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
On Thu, Jun 7, 2012 at 12:59 PM, Dirk Pranke wrote: > > FAIL was supposed to have been removed long ago in favor of > TEXT/IMAGE/IMAGE+TEXT as necessary as Peter suggests, but I never got > around to it :) I agree with his reasoning, and have not been > convinced that not having to be specific about what is failing is a > good thing. > Let's do that then. I also agree that we shouldn't TEXT/IMAGE/IMAGE+TEXT in the file at > all for extended periods of time since you can just check in updated > baselines. > I'd argue that we should never add test expectations pre-emptively just to keep bots green. It appears to do more harm than good. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Wed, Jun 6, 2012 at 1:51 PM, Annie Sullivan wrote: > I wanted to let you know that I plan to add support for > navigator.buildType (e.g., "nightly", "beta", "final") to WebKit. This > feature isn't on the standards track (but neither are a bunch of other > similar properties on Navigator). This feature will be behind the > ENABLE(NAVIGATOR_BUILDTYPE) feature define. See: > https://bugs.webkit.org/show_bug.cgi?id=88358 > http://html.spec.whatwg.org/#navigator > What is the rationale for adding this property on the navigator object instead of the chrome object we also expose if your'e not expecting this property to be ever standarized? The feedback the WebKit community at large has given us so far appears to be strictly negative about adding this to the navigator object. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
On Wed, Jun 6, 2012 at 10:36 PM, Peter Kasting wrote: > On Wed, Jun 6, 2012 at 10:22 PM, Ryosuke Niwa wrote: >> >> Now that everyone knows the problem, I propose to rename FAIL to DIFF. >> >> FAIL should mean that the test fails, not that it fails with image, text, >> or image and text failures. >> >> DIFF, on the other hand, has no ambiguity. It can't be interpreted as >> timeout, crash, or pass but can easily be associated with image and text >> differences. > > > I don't think DIFF is any better. It sounds like it means the output is > "different than what we wanted", thus it effectively means "didn't pass", > and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would > expect FAIL to. > > Personally I'd prefer to resolve this -- if we need to -- by removing FAIL > entirely. Being explicit about your expectations isn't a bad thing. Plus, > the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems likely to be > small. FAIL was supposed to have been removed long ago in favor of TEXT/IMAGE/IMAGE+TEXT as necessary as Peter suggests, but I never got around to it :) I agree with his reasoning, and have not been convinced that not having to be specific about what is failing is a good thing. I also agree that we shouldn't TEXT/IMAGE/IMAGE+TEXT in the file at all for extended periods of time since you can just check in updated baselines. -- Dirk ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Thu, Jun 7, 2012 at 12:53 PM, Annie Sullivan wrote: > On Thu, Jun 7, 2012 at 3:43 PM, Dean Jackson wrote: >> On 07/06/2012, at 12:05 PM, Annie Sullivan wrote: >>> In many browsers in the past, it's been >>> pretty easy to determine from "a" and "b" characters in the user agent >>> of many browsers which builds are "alpha" and "beta", and I haven't >>> heard of bugs caused specifically by checking for build type there. >> >> So why not just do that then? > > While it's nice that web developers don't seem to be using the build > type info in the user agent string in their code, user agent parsing > code is still very brittle. Some browsers, like Firefox, have had > buildtype characters in the user agent string for many years, so > parsing code can handle things like "Firefox/14.0a2". But Chrome > hasn't ever changed its version format, so we're worried about > breaking user agent parsers. Even beyond that, putting the buildType in the User-Agent seems strictly worse that exposing it as a separate property to JavaScript. Adam ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Thu, Jun 7, 2012 at 3:43 PM, Dean Jackson wrote: > > On 07/06/2012, at 12:05 PM, Annie Sullivan wrote: > >> In many browsers in the past, it's been >> pretty easy to determine from "a" and "b" characters in the user agent >> of many browsers which builds are "alpha" and "beta", and I haven't >> heard of bugs caused specifically by checking for build type there. > > So why not just do that then? While it's nice that web developers don't seem to be using the build type info in the user agent string in their code, user agent parsing code is still very brittle. Some browsers, like Firefox, have had buildtype characters in the user agent string for many years, so parsing code can handle things like "Firefox/14.0a2". But Chrome hasn't ever changed its version format, so we're worried about breaking user agent parsers. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
On Thu, Jun 7, 2012 at 12:44 PM, Peter Kasting wrote: > On Thu, Jun 7, 2012 at 12:33 PM, Ryosuke Niwa wrote: > >> Not if the test was padding. I'm talking about the case where you're >> modifying WebCore and know that some tests are going to need rebaselines. >> People have advised in the past that patch authors add failing test >> expectations to TestExpectations files to avoid turning bots red. >> > > I think this is bad advice. When I've been sheriff it seems like people > who try this inevitably miss some tests and platforms anyway, get wrong > expectations, etc. All this does is add more work for the submitter that > is difficult to check ahead of time. We should just advise people to land > and then fix (and be around on IRC/notify sheriffs about what's going on). > Yes. I'd strongly advocate for not adding test expectations. I've seen too many patch authors adding test expectations and then forgetting about them. However, there is a practical problem that the commit queue uses Chromium Linux port and rejects patches that need rebaselines unless the authors add test expectations. It seems like this is what you were saying as your general statement as > well. > Right. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
On Thu, Jun 7, 2012 at 12:33 PM, Ryosuke Niwa wrote: > Not if the test was padding. I'm talking about the case where you're > modifying WebCore and know that some tests are going to need rebaselines. > People have advised in the past that patch authors add failing test > expectations to TestExpectations files to avoid turning bots red. > I think this is bad advice. When I've been sheriff it seems like people who try this inevitably miss some tests and platforms anyway, get wrong expectations, etc. All this does is add more work for the submitter that is difficult to check ahead of time. We should just advise people to land and then fix (and be around on IRC/notify sheriffs about what's going on). It seems like this is what you were saying as your general statement as well. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On 07/06/2012, at 12:05 PM, Annie Sullivan wrote: > In many browsers in the past, it's been > pretty easy to determine from "a" and "b" characters in the user agent > of many browsers which builds are "alpha" and "beta", and I haven't > heard of bugs caused specifically by checking for build type there. So why not just do that then? Dean ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT
The reason for this (which is debatable) is that CRASH and TIMEOUT are deemed to be more serious and shouldn't be suppressed as lightly. MISSING, on the other hand, just indicates that there's something wrong (tests should never have missing results for very long). -- Dirk On Wed, Jun 6, 2012 at 9:42 PM, Ryosuke Niwa wrote: > Hi, > > As much as it doesn't make any sense to me, FAIL test expectation doesn't > cover MISSING, CRASH, and TIMEOUT test expectations as it is currently > interpreted by new-run-webkit-tests. Meaning that > > BUGRNIWA : some-pixel-test-i-am-adding.html = FAIL > > is logically equivalent to > > BUGRNIWA : some-pixel-test-i-am-adding.html = IMAGE TEXT IMAGE+TEXT > > Thus, if you're adding a pixel test and only adding the corresponding > expected result to, say, Mac, then you need to add MISSING results to some > platforms (e.g. Qt, GTK+, etc...) that cannot find this expected result as > in: > > BUGRNIWA : some-pixel-test-i-am-adding.html = MISSING > > If you're expecting the test to fail with image, text, or image and text > failures, or pass, then do: > > BUGRNIWA : some-pixel-test-i-am-adding.html = FAIL PASS > > If you're expecting the test to timeout, crash, or, fail (image, text, or > image and text), or pass, then do: > > BUGRNIWA : some-pixel-test-i-am-adding.html = TIMEOUT CRASH FAIL PASS > > Best, > Ryosuke Niwa > Software Engineer > Google Inc. > > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
On Thu, Jun 7, 2012 at 11:03 AM, Peter Kasting wrote: > On Wed, Jun 6, 2012 at 11:28 PM, Ryosuke Niwa wrote: > >> > I don't think DIFF is any better. It sounds like it means the output >> is "different than what we wanted", thus it effectively means "didn't >> pass", and one would expect it to match MISSING/CRASH/TIMEOUT as much as >> one would expect FAIL to. >> >> The output being different implies that we have an output, which isn't >> true when DRT/WTR crashes or times out. >> > Those are outputs as well. Or if you don't like the work "output" > substitute "outcome". > I don't buy it. > > Personally I'd prefer to resolve this -- if we need to -- by removing >> FAIL entirely. Being explicit about your expectations isn't a bad thing. >> Plus, the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems >> likely to be small. >> >> People use FAIL when they don't know what to expect; e.g. adding or >> rebaselining tests. zit's utterky unreasonable to expect patch authors to >> add TEXT IMAGE TEXT+IMAGE to every test they're expecting to rebaseline. >> > If you're rebaselining an existing test, presumably the test already has > an expectation that matches what it's doing, or you can see what it's doing > to write one. > Not if the test was padding. I'm talking about the case where you're modifying WebCore and know that some tests are going to need rebaselines. People have advised in the past that patch authors add failing test expectations to TestExpectations files to avoid turning bots red. If the rebaselining tools can already figure out the right behavior when an > expectation is FAIL, why don't we just make them work correctly regardless > of what the expectation says? Or work correctly when the expectation is > the empty string? Those would let people just write "foo.html =" and > rebaseline which is even easier on them. > That isn't really the issue. The problem is that people add test expectations pre-emptively to keep bots green but end up adding wrong expectations sometimes because FAIL doesn't encompass MISSING, and that's the failure you'll get on qt and gtk if you're adding expected results to mac. If you're adding expected results to just chromium linux, for example, then you'll get MISSING failure on all but chromium linux. > I also think it's a bad practice to add test expections just to keep >> bots green. It's much better if the tests started to fail on the waterfall >> so that people who pay attention to bots can rebaseline them since most >> people forget about rebaselining tests once their patches are landed. >> > I can't tell what you're arguing here. As far as I know nothing in what I > was saying related to the question of the reason why people are adding > expectations. > It was nothing to do with your response. I was making a general statement. - Ryosuke ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Thu, Jun 7, 2012 at 2:08 PM, Dean Jackson wrote: > > On 07/06/2012, at 10:07 AM, Annie Sullivan wrote: > >> Oops, forgot to reply-all. >> >> On Thu, Jun 7, 2012 at 9:05 AM, Annie Sullivan wrote: >>> On Wed, Jun 6, 2012 at 4:59 PM, Darin Adler wrote: Our past experience with this sort of thing at Apple is that it led to bugs we didn’t find until after we shipped “final” software because they did not show up during earlier testing. Since then, we’ve gone out of our way to avoid differences, since one of the main things we want to do with non-final builds is to approximate the final releases as closely as possible to get useful testing. To oversimplify, website developers notoriously make mistakes with these types of attributes doing things like version and OS checks, leading to website incompatibilities. >>> >>> Yes, this is definitely a concern we have as well. >>> Maybe you could make your case for the usefulness of the attribute? >>> >>> The problem we're experiencing in Chrome is for sites that collect >>> usage data--it turns out there's a selection bias caused by users who >>> opt in to our canary, dev, and beta channels. >>> >>> The specific case I'm looking at right now is sites collecting >>> performance data from their user base. We'd love for these sites to be >>> able to report regressions they see in Chrome's performance as early >>> as possible. But it turns out users on different channels actually >>> show different performance characteristics. Beta users seem to have >>> faster machines, for example. So in order to compare two versions of >>> Chrome, we need to compare data from users on the same release >>> channel. So we'd like sites who collect performance information to be >>> able to collect the build type in order to do that comparison. >>> >>> We considered a few alternatives: >>> 1) Adding a marker in the user agent string that indicates the >>> channel. The problem with this is that there is a lot of very fragile >>> code in the wild which parses user agent strings and breaks at the >>> slightest change. For example, code that parses Opera 10 as Opera 1. >>> 2) Updating the version number as Chrome advances from canary to dev >>> to beta to stable, or encoding the build type in one of the octets of >>> the version number. The problem with this is that there's a big >>> possibility of sites that do version checking accidentally turning off >>> features on some channels. There are also a lot of things that we *do* >>> want to track across release channels for a specific version, like >>> bugs. So changing the version number could cause confusion there. >>> 3) Sending an "x-buildtype" header. If we do this on every request, >>> it's a lot of bloat. We can't restrict it to requests that send a >>> corresponding "x-tell-me-the-buildtype" request header because most >>> metrics collection libraries send their metrics in an image request, >>> so they can't send custom headers. > > navigator.buildType might address your particular problem, but introduces a > significant risk of future issues. I can imagine keen web authors adding > features based on detecting "nightly" or "beta" that then break in "final". > > This is similar to prefixing CSS properties which, as you probably know, is > an extremely hot discussion topic right now in the community. I don't think > people will be especially excited for another potential point of failure. I think this is different from CSS prefixing because of size of the potential user base involved. With CSS prefixing, on mobile phones right now web developers can reach nearly 100% of their audience with only -webkit prefixes. So they have no reason to fix their sites to drop the prefixes. With the buildType, it's pretty clear that web developers are going to reach only a small percentage of their users if they check for "beta" or "nightly". So it doesn't really make sense for them to create a site that depends on this. In many browsers in the past, it's been pretty easy to determine from "a" and "b" characters in the user agent of many browsers which builds are "alpha" and "beta", and I haven't heard of bugs caused specifically by checking for build type there. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On Thu, Jun 7, 2012 at 11:48 AM, Konstantin Tokarev wrote: > > > 07.06.2012, 21:07, "Annie Sullivan" : >> Oops, forgot to reply-all. >> >> On Thu, Jun 7, 2012 at 9:05 AM, Annie Sullivan wrote: >> >>> On Wed, Jun 6, 2012 at 4:59 PM, Darin Adler wrote: Our past experience with this sort of thing at Apple is that it led to bugs we didn't find until after we shipped "final" software because they did not show up during earlier testing. Since then, we've gone out of our way to avoid differences, since one of the main things we want to do with non-final builds is to approximate the final releases as closely as possible to get useful testing. To oversimplify, website developers notoriously make mistakes with these types of attributes doing things like version and OS checks, leading to website incompatibilities. >>> Yes, this is definitely a concern we have as well. Maybe you could make your case for the usefulness of the attribute? >>> The problem we're experiencing in Chrome is for sites that collect >>> usage data--it turns out there's a selection bias caused by users who >>> opt in to our canary, dev, and beta channels. >>> >>> The specific case I'm looking at right now is sites collecting >>> performance data from their user base. We'd love for these sites to be >>> able to report regressions they see in Chrome's performance as early >>> as possible. But it turns out users on different channels actually >>> show different performance characteristics. Beta users seem to have >>> faster machines, for example. > > Are you sure that this differences are statistically significant? Yes. Adam >>> So in order to compare two versions of >>> Chrome, we need to compare data from users on the same release >>> channel. So we'd like sites who collect performance information to be >>> able to collect the build type in order to do that comparison. >>> >>> We considered a few alternatives: >>> 1) Adding a marker in the user agent string that indicates the >>> channel. The problem with this is that there is a lot of very fragile >>> code in the wild which parses user agent strings and breaks at the >>> slightest change. For example, code that parses Opera 10 as Opera 1. >>> 2) Updating the version number as Chrome advances from canary to dev >>> to beta to stable, or encoding the build type in one of the octets of >>> the version number. The problem with this is that there's a big >>> possibility of sites that do version checking accidentally turning off >>> features on some channels. There are also a lot of things that we *do* >>> want to track across release channels for a specific version, like >>> bugs. So changing the version number could cause confusion there. >>> 3) Sending an "x-buildtype" header. If we do this on every request, >>> it's a lot of bloat. We can't restrict it to requests that send a >>> corresponding "x-tell-me-the-buildtype" request header because most >>> metrics collection libraries send their metrics in an image request, >>> so they can't send custom headers. >> >> ___ >> webkit-dev mailing list >> webkit-dev@lists.webkit.org >> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > > -- > Regards, > Konstantin > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
07.06.2012, 21:07, "Annie Sullivan" : > Oops, forgot to reply-all. > > On Thu, Jun 7, 2012 at 9:05 AM, Annie Sullivan wrote: > >> On Wed, Jun 6, 2012 at 4:59 PM, Darin Adler wrote: >>> Our past experience with this sort of thing at Apple is that it led to >>> bugs we didn’t find until after we shipped “final” software because they >>> did not show up during earlier testing. Since then, we’ve gone out of our >>> way to avoid differences, since one of the main things we want to do with >>> non-final builds is to approximate the final releases as closely as >>> possible to get useful testing. >>> >>> To oversimplify, website developers notoriously make mistakes with these >>> types of attributes doing things like version and OS checks, leading to >>> website incompatibilities. >> Yes, this is definitely a concern we have as well. >>> Maybe you could make your case for the usefulness of the attribute? >> The problem we're experiencing in Chrome is for sites that collect >> usage data--it turns out there's a selection bias caused by users who >> opt in to our canary, dev, and beta channels. >> >> The specific case I'm looking at right now is sites collecting >> performance data from their user base. We'd love for these sites to be >> able to report regressions they see in Chrome's performance as early >> as possible. But it turns out users on different channels actually >> show different performance characteristics. Beta users seem to have >> faster machines, for example. Are you sure that this differences are statistically significant? >> So in order to compare two versions of >> Chrome, we need to compare data from users on the same release >> channel. So we'd like sites who collect performance information to be >> able to collect the build type in order to do that comparison. >> >> We considered a few alternatives: >> 1) Adding a marker in the user agent string that indicates the >> channel. The problem with this is that there is a lot of very fragile >> code in the wild which parses user agent strings and breaks at the >> slightest change. For example, code that parses Opera 10 as Opera 1. >> 2) Updating the version number as Chrome advances from canary to dev >> to beta to stable, or encoding the build type in one of the octets of >> the version number. The problem with this is that there's a big >> possibility of sites that do version checking accidentally turning off >> features on some channels. There are also a lot of things that we *do* >> want to track across release channels for a specific version, like >> bugs. So changing the version number could cause confusion there. >> 3) Sending an "x-buildtype" header. If we do this on every request, >> it's a lot of bloat. We can't restrict it to requests that send a >> corresponding "x-tell-me-the-buildtype" request header because most >> metrics collection libraries send their metrics in an image request, >> so they can't send custom headers. > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev -- Regards, Konstantin ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
On 07/06/2012, at 10:07 AM, Annie Sullivan wrote: > Oops, forgot to reply-all. > > On Thu, Jun 7, 2012 at 9:05 AM, Annie Sullivan wrote: >> On Wed, Jun 6, 2012 at 4:59 PM, Darin Adler wrote: >>> >>> Our past experience with this sort of thing at Apple is that it led to bugs >>> we didn’t find until after we shipped “final” software because they did not >>> show up during earlier testing. Since then, we’ve gone out of our way to >>> avoid differences, since one of the main things we want to do with >>> non-final builds is to approximate the final releases as closely as >>> possible to get useful testing. >>> >>> To oversimplify, website developers notoriously make mistakes with these >>> types of attributes doing things like version and OS checks, leading to >>> website incompatibilities. >> >> Yes, this is definitely a concern we have as well. >> >>> Maybe you could make your case for the usefulness of the attribute? >> >> The problem we're experiencing in Chrome is for sites that collect >> usage data--it turns out there's a selection bias caused by users who >> opt in to our canary, dev, and beta channels. >> >> The specific case I'm looking at right now is sites collecting >> performance data from their user base. We'd love for these sites to be >> able to report regressions they see in Chrome's performance as early >> as possible. But it turns out users on different channels actually >> show different performance characteristics. Beta users seem to have >> faster machines, for example. So in order to compare two versions of >> Chrome, we need to compare data from users on the same release >> channel. So we'd like sites who collect performance information to be >> able to collect the build type in order to do that comparison. >> >> We considered a few alternatives: >> 1) Adding a marker in the user agent string that indicates the >> channel. The problem with this is that there is a lot of very fragile >> code in the wild which parses user agent strings and breaks at the >> slightest change. For example, code that parses Opera 10 as Opera 1. >> 2) Updating the version number as Chrome advances from canary to dev >> to beta to stable, or encoding the build type in one of the octets of >> the version number. The problem with this is that there's a big >> possibility of sites that do version checking accidentally turning off >> features on some channels. There are also a lot of things that we *do* >> want to track across release channels for a specific version, like >> bugs. So changing the version number could cause confusion there. >> 3) Sending an "x-buildtype" header. If we do this on every request, >> it's a lot of bloat. We can't restrict it to requests that send a >> corresponding "x-tell-me-the-buildtype" request header because most >> metrics collection libraries send their metrics in an image request, >> so they can't send custom headers. navigator.buildType might address your particular problem, but introduces a significant risk of future issues. I can imagine keen web authors adding features based on detecting "nightly" or "beta" that then break in "final". This is similar to prefixing CSS properties which, as you probably know, is an extremely hot discussion topic right now in the community. I don't think people will be especially excited for another potential point of failure. Dean ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Rename FAIL to DIFF Was (Re: PSA: FAIL test expectation does not encompass MISSING, CRASH, or TIMEOUT)
On Wed, Jun 6, 2012 at 11:28 PM, Ryosuke Niwa wrote: > > I don't think DIFF is any better. It sounds like it means the output is > "different than what we wanted", thus it effectively means "didn't pass", > and one would expect it to match MISSING/CRASH/TIMEOUT as much as one would > expect FAIL to. > > The output being different implies that we have an output, which isn't > true when DRT/WTR crashes or times out. > Those are outputs as well. Or if you don't like the work "output" substitute "outcome". > > Personally I'd prefer to resolve this -- if we need to -- by removing > FAIL entirely. Being explicit about your expectations isn't a bad thing. > Plus, the number of cases that are truly TEXT IMAGE IMAGE+TEXT seems > likely to be small. > > People use FAIL when they don't know what to expect; e.g. adding or > rebaselining tests. zit's utterky unreasonable to expect patch authors to > add TEXT IMAGE TEXT+IMAGE to every test they're expecting to rebaseline. > If you're rebaselining an existing test, presumably the test already has an expectation that matches what it's doing, or you can see what it's doing to write one. If the rebaselining tools can already figure out the right behavior when an expectation is FAIL, why don't we just make them work correctly regardless of what the expectation says? Or work correctly when the expectation is the empty string? Those would let people just write "foo.html =" and rebaseline which is even easier on them. > I also think it's a bad practice to add test expections just to keep bots > green. It's much better if the tests started to fail on the waterfall so > that people who pay attention to bots can rebaseline them since most people > forget about rebaselining tests once their patches are landed. > I can't tell what you're arguing here. As far as I know nothing in what I was saying related to the question of the reason why people are adding expectations. PK ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] Adding ENABLE_NAVIGATOR_BUILDTYPE to WebCore
Oops, forgot to reply-all. On Thu, Jun 7, 2012 at 9:05 AM, Annie Sullivan wrote: > On Wed, Jun 6, 2012 at 4:59 PM, Darin Adler wrote: >> >> Our past experience with this sort of thing at Apple is that it led to bugs >> we didn’t find until after we shipped “final” software because they did not >> show up during earlier testing. Since then, we’ve gone out of our way to >> avoid differences, since one of the main things we want to do with non-final >> builds is to approximate the final releases as closely as possible to get >> useful testing. >> >> To oversimplify, website developers notoriously make mistakes with these >> types of attributes doing things like version and OS checks, leading to >> website incompatibilities. > > Yes, this is definitely a concern we have as well. > >> Maybe you could make your case for the usefulness of the attribute? > > The problem we're experiencing in Chrome is for sites that collect > usage data--it turns out there's a selection bias caused by users who > opt in to our canary, dev, and beta channels. > > The specific case I'm looking at right now is sites collecting > performance data from their user base. We'd love for these sites to be > able to report regressions they see in Chrome's performance as early > as possible. But it turns out users on different channels actually > show different performance characteristics. Beta users seem to have > faster machines, for example. So in order to compare two versions of > Chrome, we need to compare data from users on the same release > channel. So we'd like sites who collect performance information to be > able to collect the build type in order to do that comparison. > > We considered a few alternatives: > 1) Adding a marker in the user agent string that indicates the > channel. The problem with this is that there is a lot of very fragile > code in the wild which parses user agent strings and breaks at the > slightest change. For example, code that parses Opera 10 as Opera 1. > 2) Updating the version number as Chrome advances from canary to dev > to beta to stable, or encoding the build type in one of the octets of > the version number. The problem with this is that there's a big > possibility of sites that do version checking accidentally turning off > features on some channels. There are also a lot of things that we *do* > want to track across release channels for a specific version, like > bugs. So changing the version number could cause confusion there. > 3) Sending an "x-buildtype" header. If we do this on every request, > it's a lot of bloat. We can't restrict it to requests that send a > corresponding "x-tell-me-the-buildtype" request header because most > metrics collection libraries send their metrics in an image request, > so they can't send custom headers. ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
Re: [webkit-dev] EditBugs permissions for bugs.webkit.org account
Done. On Thu, Jun 7, 2012 at 8:35 AM, Simon Pena wrote: > Hi, > > I'd like to request EditBugs permissions to be added to my > bugs.webkit.org account (spena at igalia.com) > > I'm not a committer, but I have worked on several patches already, > mainly for the GTK port. > > Thanks in advance, > > -- > Simon Pena > Igalia - Free Software Engineering > > > ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
[webkit-dev] EditBugs permissions for bugs.webkit.org account
Hi, I'd like to request EditBugs permissions to be added to my bugs.webkit.org account (spena at igalia.com) I'm not a committer, but I have worked on several patches already, mainly for the GTK port. Thanks in advance, -- Simon Pena Igalia - Free Software Engineering signature.asc Description: OpenPGP digital signature ___ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
