Re: [webkit-dev] Standard process for exporting new WPT tests?
> However, one thing I really like about your order is that it makes it more > straightforward to make WPT repo failures block browser repo merging. In > the WPT Travis job, we already detect flakiness for Chrome and Firefox and > I think we should also detect harness errors, and eventually also flag when > a test goes from passing everywhere to failing everywhere, which is > probably a mistake. We haven't really figured out how to make that block > Chromium's CQ yet, and maybe flipping the order would do it. > Exactly, WPT infrastructure has some really nice sanity checks that we do not have in WebKit and it sounds really great to benefit from it. That said, this will put some burden on WPT infrastructure like being fast enough to not slow down our own process. We should also probably run as much as possible the WPT sanity checks locally. We are now checking WPT linter as you suggested a while back. Maybe there are some other WPT checks that could be run in the WebKit context. ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Standard process for exporting new WPT tests?
> Interesting. What happens happens if the WebKit patch then fails to land > in WebKit, perhaps because some bot fails the test, a conflict, or anything > else? Is resolving that a manual affair? > If patch lands in WPT and not in WebKit due to conflicts, that is fine. One will need to resolve the conflict manually as done for other conflicts. > > When imports are automated and much more frequent, I take it the upside is > that you never need to reapply still-being-exported patches like we do in > Blink. But what about the above case, when the change hasn't been applied > in WebKit yet? Seems like the importer instead needs to *revert* the > in-flight change? > I guess the question is if the change is made in WPT, not yet in WebKit and we are reimporting in the middle. In that case, we will probably have the new test failing, timing out or crashing. This will get fixed once the patch lands in WebKit (after conflict resolution). ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Standard process for exporting new WPT tests?
Current maintainer of Chromium's WPT sync chiming in: Because of the lack of atomic operations across two separate repos, some kind of transient inconsistency and race conditions are unavoidable. FWIW, when Chromium's WPT sync was originally designed, there was concern about increasing the wait time of landing a Chromium change, which partly led to our design choice of landing the Chromium change first and then the GitHub export PR. One important mechanism to avoid losing the Chromium change (because of being overwritten by the next import) that hasn't been mentioned is that *we re-apply landed-in-Chromium-but-not-merged-on-Github patches during import*. Of course, this mechanism caused quite some extra complexities, but has been working well for us. The approach being taken by the WebKit community (merging GitHub PRs before landing WebKit patches) is sound and should be simpler. And I'm very excited to see more contributions to WPT! On Fri, May 25, 2018 at 4:16 AM Philip Jägenstedt wrote: > On Fri, May 25, 2018, 00:52 Ryosuke Niwa wrote: > >> >> On Thu, May 24, 2018 at 3:22 AM, Philip Jägenstedt >> wrote: >> >>> On Wed, May 23, 2018, 23:43 youenn fablet wrote: >>> Le mer. 23 mai 2018 à 14:11, Frédéric Wang a écrit : > On 23/05/2018 22:50, Ryosuke Niwa wrote: > > As we have preciously discussed, we should NEVER commit new tests > into > > LayoutTests/imported/w3c/web-platform-tests. > Ryosuke, correct me if I am wrong, I think you are pointing out the following rule: Changes to LayoutTests/imported/w3c/web-platform-tests tests should land first in WPT repository, then in WebKit repository. >>> >>> Oh, that is surprising. >>> >>> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit >>> export, and https://trac.webkit.org/changeset/231788/webkit did modify >>> the test in place. Do you mean that the WPT PR was merged first, or should >>> be in general? Chromium and Gecko do it in the other order, and I'd be >>> interested to understand the trade-offs of flipping the order. >>> >> >> Yes, it's okay for a WebKit commit to merge the change which got merged >> into WPT but it's never okay to first commit the test into WebKit and then >> later upstream it to WPT... >> >> Any process like this where changes end up in WebKit trunk via anything >>> except a full WPT import will mean that divergence is possible. >>> >> >> Precisely to avoid these problems. >> > > I'm saying that there's temporary divergence with the WPT-then-WebKit > order as well. > > Whichever side is merged first, the other side can fail to merge because > any number of reasons, and something/someone then needs to deal with that. > (See question in reply to Youenn.) > > However, one thing I really like about your order is that it makes it more > straightforward to make WPT repo failures block browser repo merging. In > the WPT Travis job, we already detect flakiness for Chrome and Firefox and > I think we should also detect harness errors, and eventually also flag when > a test goes from passing everywhere to failing everywhere, which is > probably a mistake. We haven't really figured out how to make that block > Chromium's CQ yet, and maybe flipping the order would do it. > >> ___ > webkit-dev mailing list > webkit-dev@lists.webkit.org > https://lists.webkit.org/mailman/listinfo/webkit-dev > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
Re: [webkit-dev] Standard process for exporting new WPT tests?
On Fri, May 25, 2018, 00:52 Ryosuke Niwa wrote: > > On Thu, May 24, 2018 at 3:22 AM, Philip Jägenstedt > wrote: > >> On Wed, May 23, 2018, 23:43 youenn fablet wrote: >> >>> Le mer. 23 mai 2018 à 14:11, Frédéric Wang a écrit : >>> On 23/05/2018 22:50, Ryosuke Niwa wrote: > As we have preciously discussed, we should NEVER commit new tests into > LayoutTests/imported/w3c/web-platform-tests. >>> >>> Ryosuke, correct me if I am wrong, I think you are pointing out the >>> following rule: >>> Changes to LayoutTests/imported/w3c/web-platform-tests tests should land >>> first in WPT repository, then in WebKit repository. >>> >> >> Oh, that is surprising. >> >> https://github.com/w3c/web-platform-tests/pull/10964 is a recent WebKit >> export, and https://trac.webkit.org/changeset/231788/webkit did modify >> the test in place. Do you mean that the WPT PR was merged first, or should >> be in general? Chromium and Gecko do it in the other order, and I'd be >> interested to understand the trade-offs of flipping the order. >> > > Yes, it's okay for a WebKit commit to merge the change which got merged > into WPT but it's never okay to first commit the test into WebKit and then > later upstream it to WPT... > > Any process like this where changes end up in WebKit trunk via anything >> except a full WPT import will mean that divergence is possible. >> > > Precisely to avoid these problems. > I'm saying that there's temporary divergence with the WPT-then-WebKit order as well. Whichever side is merged first, the other side can fail to merge because any number of reasons, and something/someone then needs to deal with that. (See question in reply to Youenn.) However, one thing I really like about your order is that it makes it more straightforward to make WPT repo failures block browser repo merging. In the WPT Travis job, we already detect flakiness for Chrome and Firefox and I think we should also detect harness errors, and eventually also flag when a test goes from passing everywhere to failing everywhere, which is probably a mistake. We haven't really figured out how to make that block Chromium's CQ yet, and maybe flipping the order would do it. > ___ webkit-dev mailing list webkit-dev@lists.webkit.org https://lists.webkit.org/mailman/listinfo/webkit-dev
