Greetings, people of Chromium! Last quarter, the Layout Test Task Force done some pretty good work. I bragged about it in a separate email. Now it's time to grab the bull by the horns and kick it up a notch. Isn't idiomatic English great?
This quarter, the LTTF is aiming right at the heart of the problem: eliminating the need to continually roll WebKit DEPS, baking in layers upon layers of regressions in the phyllo dough that is our test_expectations file. To do that, we must move all of our layout test infrastructure, from test_shell to test expectations to flakiness dashboard upstream. Based on a few hallway/online/cross-continental conversations, our plan of action consists of three efforts (drivers in parentheses): - Port test_shell into DumpRenderTree (DRT) upstream (Tokyo LTTF team + dglazkov + darin) - Take care of chromium dependencies: base, net, skia (src/skia, that is). How will they manifest themselves upstream to avoid breakage due to changes in chromium tree? (dglazkov, darin, tc) - Identify strategy for DRT and V8 coexistence: currently all of DRT is heavily JSC-dependent. We should either introduce script-independent abstractions or convert to NPAPI that we use downstream (dglazkov, darin, tc). - Figure out for inflight testing. How will we ensure that porting doesn't introduce new bugs? Need to come up with a way to have a workable DRT quickly, and a way to produce differential of layout test failures to quickly find porting bugs. - Upstream src/webkit/tools/test_shell to WebKitTools/DumpRenderTree/chromium. - Coordinate with harness upstreaming effort, so that both run-webkit-tests and the new DRT work well together while porting. - Determine how we will remove code downstream. We won't need DRT-specific code, but may still need test_shell as a minimum-capabilities embedder of WebKit. Or we could go with a MiniBrowser-style project (open-source sample for WebKit Apple port) upstream. - Drive effort to completion. *Completion is*: DRT is ready to run layout tests on build.webkit.org, downstream cleaned up and ready to stop running layout tests on build.chromium.org. - Upstream test infrastructure (watch detailed plan and assignments develop at http://dev.chromium.org/developers/design-documents/upstreaminglayouttests). Steps include: - Move run_webkit_tests.py to live upstream - While DRT is being upstreamed, configure to run upstreamed run_webkit_tests on the Chromium builders - Move test expectations upstream - Move ancilliary tools, like rebaseline.py and flakiness dashboard upstream - Modify run_webkit_tests to use the newly upstreamed DRT - Drive effort to completion. *Completion is*: build.webkit.org is running layout tests with no regressions from downstream, build.chromium.org no longer runs layout tests. - Continue fixing/deflakifying layout tests and improving the process for sustaining compatibility. - Fix chrome-specific failures that dramatically affect compatibility (bug triage -- which means *you*!) - Fix SVG/Skia bugs that cause layout test failures (Waterloo team) - Fix V8 bugs and develop ES5 features that affect layout tests (V8 team) - Make V8 bindings more tolerant to IDL/JSC changes. Watch the effort as a bug tree: https://bugs.webkit.org/showdependencytree.cgi?id=32630&hide_resolved=0 (japhet, kkanetkar) - Improve gardening tools and process (dglazkov) - Eliminate layout test flakiness (jparent, ojan, dglazkov) - Drive effort to completion. *Completion is*: <10 sustained layout test failures or flakes, bindings-related regressions/breaks are limited to custom code only, new process for keeping up for layout test regressions is adopted by Webkit Gardeners. As you can see, it's a bit different from "grab a test and fix it" approach we took in Q4. We learn, right? :) Despite most tasks having a driver in one form or another, we always welcome your contributions. Every little bit counts. :DG<
-- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev