Greetings, people of Chromium!
Last quarter, the Layout Test Task Force done some pretty good work. I
bragged about it in a separate email. Now it's time to grab the bull by the
horns and kick it up a notch. Isn't idiomatic English great?
This quarter, the LTTF is aiming right at the heart of the problem:
eliminating the need to continually roll WebKit DEPS, baking in layers upon
layers of regressions in the phyllo dough that is our test_expectations
file. To do that, we must move all of our layout test infrastructure, from
test_shell to test expectations to flakiness dashboard upstream.
Based on a few hallway/online/cross-continental conversations, our plan of
action consists of three efforts (drivers in parentheses):
- Port test_shell into DumpRenderTree (DRT) upstream (Tokyo LTTF team +
dglazkov + darin)
- Take care of chromium dependencies: base, net, skia (src/skia, that
is). How will they manifest themselves upstream to avoid breakage due to
changes in chromium tree? (dglazkov, darin, tc)
- Identify strategy for DRT and V8 coexistence: currently all of DRT is
heavily JSC-dependent. We should either introduce script-independent
abstractions or convert to NPAPI that we use downstream (dglazkov, darin,
tc).
- Figure out for inflight testing. How will we ensure that porting doesn't
introduce new bugs? Need to come up with a way to have a workable DRT
quickly, and a way to produce differential of layout test failures to
quickly find porting bugs.
- Upstream src/webkit/tools/test_shell to
WebKitTools/DumpRenderTree/chromium.
- Coordinate with harness upstreaming effort, so that both
run-webkit-tests and the new DRT work well together while porting.
- Determine how we will remove code downstream. We won't need
DRT-specific code, but may still need test_shell as a
minimum-capabilities
embedder of WebKit. Or we could go with a MiniBrowser-style project
(open-source sample for WebKit Apple port) upstream.
- Drive effort to completion. *Completion is*: DRT is ready to run
layout tests on build.webkit.org, downstream cleaned up and ready to
stop running layout tests on build.chromium.org.
- Upstream test infrastructure (watch detailed plan and assignments
develop at
http://dev.chromium.org/developers/design-documents/upstreaminglayouttests).
Steps include:
- Move run_webkit_tests.py to live upstream
- While DRT is being upstreamed, configure to run upstreamed
run_webkit_tests
on the Chromium builders
- Move test expectations upstream
- Move ancilliary tools, like rebaseline.py and flakiness
dashboard upstream
- Modify run_webkit_tests to use the newly upstreamed DRT
- Drive effort to completion. *Completion is*: build.webkit.org is
running layout tests with no regressions from downstream,
build.chromium.org no longer runs layout tests.
- Continue fixing/deflakifying layout tests and improving the process for
sustaining compatibility.
- Fix chrome-specific failures that dramatically affect compatibility
(bug triage -- which means *you*!)
- Fix SVG/Skia bugs that cause layout test failures (Waterloo team)
- Fix V8 bugs and develop ES5 features that affect layout tests (V8
team)
- Make V8 bindings more tolerant to IDL/JSC changes. Watch the effort as
a bug tree:
https://bugs.webkit.org/showdependencytree.cgi?id=32630&hide_resolved=0
(japhet, kkanetkar)
- Improve gardening tools and process (dglazkov)
- Eliminate layout test flakiness (jparent, ojan, dglazkov)
- Drive effort to completion. *Completion is*: <10 sustained layout test
failures or flakes, bindings-related regressions/breaks are limited to
custom code only, new process for keeping up for layout test
regressions is
adopted by Webkit Gardeners.
As you can see, it's a bit different from "grab a test and fix it" approach
we took in Q4. We learn, right? :) Despite most tasks having a driver in one
form or another, we always welcome your contributions. Every little bit
counts.
:DG<
--
Chromium Developers mailing list: [email protected]
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
