Vào 06:37:53 UTC+7 Thứ Bảy, ngày 23 tháng 6 năm 2018, Kyle Lahnakoski đã viết:
> ## Overview
>
> * **Coverage on recent changesets** - A list of recent changesets and
> the percent of new lines covered by tests.
> https://firefox-code-coverage.herokuapp.com
> * **Coverage addon** - Use it to show coverage on your patches in
> Bugzilla -
> https://addons.mozilla.org/en-US/firefox/addon/gecko-code-coverage/
> * **Daily coverage aggregates** - by directory and file -
> https://codecov.io/gh/mozilla/gecko-dev
> * **Zero coverage reports** -
> https://marco-c.github.io/code-coverage-reports/ with ore details here:
> https://release.mozilla.org/tooling/codecoverage/2018/03/23/code-coverage.html
> * **Not-so-raw coverage artifacts** - If you are interested in the
> details, you may want to see the artifacts produced by coverage builds:
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov
>
>
> ## Contact Us
>
> If you want to know more, or want to participate, stop by and say "hi":
>
> * **Mail list** - codecover...@mozilla.com
> * **IRC** - #codecoverage
>
>
> ## Details
>
> This past 6 months have been spent working on minutia and hard problems
> of the CodeCoverage system. There are many platforms, many suites, a
> complex build, and a lot of data. Taming the complexity so we can
> provide "clean" coverage artifacts is proving arduous. Here are some
> examples:
>
> * **translating filenames** - Our build has generated source files, and
> files copied to appropriate directories before compile. The file
> reported in coverage artifacts must be mapped back to files we see in
> the tree. Our coverage processing now does this work.
> * **chunk <-> file mapping** - Our coverage is collected by running
> hundreds of tasks, each responsible for running some part of a test
> suite; called a "chunk". An interesting question is "if we have a
> changeset, and we know what files it changed, can we schedule less
> chunks?" The chunk<->file mapping allows use to answer that question.
> Turns out the answer is "not really".
> * **windows coverage** - If you visit Treeherder you will see we are
> collecting coverage on Windows (using clang-cl). This allows us to see
> platform-specific lines.
> * **grcov improvements** - now faster than before, and can process llvm
> coverage output
> * **jsdcov e10s** - We had to verify that coverage was getting
> collected by all processes
> * **frontend refinements** - There has been continued work on the
> frontend to meet Release Management's use cases. Another cohort of
> UCOSP students has helped fix bugs, and added functionality to the
> frontend.
> * **variability analysis** - Coverage at the Firefox-scale has
> variability from many different sources. For instance, running the same
> test on the same revision multiple times shows differences from one run
> to the next. Plus, in some other cases, tests fail, which results in
> different coverage. These situations were explored in more detail to
> better understand what variability we are dealing with.
> * **jsdcov vs jsvm** - Both are collecting coverage on javascript; they
> appear to be capturing different coverage; we must look into this more
> * **bugs!** - Many bugs, timing bugs in the tests, and bugs in the
> coverage collection/reporting pipeline.
> * **android experiments** - Android will be be given more focus in the
> coming year, so we should look at how coverage will help.
> * **backend improvements** - Work that must be done to deal with data
> volume, but is barely visible to the end user.
> * **backend moved to production** - CodeCoverage has moved to
> production; and this took time.
> * **subtracting baselines** - We reduce our coverage artifact size by
> subtracting no-test coverage from test coverage. In theory, after
> subtraction, the coverage represents what is uniquely used by the test
> and removes all standard browser/test start and shutdown coverage.
> However, we have found that coverage variability is affecting these
> results as well and are now looking into mitigation strategies.
> * **per test coverage** - One of our goals is to collect coverage at
> the test level; this can help guide what tests should be run when a file
> changes. It may also help with understanding gaining a deeper
> understanding of what our tests are testing.
>
> ## The Future
>
> * **move off codecov.io** - Our data volume and our use cases, do not
> fit with codecov.io. We will use our own database to store coverage
> artifacts, and enable us to do more.
> * **per test coverage** - One of our goals is to collect coverage at
> the test level; this can help guide what tests should be run when a file
> changes. It might also help understanding which tests are important
> (e.g. for evaluating whether to disable or fix intermittent tests)
> * **compare coverage across revisions** - We want to markup the
> coverage so a reasonable comparison can be done
