Did you have that sense you were missing something? Well, you were
right: You were missing your ...
# *Monthly CodeCoverage! update! \o/ *
/If you //want to hear more about an//y of the//s//e items, please
contact me and I will get you more detailed information/*
*
*## **Summary of July*
* *More hard work on the ETL pipeline*: Condensing the data to make it
manageable, and writing more efficient code for faster processing.
There is still more work to be done, but it's working. Marco
summarized this work with an excellent blog post: [1]
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
* *Analyzing coverage variability* - If you recall from previous
months, I mentioned that different coverage runs show different
coverage numbers: We call this "coverage variability", and it even
exists when comparing two runs from the same revision. gmierz has
been looking into this variability to better understand its size,
and character. [2], [3]
* *Finished detailed the plan for the MVP *[4] - This MVP is for
visualizing coverage of net-new lines: Relman is interested in the
coverage on the net-new lines of every changeset This is hard given
we can only run coverage on every central push. We now have a plan
for how to get this done, as best as possible, with the information
given.
*## Plans for August*
* *Build the MVP* - Visualize coverage of net new lines by changeset:
Now that we have a plan, armenzg has already started the frontend
UI work [5], [6], and will be working with marco working on the
backend [7]
* *Continue work on optimizing the ETL pipelines* - necessary work *
*
*
**## Meetings*
We have weekly CodeCoverage meetings, and you are welcome to attend:
* When: Held every Friday @ 11:30 EDT (08:30 PDT)
* Where: Kyle's video room
https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC
* Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
*## Reference*
[1] Blog post on coverage collection and aggregation:
https://marco-c.github.io/2017/07/28/code-coverage-architecture.html
[2] Variability analysis code -
https://github.com/gmierz/coco-variability-tools
[3] Example variability output -
https://gmierz.github.io/variability/variability_index.html
[4] Details for UI -
https://public.etherpad-mozilla.org/p/code_coverage_Q3_17
[5] Code for UI - https://github.com/armenzg/code_cov_experiments
[6] Example UI (very rough, but stay tuned!) -
https://armenzg.github.io/code_cov_experiments/
[7] Code for backend -
https://github.com/mozilla-releng/services/tree/master/src/shipit_code_coverage
On 2017-07-06 21:37, Kyle Lahnakoski wrote:
>
>
>
> ## Summary of Past Quarter
>
> Coverage is enabled for nearly all tests, and scheduled every push [1]!!
>
> * All c/c++ test suites have coverage enabled
> * talos coverage is enabled
> * jsvm[7] coverage is enabled, and running
> * codecov.io [2] shows the results, broken down by directory
>
> ## Plans for Q3
>
> The overall plan for Q3 is laid out in the planning document [12].
> Put simply, a **coverage differential viewer**, operating at low
> resolution (per central push), has enough promise to justify Q3 effort
> on CodeCoverage.
>
> ## The Complications
>
> * Rust code coverage is still delayed [6] - maybe by mid quarter
> * The data volume is very large; coveralls.io and codecov.io are
> having some difficulty dealing with the volume.
> * There is instability in the coverage numbers due to variability in
> our test runs; think GC and telemetry logic. Multiple coverage
> runs will be required to get a total coverage number
> * Intermittents are impacting coverage reliability - we will require
> a coverage health monitor to know if coverage is "complete"
>
> ## Summary of this past June
>
> * upgrading tests to use Ubuntu 16.04
> * fixing blockers that stood in the way of rust coverage[6]
> * enabling coverage to even more suites, like talos [10]
> * adding JavaScript to the codecov/coveralls report
> * getting a handle on the volume of data code coverage is producing
>
> ## Plans for July
>
> * continued work on ETL pipeline
> * enable coverage for spidermonkey [11]
> * see the first hints of Rust coverage
> * build a coverage health monitor to deal with "the complications"
> (above)
>
> ## Meetings
>
> We have weekly CodeCoverage meetings, and you are welcome to attend:
>
> * When: Held every Friday @ 11:30 EDT (08:30 PDT)
> * Where: Kyle's video room
> https://v.mozilla.com/flex.html?roomdirect.html&key=huhL8WaTwCwC
> * Etherpad: https://public.etherpad-mozilla.org/p/code_coverage_Q1_17
>
>
> ## Reference
>
> [1] See coverage on TH
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=ccov
>
> [1b] Example on TH:
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=433c379d6e448dc64df25a6b4d8739c99e84d102&filter-searchStr=ccov
>
>
> [2] codecov.io https://codecov.io/gh/marco-c/gecko-dev
>
> [3] Local Coverage
> https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_(or_any_other_treeherder_build
> <https://developer.mozilla.org/en-US/docs/Mozilla/Testing/Measuring_Code_Coverage_on_Firefox?document_saved=true#Generate_Code_Coverage_report_from_a_try_build_%28or_any_other_treeherder_build>)
>
> [4] iOS Coverage
> https://codecov.io/gh/mozilla-mobile/firefox-ios/branch/master
>
> [5] User Cases
> https://docs.google.com/document/d/1JUEPS8Xdtx4y8fXA4Au_Ggme0fMyJQxZbby0X2e4yDI/edit#heading=h.ddwyjkvxus4-
>
>
> [6] Rust coverage https://bugzilla.mozilla.org/show_bug.cgi?id=1335518
> and https://github.com/rust-lang/rust/pull/42433
>
> [7] JSVM coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1301174
>
> [8] e10s coverage example:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=b6e9cefe95adc3dd281bf8e2a2f897e8f4839e51
>
> [9] Everything:
> https://bugzilla.mozilla.org/showdependencytree.cgi?id=1278393
>
> [10] talos coverage: https://bugzilla.mozilla.org/show_bug.cgi?id=1372324
>
> [11] spidermonkey coverage:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1362013
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1372324>
>
> [12] Planning document:
> https://docs.google.com/document/d/1dOWi18qrudwaOThNAYoCMS3e9LzhxGUiMLLrQ_WVR9w/edit?usp=sharing
> <https://bugzilla.mozilla.org/show_bug.cgi?id=1372324>
>
>
>
>
>
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
