Re: Updated mozilla-central code coverage
Thanks for the explanation. IIRC content process is closed by SIGKILL in Gecko. Looks like we'll have to tweak the timing. Best Regards, Shih-Chiang Chien Mozilla Taiwan On Wed, May 27, 2015 at 10:10 AM, Joshua Cranmer pidgeo...@gmail.com wrote: On 5/26/2015 8:54 PM, Shih-Chiang Chien wrote: Hi Joshua, Great job for working on this! However I found that the coverage doesn't include those ran on child process (e.g. ContentChild). It would be wonderful if we can add the support on it. The coverage files are emitted by a process when it exits via an atexit hook. If anything causes that hook not to fire (e.g., the process is still running at the time the testsuite exits, or if the process is killed by a segfault or other signal), then no coverage data would be emitted. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Updated mozilla-central code coverage
On 5/26/2015 10:20 PM, Shih-Chiang Chien wrote: Thanks for the explanation. IIRC content process is closed by SIGKILL in Gecko. Looks like we'll have to tweak the timing. A SIGKILL would definitely not trigger the information to be dumped. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Updated mozilla-central code coverage
Does this coverage info also include gtests? From a quick glance it looks like not. On Tuesday, May 26, 2015 at 2:59:16 PM UTC-4, Joshua Cranmer wrote: I've posted updated code coverage information for mozilla-central to https://www.tjhsst.edu/~jcranmer/m-ccov/. This data is accurate as of yesterday. For those of you who are unaware, I periodically run these code coverage statistics by use of the try server and instrumented runs. This has been made easier over the years by standardization of some of the hacks, such that you can now push to linux64-cc and get most of the same information. Notable changes I've made since the last upload: 1. I dropped debug builds, so all the information comes from Linux opt, both 32 and 64-bit. 2. Test names now derive from builder names directly, removing the need for a very long hardcoded list of M-bc means mochitest-browser-chrome. 2a. This means that what was once mochitest-1, mochitest-2, etc. is now condensed into mochitest. Mochitest-e10s-browser-chrome, etc., remain split out. 3. Minor changes in the UI frontend to help deal with the fact that my hosting webserver changed to forcing https. 4. I can now generate the ultimate combined .info file without needing manual post-processing, for the first time ever. The marionette and web-platform tests remain unaccounted for in coverage (Mn, Mn-e10s, Wr, W-* in treeherder lingo), and the new Ld (luciddream?) appears to be broken as well. On the possibility of expanding code coverage information to different platforms, languages, and tests: 1. OS X still has a link error and/or fail-to-run issue. I suspect a newer clang would help, but I lack a local OS X instance with which to do any detailed tests. I've never tested the ability of my scripts to adequately collect clang code coverage data, and I suspect they would themselves need some modification to do so. 2. Android builds work and can report back code coverage data, but so intermittently that I didn't bother to try including them. In my try run that I used to generate these results, mochitest-2 reported back data but mochitest-6 did not, yet both testsuites reported back success. The reason for this is not clear, so any help people could give in debugging issues would be most appreciated. 3. B2G desktop builds and Mulet builds on Linux appeared to work. However, the builds didn't appear to upload the gcno package for unknown reasons, and taskcluster uses such different mechanisms to upload the files that my scripts are of no use in collecting the gcda packages. 4. Windows is... special when it comes to code coverage. This is the last platform I would look at tackling. 5. JS code coverage is of course a hole I'd like to see rectified, but I don't have the time to invest in solving it myself. 6. Are we actually using Rust yet on our tryserver builds? 7. Android Java coverage is deferred until after I can get reliable Android code coverage in the first place. 8. I'd have to look into modifying mozharness to run code coverage on marionette et al builds. It shouldn't be hard, but it is annoying to have to hook into so many places to insert code coverage. 9. Ditto for Android xpcshell and cppunit tests. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Updated mozilla-central code coverage
I've posted updated code coverage information for mozilla-central to https://www.tjhsst.edu/~jcranmer/m-ccov/. This data is accurate as of yesterday. For those of you who are unaware, I periodically run these code coverage statistics by use of the try server and instrumented runs. This has been made easier over the years by standardization of some of the hacks, such that you can now push to linux64-cc and get most of the same information. Notable changes I've made since the last upload: 1. I dropped debug builds, so all the information comes from Linux opt, both 32 and 64-bit. 2. Test names now derive from builder names directly, removing the need for a very long hardcoded list of M-bc means mochitest-browser-chrome. 2a. This means that what was once mochitest-1, mochitest-2, etc. is now condensed into mochitest. Mochitest-e10s-browser-chrome, etc., remain split out. 3. Minor changes in the UI frontend to help deal with the fact that my hosting webserver changed to forcing https. 4. I can now generate the ultimate combined .info file without needing manual post-processing, for the first time ever. The marionette and web-platform tests remain unaccounted for in coverage (Mn, Mn-e10s, Wr, W-* in treeherder lingo), and the new Ld (luciddream?) appears to be broken as well. On the possibility of expanding code coverage information to different platforms, languages, and tests: 1. OS X still has a link error and/or fail-to-run issue. I suspect a newer clang would help, but I lack a local OS X instance with which to do any detailed tests. I've never tested the ability of my scripts to adequately collect clang code coverage data, and I suspect they would themselves need some modification to do so. 2. Android builds work and can report back code coverage data, but so intermittently that I didn't bother to try including them. In my try run that I used to generate these results, mochitest-2 reported back data but mochitest-6 did not, yet both testsuites reported back success. The reason for this is not clear, so any help people could give in debugging issues would be most appreciated. 3. B2G desktop builds and Mulet builds on Linux appeared to work. However, the builds didn't appear to upload the gcno package for unknown reasons, and taskcluster uses such different mechanisms to upload the files that my scripts are of no use in collecting the gcda packages. 4. Windows is... special when it comes to code coverage. This is the last platform I would look at tackling. 5. JS code coverage is of course a hole I'd like to see rectified, but I don't have the time to invest in solving it myself. 6. Are we actually using Rust yet on our tryserver builds? 7. Android Java coverage is deferred until after I can get reliable Android code coverage in the first place. 8. I'd have to look into modifying mozharness to run code coverage on marionette et al builds. It shouldn't be hard, but it is annoying to have to hook into so many places to insert code coverage. 9. Ditto for Android xpcshell and cppunit tests. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Updated mozilla-central code coverage
On 5/26/2015 3:21 PM, kgu...@mozilla.com wrote: Does this coverage info also include gtests? From a quick glance it looks like not. The code coverage includes all tests run on Linux opt or Linux-64 opt excluding those run under check, marionette, web-platform tests, or luciddream. If gtests are being run under Linux opt cppunittests, then they should be included. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Updated mozilla-central code coverage
On Tue, May 26, 2015 at 03:48:06PM -0500, Joshua Cranmer ? wrote: On 5/26/2015 3:21 PM, kgu...@mozilla.com wrote: Does this coverage info also include gtests? From a quick glance it looks like not. The code coverage includes all tests run on Linux opt or Linux-64 opt excluding those run under check, marionette, web-platform tests, or luciddream. If gtests are being run under Linux opt cppunittests, then they should be included. They are currently running during make check. Mike ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
