I would recommend a search for "github actions jacoco" to review what's common out there.
If we change 'gradle test' to 'gradle coverage' in our existing PR-test action, the next step is to just not throw away the reports, but make them available. See https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts for some documentation on this. Seems common for PR workflows to have the action "comment on the PR" with coverage information. Not sure if we want that as it could result in a ton of comments. Finally, the current "gradle coverage" builds a separate coverage report for each lucene module, I think. So we may want to think about adding support to "merge" the jacoco data across all the modules and build one monster report for all of lucene, too. This would just be some work with the gradle build: but I think it would make the information a lot easier to digest. This is already happening with the "jenkins coverage build" which presents one monster report, but I think it may be something on the jenkins side doing it? https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ On Wed, Oct 5, 2022 at 8:58 AM Patrick Zhai <zhai7...@gmail.com> wrote: > > Make sense to me, I'll try to look into it! > > On Tue, Oct 4, 2022, 16:50 Robert Muir <rcm...@gmail.com> wrote: >> >> We already have code coverage integrated into the build. See the >> documentation on how to generate the reports: >> https://github.com/apache/lucene/blob/main/help/tests.txt >> >> I think we should stick with jacoco and not some commercial stuff for >> measuring coverage. Jacoco works great. We just have to put the >> reports or stats somewhere useful. >> >> On Tue, Oct 4, 2022 at 5:45 PM Patrick Zhai <zhai7...@gmail.com> wrote: >> > >> > Hi Robert, thank you for commenting, yeah the functionality I want to add >> > is actually the line by line code coverage stats for the new/changed line >> > that are in the patch so that we don't need to wonder about "whether that >> > line is covered by the test?". But I'm against using the code coverage as >> > any kind of hard criteria, like coverage must be kept at a certain % or >> > all the new lines must be covered, that will drive people crazy. I think >> > that should be just treated as a helpful thing to check when >> > reviewing/creating the PR. >> > >> > I searched a little on google and found this: https://about.codecov.io/, >> > it's free for open source and seems to have the functionality we need. Let >> > me know if anyone has ideas about this, or otherwise I can try it a little >> > bit with my own repo first and then try to add it to lucene. >> > >> > Best >> > Patrick >> > >> > >> > >> > On Tue, Oct 4, 2022, 06:36 Robert Muir <rcm...@gmail.com> wrote: >> >> >> >> btw, you can look at the current reports created by jenkins here: >> >> https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ >> >> >> >> On Tue, Oct 4, 2022 at 6:51 AM Robert Muir <rcm...@gmail.com> wrote: >> >> > >> >> > we can run the tests with coverage option and produce coverage graph >> >> > from the github actions, but need to look at the docs to see where to >> >> > put it so it will be available. >> >> > >> >> > I want us to be careful about the word "check" as I'm adamantly >> >> > against any such automated check (e.g. coverage > N%) in the logic. >> >> > Coverage report is just a tool to help us and the moment we do stupid >> >> > shit like that, is the moment people start gaming it just to make the >> >> > build pass. >> >> > >> >> > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai <zhai7...@gmail.com> wrote: >> >> > > >> >> > > Hi folks, >> >> > > I'm not sure whether people have already discussed this but I'm >> >> > > wondering whether we want to add a workflow that pulls out the code >> >> > > coverage whenever a PR was created? It should be easier for both the >> >> > > reviewers and the contributors to figure out what can be improved, or >> >> > > at least figure out a part that is probably not covered by the tests? >> >> > > >> >> > > Best >> >> > > Patrick >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
