Re: Code coverage check for PRs
One thing codecov gives is a sense of what the coverage was previously without having to go hunt down past builds in jenkins. It is a coverage focused view essentially. It just uses the coverage data the build already calculates. It used to have some nifty graphical visualizations but those seem to have disappeared or become hard to find. On Wed, Oct 5, 2022 at 9:24 AM Robert Muir wrote: > I would recommend a search for "github actions jacoco" to review > what's common out there. > > If we change 'gradle test' to 'gradle coverage' in our existing > PR-test action, the next step is to just not throw away the reports, > but make them available. See > > https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts > for some documentation on this. > > Seems common for PR workflows to have the action "comment on the PR" > with coverage information. Not sure if we want that as it could result > in a ton of comments. > > Finally, the current "gradle coverage" builds a separate coverage > report for each lucene module, I think. So we may want to think about > adding support to "merge" the jacoco data across all the modules and > build one monster report for all of lucene, too. This would just be > some work with the gradle build: but I think it would make the > information a lot easier to digest. This is already happening with the > "jenkins coverage build" which presents one monster report, but I > think it may be something on the jenkins side doing it? > > https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ > > > On Wed, Oct 5, 2022 at 8:58 AM Patrick Zhai wrote: > > > > Make sense to me, I'll try to look into it! > > > > On Tue, Oct 4, 2022, 16:50 Robert Muir wrote: > >> > >> We already have code coverage integrated into the build. See the > >> documentation on how to generate the reports: > >> https://github.com/apache/lucene/blob/main/help/tests.txt > >> > >> I think we should stick with jacoco and not some commercial stuff for > >> measuring coverage. Jacoco works great. We just have to put the > >> reports or stats somewhere useful. > >> > >> On Tue, Oct 4, 2022 at 5:45 PM Patrick Zhai wrote: > >> > > >> > Hi Robert, thank you for commenting, yeah the functionality I want to > add is actually the line by line code coverage stats for the new/changed > line that are in the patch so that we don't need to wonder about "whether > that line is covered by the test?". But I'm against using the code coverage > as any kind of hard criteria, like coverage must be kept at a certain % or > all the new lines must be covered, that will drive people crazy. I think > that should be just treated as a helpful thing to check when > reviewing/creating the PR. > >> > > >> > I searched a little on google and found this: > https://about.codecov.io/, it's free for open source and seems to have > the functionality we need. Let me know if anyone has ideas about this, or > otherwise I can try it a little bit with my own repo first and then try to > add it to lucene. > >> > > >> > Best > >> > Patrick > >> > > >> > > >> > > >> > On Tue, Oct 4, 2022, 06:36 Robert Muir wrote: > >> >> > >> >> btw, you can look at the current reports created by jenkins here: > >> >> > https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ > >> >> > >> >> On Tue, Oct 4, 2022 at 6:51 AM Robert Muir wrote: > >> >> > > >> >> > we can run the tests with coverage option and produce coverage > graph > >> >> > from the github actions, but need to look at the docs to see where > to > >> >> > put it so it will be available. > >> >> > > >> >> > I want us to be careful about the word "check" as I'm adamantly > >> >> > against any such automated check (e.g. coverage > N%) in the logic. > >> >> > Coverage report is just a tool to help us and the moment we do > stupid > >> >> > shit like that, is the moment people start gaming it just to make > the > >> >> > build pass. > >> >> > > >> >> > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai > wrote: > >> >> > > > >> >> > > Hi folks, > >> >> > > I'm not sure whether people have already discussed this but I'm > wondering whether we want to add a workflow that pulls out the code > coverage whenever a PR was created? It should be easier for both the > reviewers and the contributors to figure out what can be improved, or at > least figure out a part that is probably not covered by the tests? > >> >> > > > >> >> > > Best > >> >> > > Patrick > >> >> > >> >> - > >> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> >> > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> > >
Re: Code coverage check for PRs
I would recommend a search for "github actions jacoco" to review what's common out there. If we change 'gradle test' to 'gradle coverage' in our existing PR-test action, the next step is to just not throw away the reports, but make them available. See https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts for some documentation on this. Seems common for PR workflows to have the action "comment on the PR" with coverage information. Not sure if we want that as it could result in a ton of comments. Finally, the current "gradle coverage" builds a separate coverage report for each lucene module, I think. So we may want to think about adding support to "merge" the jacoco data across all the modules and build one monster report for all of lucene, too. This would just be some work with the gradle build: but I think it would make the information a lot easier to digest. This is already happening with the "jenkins coverage build" which presents one monster report, but I think it may be something on the jenkins side doing it? https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ On Wed, Oct 5, 2022 at 8:58 AM Patrick Zhai wrote: > > Make sense to me, I'll try to look into it! > > On Tue, Oct 4, 2022, 16:50 Robert Muir wrote: >> >> We already have code coverage integrated into the build. See the >> documentation on how to generate the reports: >> https://github.com/apache/lucene/blob/main/help/tests.txt >> >> I think we should stick with jacoco and not some commercial stuff for >> measuring coverage. Jacoco works great. We just have to put the >> reports or stats somewhere useful. >> >> On Tue, Oct 4, 2022 at 5:45 PM Patrick Zhai wrote: >> > >> > Hi Robert, thank you for commenting, yeah the functionality I want to add >> > is actually the line by line code coverage stats for the new/changed line >> > that are in the patch so that we don't need to wonder about "whether that >> > line is covered by the test?". But I'm against using the code coverage as >> > any kind of hard criteria, like coverage must be kept at a certain % or >> > all the new lines must be covered, that will drive people crazy. I think >> > that should be just treated as a helpful thing to check when >> > reviewing/creating the PR. >> > >> > I searched a little on google and found this: https://about.codecov.io/, >> > it's free for open source and seems to have the functionality we need. Let >> > me know if anyone has ideas about this, or otherwise I can try it a little >> > bit with my own repo first and then try to add it to lucene. >> > >> > Best >> > Patrick >> > >> > >> > >> > On Tue, Oct 4, 2022, 06:36 Robert Muir wrote: >> >> >> >> btw, you can look at the current reports created by jenkins here: >> >> https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ >> >> >> >> On Tue, Oct 4, 2022 at 6:51 AM Robert Muir wrote: >> >> > >> >> > we can run the tests with coverage option and produce coverage graph >> >> > from the github actions, but need to look at the docs to see where to >> >> > put it so it will be available. >> >> > >> >> > I want us to be careful about the word "check" as I'm adamantly >> >> > against any such automated check (e.g. coverage > N%) in the logic. >> >> > Coverage report is just a tool to help us and the moment we do stupid >> >> > shit like that, is the moment people start gaming it just to make the >> >> > build pass. >> >> > >> >> > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai wrote: >> >> > > >> >> > > Hi folks, >> >> > > I'm not sure whether people have already discussed this but I'm >> >> > > wondering whether we want to add a workflow that pulls out the code >> >> > > coverage whenever a PR was created? It should be easier for both the >> >> > > reviewers and the contributors to figure out what can be improved, or >> >> > > at least figure out a part that is probably not covered by the tests? >> >> > > >> >> > > Best >> >> > > Patrick >> >> >> >> - >> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Code coverage check for PRs
Make sense to me, I'll try to look into it! On Tue, Oct 4, 2022, 16:50 Robert Muir wrote: > We already have code coverage integrated into the build. See the > documentation on how to generate the reports: > https://github.com/apache/lucene/blob/main/help/tests.txt > > I think we should stick with jacoco and not some commercial stuff for > measuring coverage. Jacoco works great. We just have to put the > reports or stats somewhere useful. > > On Tue, Oct 4, 2022 at 5:45 PM Patrick Zhai wrote: > > > > Hi Robert, thank you for commenting, yeah the functionality I want to > add is actually the line by line code coverage stats for the new/changed > line that are in the patch so that we don't need to wonder about "whether > that line is covered by the test?". But I'm against using the code coverage > as any kind of hard criteria, like coverage must be kept at a certain % or > all the new lines must be covered, that will drive people crazy. I think > that should be just treated as a helpful thing to check when > reviewing/creating the PR. > > > > I searched a little on google and found this: https://about.codecov.io/, > it's free for open source and seems to have the functionality we need. Let > me know if anyone has ideas about this, or otherwise I can try it a little > bit with my own repo first and then try to add it to lucene. > > > > Best > > Patrick > > > > > > > > On Tue, Oct 4, 2022, 06:36 Robert Muir wrote: > >> > >> btw, you can look at the current reports created by jenkins here: > >> > https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ > >> > >> On Tue, Oct 4, 2022 at 6:51 AM Robert Muir wrote: > >> > > >> > we can run the tests with coverage option and produce coverage graph > >> > from the github actions, but need to look at the docs to see where to > >> > put it so it will be available. > >> > > >> > I want us to be careful about the word "check" as I'm adamantly > >> > against any such automated check (e.g. coverage > N%) in the logic. > >> > Coverage report is just a tool to help us and the moment we do stupid > >> > shit like that, is the moment people start gaming it just to make the > >> > build pass. > >> > > >> > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai > wrote: > >> > > > >> > > Hi folks, > >> > > I'm not sure whether people have already discussed this but I'm > wondering whether we want to add a workflow that pulls out the code > coverage whenever a PR was created? It should be easier for both the > reviewers and the contributors to figure out what can be improved, or at > least figure out a part that is probably not covered by the tests? > >> > > > >> > > Best > >> > > Patrick > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: dev-h...@lucene.apache.org > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: Code coverage check for PRs
We already have code coverage integrated into the build. See the documentation on how to generate the reports: https://github.com/apache/lucene/blob/main/help/tests.txt I think we should stick with jacoco and not some commercial stuff for measuring coverage. Jacoco works great. We just have to put the reports or stats somewhere useful. On Tue, Oct 4, 2022 at 5:45 PM Patrick Zhai wrote: > > Hi Robert, thank you for commenting, yeah the functionality I want to add is > actually the line by line code coverage stats for the new/changed line that > are in the patch so that we don't need to wonder about "whether that line is > covered by the test?". But I'm against using the code coverage as any kind of > hard criteria, like coverage must be kept at a certain % or all the new lines > must be covered, that will drive people crazy. I think that should be just > treated as a helpful thing to check when reviewing/creating the PR. > > I searched a little on google and found this: https://about.codecov.io/, it's > free for open source and seems to have the functionality we need. Let me know > if anyone has ideas about this, or otherwise I can try it a little bit with > my own repo first and then try to add it to lucene. > > Best > Patrick > > > > On Tue, Oct 4, 2022, 06:36 Robert Muir wrote: >> >> btw, you can look at the current reports created by jenkins here: >> https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ >> >> On Tue, Oct 4, 2022 at 6:51 AM Robert Muir wrote: >> > >> > we can run the tests with coverage option and produce coverage graph >> > from the github actions, but need to look at the docs to see where to >> > put it so it will be available. >> > >> > I want us to be careful about the word "check" as I'm adamantly >> > against any such automated check (e.g. coverage > N%) in the logic. >> > Coverage report is just a tool to help us and the moment we do stupid >> > shit like that, is the moment people start gaming it just to make the >> > build pass. >> > >> > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai wrote: >> > > >> > > Hi folks, >> > > I'm not sure whether people have already discussed this but I'm >> > > wondering whether we want to add a workflow that pulls out the code >> > > coverage whenever a PR was created? It should be easier for both the >> > > reviewers and the contributors to figure out what can be improved, or at >> > > least figure out a part that is probably not covered by the tests? >> > > >> > > Best >> > > Patrick >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Code coverage check for PRs
Hi Robert, thank you for commenting, yeah the functionality I want to add is actually the line by line code coverage stats for the new/changed line that are in the patch so that we don't need to wonder about "whether that line is covered by the test?". But I'm against using the code coverage as any kind of hard criteria, like coverage must be kept at a certain % or all the new lines must be covered, that will drive people crazy. I think that should be just treated as a helpful thing to check when reviewing/creating the PR. I searched a little on google and found this: https://about.codecov.io/, it's free for open source and seems to have the functionality we need. Let me know if anyone has ideas about this, or otherwise I can try it a little bit with my own repo first and then try to add it to lucene. Best Patrick On Tue, Oct 4, 2022, 06:36 Robert Muir wrote: > btw, you can look at the current reports created by jenkins here: > > https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ > > On Tue, Oct 4, 2022 at 6:51 AM Robert Muir wrote: > > > > we can run the tests with coverage option and produce coverage graph > > from the github actions, but need to look at the docs to see where to > > put it so it will be available. > > > > I want us to be careful about the word "check" as I'm adamantly > > against any such automated check (e.g. coverage > N%) in the logic. > > Coverage report is just a tool to help us and the moment we do stupid > > shit like that, is the moment people start gaming it just to make the > > build pass. > > > > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai wrote: > > > > > > Hi folks, > > > I'm not sure whether people have already discussed this but I'm > wondering whether we want to add a workflow that pulls out the code > coverage whenever a PR was created? It should be easier for both the > reviewers and the contributors to figure out what can be improved, or at > least figure out a part that is probably not covered by the tests? > > > > > > Best > > > Patrick > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org > >
Re: Code coverage check for PRs
btw, you can look at the current reports created by jenkins here: https://ci-builds.apache.org/job/Lucene/job/Lucene-Coverage-main/lastBuild/jacoco/ On Tue, Oct 4, 2022 at 6:51 AM Robert Muir wrote: > > we can run the tests with coverage option and produce coverage graph > from the github actions, but need to look at the docs to see where to > put it so it will be available. > > I want us to be careful about the word "check" as I'm adamantly > against any such automated check (e.g. coverage > N%) in the logic. > Coverage report is just a tool to help us and the moment we do stupid > shit like that, is the moment people start gaming it just to make the > build pass. > > On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai wrote: > > > > Hi folks, > > I'm not sure whether people have already discussed this but I'm wondering > > whether we want to add a workflow that pulls out the code coverage whenever > > a PR was created? It should be easier for both the reviewers and the > > contributors to figure out what can be improved, or at least figure out a > > part that is probably not covered by the tests? > > > > Best > > Patrick - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Code coverage check for PRs
we can run the tests with coverage option and produce coverage graph from the github actions, but need to look at the docs to see where to put it so it will be available. I want us to be careful about the word "check" as I'm adamantly against any such automated check (e.g. coverage > N%) in the logic. Coverage report is just a tool to help us and the moment we do stupid shit like that, is the moment people start gaming it just to make the build pass. On Mon, Oct 3, 2022 at 10:57 PM Patrick Zhai wrote: > > Hi folks, > I'm not sure whether people have already discussed this but I'm wondering > whether we want to add a workflow that pulls out the code coverage whenever a > PR was created? It should be easier for both the reviewers and the > contributors to figure out what can be improved, or at least figure out a > part that is probably not covered by the tests? > > Best > Patrick - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Code coverage check for PRs
Hi folks, I'm not sure whether people have already discussed this but I'm wondering whether we want to add a workflow that pulls out the code coverage whenever a PR was created? It should be easier for both the reviewers and the contributors to figure out what can be improved, or at least figure out a part that is probably not covered by the tests? Best Patrick