Hi everyone, I was wondering why we don't run every test on every PR. I saw the earlier comment which said it was too expensive but it would be good if we had that feature. As an experiment I ran all tests in a Github PR action here <https://github.com/gautamworah96/lucene-solr/pull/2/checks?check_run_id=1223630542> and it took ~3h 51m.
I think it is useful because sometimes we unknowingly break something in another module and might not catch it locally because we run `./gradlew test` only on the submodule that we've modified. It would also provide some benefit to reviewers (more confidence in merging?) and reduce breakages in builds. Looking forward to your feedback! - Gautam On Fri, Sep 18, 2020 at 2:04 PM Houston Putman <houstonput...@gmail.com> wrote: > Thanks for the feedback everyone! > > I'll get us started off with the docker and SolrJ tests. We can revisit > after a few weeks and see how it has gone. > > - Houston > > On Fri, Sep 18, 2020 at 2:28 PM Atri Sharma <a...@apache.org> wrote: > >> +1. >> >> Ability to run tests in Github actions should help prevent a ton of build >> breakages. >> >> Thanks for leading this, Houston! >> >> On Fri, 18 Sep 2020 at 21:24, Houston Putman <houstonput...@gmail.com> >> wrote: >> >>> Good point on the reference_impl branch. Eventually that's the goal, but >>> given there's not a timeline for that to be merged yet I think this is a >>> good stop-gap. It's a few minutes of work to get these PR actions written, >>> so I feel like there is little downside. And we can always remove them when >>> the reference_imp branch gets merged. >>> >>> This may block my ability to merge any PR whatsoever. >>> >>> >>> The docker tests will not be run on any PRs that don't touch bin/solr, >>> solr/packaging or solr/docker. >>> >>> There are complications around running integration tests in a >>> non-containerized environment as well. And if the docker image tests are >>> failing on the PR, wouldn't you rather know before committing? Even if you >>> don't want to install docker, you can call in someone else that has it to >>> help debug. Much like PRs that affect solr.cmd, for committers without >>> access to a windows machine. >>> >>> On Thu, Sep 17, 2020 at 6:23 PM Ishan Chattopadhyaya < >>> ichattopadhy...@gmail.com> wrote: >>> >>>> > It would be great to run all the tests every time, but clearly that >>>> is too expensive. >>>> >>>> The reference_impl branch requires around 30 seconds to run all >>>> solr-core tests. That's where we should all put our collective efforts. >>>> Also, I have reservations against docker based tests blocking PRs. If I >>>> don't have docker running on my dev machine, I wouldn't be able to make >>>> those tests pass. This may block my ability to merge any PR whatsoever. >>>> Why can't we have integration tests that do not rely on docker? >>>> >>>> On Thu, Sep 17, 2020 at 9:26 PM Houston Putman <houstonput...@gmail.com> >>>> wrote: >>>> >>>>> Thought I'd make this a thread instead of a discussion on a single >>>>> JIRA ticket. >>>>> >>>>> Currently we have gradle precommit run on PRs for master, which is >>>>> very useful and gives people confidence in approving PRs. But precommit is >>>>> obviously not the only thing we care about before committing. It would be >>>>> great to run all the tests every time, but clearly that is too expensive. >>>>> >>>>> In SOLR-14856 <https://issues.apache.org/jira/browse/SOLR-14856>, I >>>>> proposed adding a github action to build and test the solr docker image >>>>> for >>>>> PRs that affected relevant parts of the repo (solr/docker, solr/bin, >>>>> solr/packaging and solr/contrib/prometheus-exporter/bin). Running the >>>>> docker tests currently takes roughly 12 minutes in the github action, >>>>> which >>>>> would be costly if it ran on every PR. But when running on the small >>>>> percentage of PRs that affect those code paths, I think the benefit >>>>> outweighs the cost. >>>>> >>>>> Beyond just the docker tests, I think we can leverage this ability for >>>>> other features that are limited to certain code paths. For example running >>>>> tests for contrib modules, testing solr/examples, and many of >>>>> the independent lucene modules. The SolrJ tests just ran in 3 minutes >>>>> locally for me, maybe that'd be a good candidate as well. >>>>> >>>>> Anyways I'm sure there are other good candidates out there, but I just >>>>> wanted to start the discussion and hear other opinions before diving any >>>>> deeper. >>>>> >>>>> >>>>> >>>> >>>> >>> >>> -- >> Regards, >> >> Atri >> Apache Concerted >> >
