I would agree that the docker-solr tests (soon to be the solr/docker tests) do a fairly good job of testing bin/solr and general runtime logic. Once that gets merged in, I think it would be great to use that test suite for all of the additional "bin/solr" tests that we want to add. Once we get the docker stuff in and it's matured a bit, we could get it running in our CI and have nightly images built and tested.
Running the tests in a container would be great, and would solve some of the cross-platform issues that exist there. At that point it might be good to add an option to run the tests via Kubernetes instead of Docker. That way we can fully containerize the tests without having to mount the docker socket within a container, which would be required if we tried to run Solr containers from within the tests container. That'd be a fairly big overhaul, though I do think it would be a good win. Ideally we'd have the option to run the tests with docker or Kube, but I'm not sure how feasible that would be. - Houston On Tue, Sep 8, 2020 at 4:39 PM David Smiley <dsmi...@apache.org> wrote: > I never really looked at the smoketester before today. > smokeTestRelease.py does very very little with a running Solr instance -- > just run techproducts and do a query. testSolrExample() is the function > where this happens. Unless I'm missing something lots more, I can > confidently say that the docker-solr project's tests actually test the > final Solr instance substantially more than our project does. > > https://github.com/docker-solr/docker-solr/tree/master/tests > > I'd like to see the Solr project incorporating the docker-solr tests into > nightly CI builds somehow. We'll be much closer to making this happen once > docker-solr is absorbed into Solr -- > https://github.com/apache/lucene-solr/pull/1769 because we'll be able to > have nightly builds of images. The docker-solr separate project is limited > to releases, which poses a distinct chicken-and-egg problem. > > Maybe those tests could themselves run from a Docker image, thus > insulating platform issues further. > > Separately, someday I do want to work on running SolrExampleTests against > the docker image, which will be more possible once the projects merge. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Tue, Sep 8, 2020 at 8:02 AM Jason Gerlowski <gerlowsk...@gmail.com> > wrote: > >> I created a few JIRAs to spike "bin/solr" tests several years back >> (SOLR-11749). I didn't have quite enough time to get it across the finish >> line previously but I think it's a great idea to have tests of this sort. >> At the time I was writing tests in bash - but realized that wasn't a >> particularly scalable approach. >> >> Would be happy to help where I can if the effort gets restarted. >> >> Best, >> >> Jason >> >> On Mon, Sep 7, 2020 at 3:14 AM Uwe Schindler <u...@thetaphi.de> wrote: >> >>> It should be part of an integration test suite, only on module "package" >>> after assembly. That's something to setup. Please make sure that it works >>> with any operating system, because we are leaving Java world here (and >>> because of this, don't mix that into default unit tests). >>> >>> Currently we have a test for bin/solr: Smoketester 😊 >>> >>> Uwe >>> >>> ----- >>> Uwe Schindler >>> Achterdiek 19, D-28357 Bremen >>> https://www.thetaphi.de >>> eMail: u...@thetaphi.de >>> >>> > -----Original Message----- >>> > From: Dawid Weiss <dawid.we...@gmail.com> >>> > Sent: Monday, September 7, 2020 9:00 AM >>> > To: Lucene Dev <dev@lucene.apache.org> >>> > Subject: Re: Tests that use bin/solr? >>> > >>> > Just a note - such integration tests should depend on (and consume) >>> > the output of solr/packaging (a ZIP file with fully assembled >>> > package). Then you're really sure you're testing the final artifact. >>> > >>> > Dawid >>> > >>> > On Mon, Sep 7, 2020 at 7:51 AM David Smiley <dsmi...@apache.org> >>> wrote: >>> > > >>> > > Do we have any tests that operate on a "real" Solr instance running >>> from >>> > "bin/solr"? Such tests could find problems with bin/solr and any >>> classpath >>> > matters in how Jetty operates. Solr does have JettySolrRunner which >>> is great >>> > but doesn't cover the aforementioned matters. >>> > > >>> > > We've got some really nice tests in SolrExampleTests which is a base >>> class and >>> > many implementations that create SolrJ clients in different ways. I >>> could >>> > imagine modifying this such that if a magic system property is >>> specified to a >>> > URL of an existing Solr instance, then the test would not create a >>> > JettySolrRunner but instead use the configured one. This would then be >>> > executed by the smoke tester and maybe a future Docker release >>> process. I >>> > have test infrastructure I wrote where I work that does this sort of >>> thing for our >>> > Solr plugins, and it works great. >>> > > >>> > > ~ David Smiley >>> > > Apache Lucene/Solr Search Developer >>> > > http://www.linkedin.com/in/davidwsmiley >>> > >>> > --------------------------------------------------------------------- >>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> > For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>>
