Hi,
After working a little bit with Solr, let me share my view on the test
failures. As I see there are certain technical aspects of the tests
themselves that make them gravitate towards ignorance.
*Randomness* makes it difficult to correlate a failure to the commit that
made the test to fail (as was pointed out earlier in the discussion). If
each execution path is different, it may very well be that a failure you
experience is introduced several commits ago, so it may not be your fault.
Also you may not be sure that if your tests are green you have not broken
anything. The relevance of the test run for your work is decreased. This
just contradicts to the aim of these tests: ship if and only if the tests
are green. So in my opinion random tests alone are not very handy to
protect against regression. They are best for an additional layer of
stability/endurance tests.
*Diagnosability* of tests could be improved. For example:
public static boolean waitForLiveAndActiveReplicaCount(ZkStateReader
zkStateReader,
String collection, int replicaCount, int timeoutInMs) {
.. this line tells if live replica count has reached the limit or not. It
does not tell how many replicas are there, what the others are doing, etc..
You have to get into the logs for details.
assertEquals(0, response3.getStatus());
This will tell if response3 failed, but will not tell what was the error.
You could tell much more because response3.getErrorMessages() tells the
details and could help in diagnosing the problem better if that was
included into the error/failure message of the test assertion.
Many tests are *slower* than a traditional unit test. Mostly because they
start a cluster of some sort. Unit tests that test a single class which
gets executed in the test thread is much faster, much easier to rerun,
debug and fix.
Many tests are* long* and could be broken into multiple of testcases. The
real expected micro-functionalities are waved inside the testcase. For
example in SharedFSAutoReplicaFailoverTest it is verified that if
autoaddReplicas is true, the failed core is recovered on another one, if it
is false it is then not recovered, replicas dont exceed the limit of the
maxShardsPerNode, etc. If these are separate testcases, it is easier to
tell which behaviour is broken, so it is easier to diagnose, and it is
easier to extend with additional functionality.
These may sound small but all these add up and gets multiplied with the
amount of testcases, the number of runs and the number of developers.
Best Regards,
Misi
> --------------------------------------------
> From: Erick Erickson <erickerick...@gmail.com>
> Date: 2018-02-22 20:39 GMT+01:00
> Subject: Re: Test failures are out of control......
> To: dev@lucene.apache.org
>
>
> See: https://issues.apache.org/jira/browse/SOLR-12016
>
> And there are already several linked issues that have been fixed, I'll
> try removing the annotations in SOLR-12017.
>
> Erick
>
> On Thu, Feb 22, 2018 at 9:55 AM, Uwe Schindler <u...@thetaphi.de> wrote:
> >> : great and very helpful! Does it only contain Solr or are there also
> Lucene
> >> tests reported?
> >>
> >> That was just a dumb choice on my part when creating the fucit.org URL
> >> since the vast majority of the jenkins failures are in Solr tests.
> >>
> >> It's really every test failure reported from any build of any jenkins
> jobs
> >> in one of the feeds it's tracking...
> >>
> >> https://github.com/hossman/jenkins-reports/blob/master/venus.ini#L66
> >>
> >> (BTW: If you could setup a "jenkins view" for only the "Lucene/Solr"
> jobs
> >> on your jenkins server -- similar to what we have on builds.apache and
> >> sarowe's jenkins -- that would help reduce the noise in the report
> if/when
> >> there are any test failures in other non-lucene jenkins jobs you have
> >> running)
> >
> > Here you are: https://jenkins.thetaphi.de/view/Lucene-Solr/
> >
> >> If, for example, you type "org.apache.lucene" in the text box in the
> class
> >> column, it will filter down to only show you test failures from lucene
> >> package tests -- like for example: in the past 24 hours,
> >> oal.search.TestSearcherManager has had suite level failures in 10% of
> the
> >> jenkins builds in which it run...
> >>
> >> http://fucit.org/solr-jenkins-reports/failure-report.html
> >
> > Wonderful!
> >
> > Uwe
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
