I did another round of beast, this time setting that flag to true (as suggested to me by Noble).
ant -Duse.perreplica=true -Dtests.dups=1 -Dtests.iters=5 -Dbeast.iters=5 -Dtestcase=SolrCloudReportersTest beast -beast: [beaster] Beast round 1 results: /home/ishan/code/lucene-solr/solr/build/solr-core/test/1 [beaster] Beast round 2 results: /home/ishan/code/lucene-solr/solr/build/solr-core/test/2 [beaster] Beast round 3 results: /home/ishan/code/lucene-solr/solr/build/solr-core/test/3 [beaster] Beast round 4 results: /home/ishan/code/lucene-solr/solr/build/solr-core/test/4 [beaster] Beast round 5 results: /home/ishan/code/lucene-solr/solr/build/solr-core/test/5 [beaster] Beasting finished Successfully. I'm not sure what's going on :-( On Tue, Feb 16, 2021 at 11:13 AM Anshum Gupta <ans...@anshumgupta.net> wrote: > Ishan/Noble, thanks for taking a look at this. > > I only just started to look at the cause, so I'm sure you have better > context on why this is failing and if it makes sense to still release with > this issue. > > FYI, I was able to get a successful smoke test run finally, but the fact > that it took me over 7 runs. > > Also, can you confirm how did you run the test? you might be getting lucky > with the randomization here. Both me and Tim just commented out the > randomization for USE_PER_REPLICA_STATE and hardcoding this value to true > consistently got the test to fail. The default (false) did get the test to > pass 100% of the times. > > If you think we can have this fix before the release, it might make more > sense to have a single release for users as it wouldn't involve tracking > the complexity of what's broken in a released version. I still would like > to spend some more time tomorrow before voting on this one, but at least > the smoke test is out of the way. I'll try and debug this tomorrow. > > > On Mon, Feb 15, 2021 at 8:40 PM Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >> I tried light beasting the test on branch_8_8: >> ant -Dtests.dups=1 -Dtests.iters=5 -Dbeast.iters=5 >> -Dtestcase=SolrCloudReportersTest beast >> >> No failures. >> >> [beaster] Beast round 1 results: >> /home/ishan/code/lucene-solr/solr/build/solr-core/test/1 >> [beaster] Beast round 2 results: >> /home/ishan/code/lucene-solr/solr/build/solr-core/test/2 >> [beaster] Beast round 3 results: >> /home/ishan/code/lucene-solr/solr/build/solr-core/test/3 >> [beaster] Beast round 4 results: >> /home/ishan/code/lucene-solr/solr/build/solr-core/test/4 >> [beaster] Beast round 5 results: >> /home/ishan/code/lucene-solr/solr/build/solr-core/test/5 >> [beaster] Beasting finished Successfully. >> >> On Tue, Feb 16, 2021 at 10:07 AM Noble Paul <noble.p...@gmail.com> wrote: >> >>> @Anshum Gupta >>> >>> I think we should not hold up the release of RC1 because of that failure. >>> >>> This is a new feature and new features take time to get hardened. >>> >>> However, We can investigate and fix this anyway. >>> >>> If required, we can do a 8.8.3 >>> >>> On Tue, Feb 16, 2021 at 3:10 PM Ishan Chattopadhyaya >>> <ichattopadhy...@gmail.com> wrote: >>> > >>> > Here's my +1 for the RC1. >>> > >>> > SUCCESS! [0:42:38.936787] >>> > >>> > On Tue, Feb 16, 2021 at 9:02 AM Ishan Chattopadhyaya < >>> ichattopadhy...@gmail.com> wrote: >>> >> >>> >> Per Replica States is a new feature introduced in 8.8.0. It will >>> require a critical bugfix (SOLR-15138) immediately after 8.8.1 (in a 8.8.2 >>> release). If this issue is confirmed to be PRS related, then I think we >>> should continue with this release and fix PRS in 8.8.2. >>> >> >>> >> However, if you still want us to investigate and fix this issue now, >>> we can take a look. If you have a failing seed handy, please let me know. >>> >> >>> >> On Tue, Feb 16, 2021 at 8:33 AM Ishan Chattopadhyaya < >>> ichattopadhy...@gmail.com> wrote: >>> >>> >>> >>> Surprising. I'll take a look. >>> >>> >>> >>> On Tue, 16 Feb, 2021, 7:29 am Anshum Gupta, <ans...@anshumgupta.net> >>> wrote: >>> >>>> >>> >>>> I've unsuccessfully tried getting the smoketester to pass and have >>> had 6 fails so far. >>> >>>> >>> >>>> At this point it seems like SolrCloudReporterTest and >>> AutoscalingHistoryTest tests are failing pretty consistently for me. >>> >>>> >>> >>>> The former is a new failure, and seems to be caused by the >>> USE_PER_REPLICA_STATE randomization. >>> >>>> >>> >>>> Both Tim and me tried running the tests without the randomization >>> and defaulting that property to false gets the tests to pass, however it >>> seems to be failing every time the value for USE_PER_REPLICA_STATE is set >>> to true. >>> >>>> >>> >>>> I'm not voting -1 yet, as I'm not sure how much this affects the >>> build vs the test, but once we have a clearer picture, we might need a fix >>> and have to respin this. >>> >>>> >>> >>>> -Anshum >>> >>>> >>> >>>> On Sun, Feb 14, 2021 at 8:31 AM Timothy Potter < >>> thelabd...@gmail.com> wrote: >>> >>>>> >>> >>>>> Looks like an extra space got added on the end of the python3 >>> command, try this one: >>> >>>>> >>> >>>>> python3 -u dev-tools/scripts/smokeTestRelease.py >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.1-RC1-rev6a50a0315ac7e4979abb0b530857c7795bb3b928 >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Sun, Feb 14, 2021 at 9:26 AM Timothy Potter < >>> thelabd...@apache.org> wrote: >>> >>>>>> >>> >>>>>> Please vote for release candidate 1 for Lucene/Solr 8.8.1 >>> >>>>>> >>> >>>>>> >>> >>>>>> The artifacts can be downloaded from: >>> >>>>>> >>> >>>>>> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.1-RC1-rev6a50a0315ac7e4979abb0b530857c7795bb3b928 >>> >>>>>> >>> >>>>>> >>> >>>>>> You can run the smoke tester directly with this command: >>> >>>>>> >>> >>>>>> >>> >>>>>> python3 -u dev-tools/scripts/smokeTestRelease.py \ >>> >>>>>> >>> >>>>>> >>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.8.1-RC1-rev6a50a0315ac7e4979abb0b530857c7795bb3b928 >>> >>>>>> >>> >>>>>> >>> >>>>>> The vote will be open for at least 72 hours i.e. until 2021-02-17 >>> 17:00 UTC. >>> >>>>>> >>> >>>>>> >>> >>>>>> Here is my +1 ~ SUCCESS! [0:50:06.728441] >>> >>>>>> >>> >>>>>> >>> >>>>>> In addition to the smoke test, I built a Docker image from >>> solr-8.8.1.tgz locally and verified: >>> >>>>>> >>> >>>>>> >>> >>>>>> a. A rolling upgrade of a 3-node 8.7.0 cluster to the 8.8.1 RC >>> completes successfully w/o any NPEs or weirdness with leader election / >>> recoveries. >>> >>>>>> >>> >>>>>> >>> >>>>>> b. The base_url property is stored in replica state after the >>> upgrade >>> >>>>>> >>> >>>>>> >>> >>>>>> c. A basic client application built with SolrJ 8.7.0 can load >>> cluster state info directly from ZK and query the 8.8.1 RC1 servers. >>> >>>>>> >>> >>>>>> >>> >>>>>> d. Same client app built with SolrJ 8.8.0 works as well. >>> >>>>>> >>> >>>>>> >>> >>>>>> As this bug-fix release is primarily needed to address a SolrJ >>> back-compat break (SOLR-15145) and unfortunately our smoke tester framework >>> does not test for backcompat of older SolrJ against the RC, I ask others to >>> please test rolling upgrades of servers (ideally multi-node clusters) >>> running pre-8.8.0 to this RC if possible. Also, please try client >>> applications that are using an older SolrJ, esp. those that load cluster >>> state directly from ZK. >>> >>>>>> >>> >>>>>> >>> >>>>>> Best regards, >>> >>>>>> >>> >>>>>> Tim >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Anshum Gupta >>> >>> >>> >>> -- >>> ----------------------------------------------------- >>> Noble Paul >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: dev-h...@lucene.apache.org >>> >>> > > -- > Anshum Gupta >
