Seriously, awesome debugging!!! I saw the lack of logs but was looking too much into the fact that it was always failing in the smoke tester. Great find!
If we need to do a respin, this will make it a lot easier haha. - Houston On Mon, Jul 21, 2025 at 6:29 PM David Smiley <dsmi...@apache.org> wrote: > Way to go Pierre! > > On Mon, Jul 21, 2025 at 11:04 AM Pierre Salagnac < > pierre.salag...@gmail.com> > wrote: > > > I tried to run smoke tests for 9.9, and they failed several times in a > row > > for me because of flaky logging tests. Since we also hit this failure > > frequently in our CI, I spent some time looking at test execution. > > > > I figured out that TestLogWatcher systematically fails when it is > executed > > after PackageManagerCLITest *in the same JVM*. By default, gradle is > likely > > to spawn several JVMs, and unless you are unlucky, these two tests will > run > > in different ones and you won't see any issue. (I can't say whether > running > > tests from the python script when validating the release has any impact > on > > test ordering and number of JVMs). > > > > On branch_9x, I have a good repro rate (50% ?) locally by running the > > following command. It's still not 100% because gradle can run tests in > any > > order. I'm not sure how to force that. > > > > ./gradlew test --tests PackageManagerCLITest --tests TestLogWatcher > > -Ptests.jvms=1 > > > > (note that's necessary to force the number of forked JVMs to 1 so tests > are > > executed in a row in the same process). > > > > > > Now, the root cause seems pretty obvious. Class PackageTool invokes > > "Configurator.setRootLevel(Level.OFF)" and never reverts that. Any test > > that later looks at was logged is very likely to fail. > > Not sure what is the best path for a fix. Shutting down all logging (even > > in a CLI tool) seems to be a bad practice to me... > > > > > > Le mar. 15 juil. 2025 à 23:53, Houston Putman <hous...@apache.org> a > > écrit : > > > > > Hey everyone, > > > > > > Update on the 9.9 release. > > > > > > There are two issues that popped up after fixing the > > > ParallelHttpShardHandler bug: > > > > > > - https://github.com/apache/solr/pull/3428 (9.x specific, unrelated > > to > > > the bug, but found while beasting tests) > > > - https://github.com/apache/solr/pull/3429 (Fixes bug introduced > into > > > HttpShardHandler) > > > > > > Both should be good to go after quick reviews. > > > > > > The other big issue we are facing is regarding the smoke tester and > > > buildAndPushRelease scripts. Both of these have started having issues > > > regarding running tests that watch logs. (Like TestLogWatcher and > > > RankFieldTest). Both of these watch for certain log events, and both of > > > them fail when running the tests via the python scripts. We can see > this > > > has been happening since May 9th on the Apache Jenkins instances, but > > there > > > is nothing introduced in Solr at the time that would explain it. > > > https://ci-builds.apache.org/job/Solr/job/Solr-Smoketest-9.x/ > > > > > > I am having a very hard time replicating this (outside of doing the > full > > > release process), but I'll hopefully have it solved by the time those 2 > > PRs > > > are approved, merged, and backported. > > > > > > - Houston > > > > > > On Mon, Jul 7, 2025 at 10:14 AM Houston Putman <hous...@apache.org> > > wrote: > > > > > > > Fair enough, I was waiting to cut the branch since we weren't ready > to > > do > > > > the release yet. But I'll start that process now. > > > > > > > > - Houston > > > > > > > > On Mon, Jul 7, 2025 at 6:39 AM David Smiley <dsmi...@apache.org> > > wrote: > > > > > > > >> I noticed recently there seems to be no release branch separation, > > which > > > >> is > > > >> designed to bring about more stability. > > > >> > > > >> I've been waiting on merging > > > >> https://issues.apache.org/jira/browse/SOLR-5707 > > > >> (a 4 digit JIRA from 2014), "Lucene Expressions for Solr" because > the > > > >> Lucene 9.12.2 bug fix includes a bug fix I worked on that > > significantly > > > >> improves the usefulness of SOLR-5707. Notwithstanding a couple > > ignored > > > >> tests (waiting for 9.12.2), that PR is ready for review. Even has a > > new > > > >> ref guide page just for it. > > > >> > > > >> > > > >> On Mon, Jul 7, 2025 at 12:07 AM Houston Putman <hous...@apache.org> > > > >> wrote: > > > >> > > > >> > This is a big blocker: https://github.com/apache/solr/pull/3398 > > > >> > > > > >> > And it needs some more eyes, but hopefully we can finish it out > > early > > > >> this > > > >> > week. > > > >> > > > > >> > We also need to actually do the lucene upgrade which I can do > > tomorrow > > > >> > morning. > > > >> > > > > >> > After those two are done, I'll create an RC. > > > >> > > > > >> > - Houston > > > >> > > > > >> > On Thu, Jul 3, 2025 at 4:26 PM Anshum Gupta <ans...@apache.org> > > > wrote: > > > >> > > > > >> > > Hi everyone, > > > >> > > > > > >> > > Now that the Lucene release is done, is there something that is > > > >> stopping > > > >> > us > > > >> > > from moving forward with this release? If not, let's build an RC > > > early > > > >> > next > > > >> > > week. > > > >> > > > > > >> > > -Anshum > > > >> > > > > > >> > > On Wed, Jun 11, 2025 at 4:12 PM David Smiley < > dsmi...@apache.org> > > > >> wrote: > > > >> > > > > > >> > > > Yes, I _just_ back ported a Lucene fix to 9.12.2 that would > make > > > >> > > > finishing/committing > > > >> https://issues.apache.org/jira/browse/SOLR-5707 > > > >> > > > (using > > > >> > > > Lucene Expressions module in Solr) way more useful. > > > >> > > > > > > >> > > > On Wed, Jun 11, 2025 at 5:54 PM Houston Putman < > > > hous...@apache.org> > > > >> > > wrote: > > > >> > > > > > > >> > > > > We might also want to wait for the next Lucene 9.12.2 > release, > > > >> which > > > >> > > > should > > > >> > > > > hopefully happen soon? > > > >> > > > > > > > >> > > > > - Houston > > > >> > > > > > > > >> > > > > On Wed, Jun 11, 2025 at 8:51 AM Bruno Roustant < > > > >> > > bruno.roust...@gmail.com > > > >> > > > > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > +1 Thanks Houston! > > > >> > > > > > > > > >> > > > > > Le mer. 11 juin 2025 à 00:29, Jan Høydahl < > > > >> jan....@cominvent.com> > > > >> > a > > > >> > > > > écrit > > > >> > > > > > : > > > >> > > > > > > > > >> > > > > > > Great plan. +1 > > > >> > > > > > > > > > >> > > > > > > > 9. juni 2025 kl. 23:14 skrev Houston Putman < > > > >> > hous...@apache.org > > > >> > > >: > > > >> > > > > > > > > > > >> > > > > > > > Hey everyone, > > > >> > > > > > > > > > > >> > > > > > > > It's been a while since the last minor release and the > > > >> > changelog > > > >> > > > > looks > > > >> > > > > > > > pretty good to get the next one in. > > > >> > > > > > > > > > > >> > > > > > > > I volunteer to do it probably sometime next week > unless > > > >> anyone > > > >> > > > > objects. > > > >> > > > > > > > > > > >> > > > > > > > - Houston > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > --------------------------------------------------------------------- > > > >> > > > > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > > >> > > > > > > For additional commands, e-mail: > dev-h...@solr.apache.org > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
