Ok, I think I've got this now. jars from the mavenLocal() repo are used
in-place by gradle, and not cached in `${gradle.lib.dir}`. This requires a
security policy addition to
`gradle/testing/randomization/policies/solr-tests.policy`:
grant {
// Allow reading gradle worker JAR.
permission java.io.FilePermission "${gradle.worker.jar}", "read";
// Allow reading from classpath JARs (resources).
permission java.io.FilePermission "${gradle.user.home}${/}-", "read";
+ // Allow reading lucene jars from mavenLocal repository
+ permission java.io.FilePermission
"${user.home}${/}.m2${/}repository${/}org${/}apache${/}lucene${/}-", "read";
};
This, along with the addition of the mavenLocal() repo and the change to
"9.0.0-SNAPSHOT" lucene version in versions.props (and subsequent update of
versions.lock file via `gradlew --write-locks`), does the trick.
If I were to take a crack at documenting this workflow, any ideas on the
best place to do so?
Michael
On Wed, Jun 2, 2021 at 9:19 AM Michael Gibney <mich...@michaelgibney.net>
wrote:
> Thanks for digging into this! I can confirm that my
> lucene-*-9.0.0-SNAPSHOT.jar files have META-INF/services/ with all the
> expected codecs. I did remove the lucene subfolder from the maven
> repo, re-ran `gradlew mavenToLocalRepo`, manually verified that the
> expected META-INF/services/ listed all codecs, and re-ran the Solr
> build (with `--refresh-dependencies`, and after having added the
> `mavenLocal()` repo, changed `versions.props` to point at
> "9.0.0-SNAPSHO", and run `gradlew --write-locks` to update the
> versions.lock file). I'm still seeing the same exception ...
>
> Michael
>
> On Wed, Jun 2, 2021 at 8:01 AM Uwe Schindler <u...@thetaphi.de> wrote:
> >
> > FYI,
> >
> > I did a recompile locally and checked the JAR files in my local Maven
> repository ("./gradlew mavenToLocalRepo"): All contain META-INF/services
> with all Codecs.
> >
> > I have no idea what goes wrong on your side. Next I will try to setup
> mavenLocal() and a SNAPSHOT dependency, too.
> >
> > -----
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> > > -----Original Message-----
> > > From: Uwe Schindler <u...@thetaphi.de>
> > > Sent: Wednesday, June 2, 2021 12:40 PM
> > > To: dev@solr.apache.org
> > > Subject: RE: Solr/Lucene joint development workflow?
> > >
> > > I'd suggest to remove the whole Lucene subfolder from your local Maven
> repo.
> > > E.g., rm -rf ~/.m2/repository/org/apache/lucene
> > >
> > > Then compile and deploy Lucene again.
> > >
> > > Uwe
> > >
> > > -----
> > > Uwe Schindler
> > > Achterdiek 19, D-28357 Bremen
> > > https://www.thetaphi.de
> > > eMail: u...@thetaphi.de
> > >
> > > > -----Original Message-----
> > > > From: Uwe Schindler <u...@thetaphi.de>
> > > > Sent: Wednesday, June 2, 2021 12:38 PM
> > > > To: 'dev@solr.apache.org' <dev@solr.apache.org>
> > > > Subject: RE: Solr/Lucene joint development workflow?
> > > >
> > > > Hi,
> > > >
> > > > this error helps. It looks like the lucene-core.jar file does not
> have the META-
> > > > INF/services. I wonder how Solr works!
> > > >
> > > > It would be good to figure out what's going on and why the JAR files
> in your
> > > > local repository do not have the required META-INF/services
> subfolder.
> > > >
> > > > Uwe
> > > >
> > > > -----
> > > > Uwe Schindler
> > > > Achterdiek 19, D-28357 Bremen
> > > > https://www.thetaphi.de
> > > > eMail: u...@thetaphi.de
> > > >
> > > > > -----Original Message-----
> > > > > From: Michael Gibney <mich...@michaelgibney.net>
> > > > > Sent: Wednesday, June 2, 2021 6:05 AM
> > > > > To: dev@solr.apache.org
> > > > > Subject: Re: Solr/Lucene joint development workflow?
> > > > >
> > > > > Uwe,
> > > > > Thanks for this advice! I followed this as closely as I could
> > > > > (including the gradle `--refresh-dependencies` flag), and was again
> > > > > able to compile (and compileTest) against mavenLocal. But on
> actually
> > > > > running tests, afaict all tests fail with the same stacktrace
> (copied
> > > > > below). iirc this was the same error I was getting previously with
> the
> > > > > SNAPSHOT build (before the introduction of the prerelease
> dependency).
> > > > > I'd be grateful for any advice on addressing this remaining error
> (and
> > > > > having encountered the error, I continue to be curious whether
> anyone
> > > > > has this workflow or something similar working in practice).
> > > > >
> > > > > David, your suggestion to add "mavenLocalI()" to Gradle's repo list
> > > > > seems reasonable -- perhaps a more viable option now that the main
> > > > > branch is pegged to specific builds (as opposed to a SNAPSHOT)? I'm
> > > > > happy to help contribute to documentation of this process (once I
> have
> > > > > it working!).
> > > > >
> > > > > One other challenge I encountered was determining which commit the
> > > > > current prerelease build is based on. The desired workflow would
> be to
> > > > > update the Lucene feature branch by merging upstream changes as
> far as
> > > > > the commit of the prerelease build (for use as a dependency of the
> > > > > Solr-side feature branch).
> > > > >
> > > > > The exception I'm getting on running tests is:
> > > > >
> > > > > java.lang.ExceptionInInitializerError
> > > > > at
> org.apache.lucene.codecs.Codec.getDefault(Codec.java:141)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetup
> > > > > AndRestoreClassEnv.java:133)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter
> > > > > Rule.java:42)
> > > > > at
> > > > >
> > > >
> > >
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> > > > > ntAdapter.java:36)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClass
> > > > > Name.java:38)
> > > > > at
> > > > >
> > > >
> > > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod
> > > > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > > > > at
> > > > >
> > > >
> > > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod
> > > > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > > > > at
> > > > >
> > > >
> > >
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> > > > > ntAdapter.java:36)
> > > > > at
> > > > >
> > > >
> > >
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> > > > > ntAdapter.java:36)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio
> > > > > nsRequired.java:53)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter
> > > > > Rule.java:43)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja
> > > > > va:44)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn
> > > > > oreAfterMaxFailures.java:60)
> > > > > at
> > > > >
> > > >
> > >
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest
> > > > > Suites.java:47)
> > > > > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > > > > at
> > > > >
> > > >
> > >
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> > > > > ntAdapter.java:36)
> > > > > at
> > > > >
> > > >
> > >
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(
> > > > > ThreadLeakControl.java:370)
> > > > > at
> > > > >
> > > >
> > >
> com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin
> > > > > gTask$0(ThreadLeakControl.java:826)
> > > > > at java.base/java.lang.Thread.run(Thread.java:834)
> > > > >
> > > > > Caused by:
> > > > > java.lang.IllegalArgumentException: An SPI class of type
> > > > > org.apache.lucene.codecs.Codec with name 'Lucene90' does not exist.
> > > > > You need to add the corresponding JAR file supporting this SPI to
> your
> > > > > classpath. The current classpath supports the following names: []
> > > > > at
> > > > >
> org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113)
> > > > > at
> org.apache.lucene.codecs.Codec$Holder.<clinit>(Codec.java:58)
> > > > > ... 19 more
> > > > >
> > > > >
> > > > > On Fri, May 28, 2021 at 4:03 PM David Smiley <dsmi...@apache.org>
> > > wrote:
> > > > > >
> > > > > > To whomever does this first, it would be great if the steps
> could be
> > > written
> > > > > down carefully in more detail so that it can be shared for its
> eventual
> > > > inclusion
> > > > > in /dev-docs
> > > > > >
> > > > > > Maybe "mavenLocalI()" should simply be added to Gradle's repo
> list by
> > > > > default? This will speed up some first-time builds in some
> environments
> > > who
> > > > > already have lots of local Maven dependencies.
> > > > > >
> > > > > > ~ David Smiley
> > > > > > Apache Lucene/Solr Search Developer
> > > > > > http://www.linkedin.com/in/davidwsmiley
> > > > > >
> > > > > >
> > > > > > On Thu, May 27, 2021 at 12:09 PM Uwe Schindler <u...@thetaphi.de>
> > > > > wrote:
> > > > > >>
> > > > > >> Hi Michael,
> > > > > >>
> > > > > >> since yesterday, the Lucene dependency is no longer a snapshot
> one, so
> > > > > parts of your mail are no longer valid. But the worksflow is very
> similar.
> > > > > >>
> > > > > >> If the Solr team wants to use a newer preview build of Lucene,
> there is a
> > > > > workflow using ASF Jenkins that can build a new "prerelease"
> release of the
> > > > > Lucene main branch and deploy it on some ASF data dump server as
> maven
> > > > > repository.
> > > > > >>
> > > > > >> The Gradle build of Solr is referring to the recent repository
> and picks a
> > > > > specific version (the build number of ASF jenkins).
> > > > > >>
> > > > > >> For joint development, I'd suggest to work like this:
> > > > > >> - Change the dependency in the gradle versions.props file to
> SNAPSHOT
> > > > > (only local, don't submit this as PR)
> > > > > >> - Add mavenLocal() as suggested in your mail (only local, don't
> submit
> > > this
> > > > as
> > > > > PR)
> > > > > >> - develop changes in lucene and install them in local repo
> > > > > >> - use them from solr through the snapshot dependency. BUT: make
> sure
> > > > the
> > > > > snapshot is updated, this can be enforced by passing a gradlew
> command
> > > line
> > > > > parameter to redownload all dependencies.
> > > > > >> - once you are done with joint development create pull requests
> in both
> > > > > repositories
> > > > > >> - Lucene's change should be merged first. Once this is done and
> all tests
> > > > > pass, ask a committer to trigger the Jenkins job to build a new
> prerelease
> > > > > >> - Update the Solr pull request and enter the new repository
> coordinates
> > > > > created by Jenkins (needs to be done in 2 files, see below).
> > > > > >>
> > > > > >> For more details how to update Lucene dependencies in Solr, run:
> > > > > >> $ gradlew :helpDeps
> > > > > >>
> > > > > >> Uwe
> > > > > >>
> > > > > >> -----
> > > > > >> Uwe Schindler
> > > > > >> Achterdiek 19, D-28357 Bremen
> > > > > >> https://www.thetaphi.de
> > > > > >> eMail: u...@thetaphi.de
> > > > > >>
> > > > > >> > -----Original Message-----
> > > > > >> > From: Michael Gibney <mich...@michaelgibney.net>
> > > > > >> > Sent: Wednesday, May 26, 2021 8:02 PM
> > > > > >> > To: dev@solr.apache.org
> > > > > >> > Subject: Solr/Lucene joint development workflow?
> > > > > >> >
> > > > > >> > I'm working on some features that involve changes to both
> Lucene and
> > > > > >> > Solr. Post-TLP-split, I'm wondering whether anyone has
> recommended
> > > > > >> > techniques to handle this kind of situation.
> > > > > >> >
> > > > > >> > Ideally one would work on Lucene changes, get them merged,
> and then
> > > > > >> > proceed with Solr development; but realistically even if this
> were as
> > > > > >> > straightforward in practice as it sounds in principal, there
> are cases
> > > > > >> > where one would still want to develop in parallel.
> > > > > >> >
> > > > > >> > I haven't been able to find any documented recommendation on
> this
> > > > > >> > subject. It's possible to have a locally built Lucene
> snapshot (via
> > > > > >> > `gradlew mavenToLocalRepo`); but I was only able to actually
> _build_
> > > > > >> > Solr against the local Lucene artifact by adding
> `mavenLocal()` to the
> > > > > >> > `allprojects/repositories` block in `gradle/defaults.gradle`
> -- and I
> > > > > >> > have yet to figure a way get the local Lucene artifact on the
> test
> > > > > >> > classpath (so I'm as yet unable to run Solr tests that depend
> on
> > > > > >> > unmerged upstream changes to Lucene).
> > > > > >> >
> > > > > >> > It's also possible that the partially-functional approach
> described
> > > > > >> > above will have to change now that Solr main depends on a
> specific
> > > > > >> > Lucene snapshot version.
> > > > > >> >
> > > > > >> > Is anybody doing something like this? Or perhaps I'm asking
> the wrong
> > > > > >> > question? I can think of solutions that involve setting up my
> own
> > > > > >> > maven repository, to which I publish my own pinned versions of
> > > Lucene,
> > > > > >> > and refer to such pinned versions/repo as part of a given Solr
> > > > > >> > "patch". But that feels both half-baked _and_ bloated, so I
> don't want
> > > > > >> > to go down that road unless I feel convinced there's no better
> > > > > >> > alternative.
> > > > > >> >
> > > > > >> > Michael
> > > > > >> >
> > > > > >> >
> ---------------------------------------------------------------------
> > > > > >> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > > > >> > For additional commands, e-mail: dev-h...@solr.apache.org
> > > > > >>
> > > > > >>
> > > > > >>
> ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > > > >> For additional commands, e-mail: dev-h...@solr.apache.org
> > > > > >>
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > > > For additional commands, e-mail: dev-h...@solr.apache.org
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > > For additional commands, e-mail: dev-h...@solr.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
> > For additional commands, e-mail: dev-h...@solr.apache.org
> >
>
