Re: Solr/Lucene joint development workflow?
Can we commit the security policy addition independently?
On Thu, Jun 3, 2021 at 2:20 PM Michael Gibney
wrote:
> Ok, I think I've got this now. jars from the mavenLocal() repo are used
> in-place by gradle, and not cached in `${gradle.lib.dir}`. This requires a
> security policy addition to
> `gradle/testing/randomization/policies/solr-tests.policy`:
>
> grant {
>// Allow reading gradle worker JAR.
>permission java.io.FilePermission "${gradle.worker.jar}", "read";
>// Allow reading from classpath JARs (resources).
>permission java.io.FilePermission "${gradle.user.home}${/}-", "read";
> + // Allow reading lucene jars from mavenLocal repository
> + permission java.io.FilePermission
> "${user.home}${/}.m2${/}repository${/}org${/}apache${/}lucene${/}-", "read";
> };
>
> This, along with the addition of the mavenLocal() repo and the change to
> "9.0.0-SNAPSHOT" lucene version in versions.props (and subsequent update of
> versions.lock file via `gradlew --write-locks`), does the trick.
>
> If I were to take a crack at documenting this workflow, any ideas on the
> best place to do so?
>
> Michael
>
>
> On Wed, Jun 2, 2021 at 9:19 AM Michael Gibney
> wrote:
>
>> Thanks for digging into this! I can confirm that my
>> lucene-*-9.0.0-SNAPSHOT.jar files have META-INF/services/ with all the
>> expected codecs. I did remove the lucene subfolder from the maven
>> repo, re-ran `gradlew mavenToLocalRepo`, manually verified that the
>> expected META-INF/services/ listed all codecs, and re-ran the Solr
>> build (with `--refresh-dependencies`, and after having added the
>> `mavenLocal()` repo, changed `versions.props` to point at
>> "9.0.0-SNAPSHO", and run `gradlew --write-locks` to update the
>> versions.lock file). I'm still seeing the same exception ...
>>
>> Michael
>>
>> On Wed, Jun 2, 2021 at 8:01 AM Uwe Schindler wrote:
>> >
>> > FYI,
>> >
>> > I did a recompile locally and checked the JAR files in my local Maven
>> repository ("./gradlew mavenToLocalRepo"): All contain META-INF/services
>> with all Codecs.
>> >
>> > I have no idea what goes wrong on your side. Next I will try to setup
>> mavenLocal() and a SNAPSHOT dependency, too.
>> >
>> > -
>> > Uwe Schindler
>> > Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > https://www.thetaphi.de
>> > eMail: u...@thetaphi.de
>> >
>> > > -Original Message-
>> > > From: Uwe Schindler
>> > > Sent: Wednesday, June 2, 2021 12:40 PM
>> > > To: dev@solr.apache.org
>> > > Subject: RE: Solr/Lucene joint development workflow?
>> > >
>> > > I'd suggest to remove the whole Lucene subfolder from your local
>> Maven repo.
>> > > E.g., rm -rf ~/.m2/repository/org/apache/lucene
>> > >
>> > > Then compile and deploy Lucene again.
>> > >
>> > > Uwe
>> > >
>> > > -
>> > > Uwe Schindler
>> > > Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > > https://www.thetaphi.de
>> > > eMail: u...@thetaphi.de
>> > >
>> > > > -Original Message-
>> > > > From: Uwe Schindler
>> > > > Sent: Wednesday, June 2, 2021 12:38 PM
>> > > > To: 'dev@solr.apache.org'
>> > > > Subject: RE: Solr/Lucene joint development workflow?
>> > > >
>> > > > Hi,
>> > > >
>> > > > this error helps. It looks like the lucene-core.jar file does not
>> have the META-
>> > > > INF/services. I wonder how Solr works!
>> > > >
>> > > > It would be good to figure out what's going on and why the JAR
>> files in your
>> > > > local repository do not have the required META-INF/services
>> subfolder.
>> > > >
>> > > > Uwe
>> > > >
>> > > > -
>> > > > Uwe Schindler
>> > > > Achterdiek 19, D-28357 Bremen
>> <https://www.google.com/maps/search/Achterdiek+19,+D-28357+Bremen?entry=gmail&source=g>
>> > > > https://www.thetaphi.de
>> > > > eMail: u...@thetaphi.de
>> > > >
>> > > > > -Original
Re: Solr/Lucene joint development workflow?
Ok, I think I've got this now. jars from the mavenLocal() repo are used
in-place by gradle, and not cached in `${gradle.lib.dir}`. This requires a
security policy addition to
`gradle/testing/randomization/policies/solr-tests.policy`:
grant {
// Allow reading gradle worker JAR.
permission java.io.FilePermission "${gradle.worker.jar}", "read";
// Allow reading from classpath JARs (resources).
permission java.io.FilePermission "${gradle.user.home}${/}-", "read";
+ // Allow reading lucene jars from mavenLocal repository
+ permission java.io.FilePermission
"${user.home}${/}.m2${/}repository${/}org${/}apache${/}lucene${/}-", "read";
};
This, along with the addition of the mavenLocal() repo and the change to
"9.0.0-SNAPSHOT" lucene version in versions.props (and subsequent update of
versions.lock file via `gradlew --write-locks`), does the trick.
If I were to take a crack at documenting this workflow, any ideas on the
best place to do so?
Michael
On Wed, Jun 2, 2021 at 9:19 AM Michael Gibney
wrote:
> Thanks for digging into this! I can confirm that my
> lucene-*-9.0.0-SNAPSHOT.jar files have META-INF/services/ with all the
> expected codecs. I did remove the lucene subfolder from the maven
> repo, re-ran `gradlew mavenToLocalRepo`, manually verified that the
> expected META-INF/services/ listed all codecs, and re-ran the Solr
> build (with `--refresh-dependencies`, and after having added the
> `mavenLocal()` repo, changed `versions.props` to point at
> "9.0.0-SNAPSHO", and run `gradlew --write-locks` to update the
> versions.lock file). I'm still seeing the same exception ...
>
> Michael
>
> On Wed, Jun 2, 2021 at 8:01 AM Uwe Schindler wrote:
> >
> > FYI,
> >
> > I did a recompile locally and checked the JAR files in my local Maven
> repository ("./gradlew mavenToLocalRepo"): All contain META-INF/services
> with all Codecs.
> >
> > I have no idea what goes wrong on your side. Next I will try to setup
> mavenLocal() and a SNAPSHOT dependency, too.
> >
> > -
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> > > -Original Message-
> > > From: Uwe Schindler
> > > Sent: Wednesday, June 2, 2021 12:40 PM
> > > To: dev@solr.apache.org
> > > Subject: RE: Solr/Lucene joint development workflow?
> > >
> > > I'd suggest to remove the whole Lucene subfolder from your local Maven
> repo.
> > > E.g., rm -rf ~/.m2/repository/org/apache/lucene
> > >
> > > Then compile and deploy Lucene again.
> > >
> > > Uwe
> > >
> > > -
> > > Uwe Schindler
> > > Achterdiek 19, D-28357 Bremen
> > > https://www.thetaphi.de
> > > eMail: u...@thetaphi.de
> > >
> > > > -Original Message-
> > > > From: Uwe Schindler
> > > > Sent: Wednesday, June 2, 2021 12:38 PM
> > > > To: 'dev@solr.apache.org'
> > > > Subject: RE: Solr/Lucene joint development workflow?
> > > >
> > > > Hi,
> > > >
> > > > this error helps. It looks like the lucene-core.jar file does not
> have the META-
> > > > INF/services. I wonder how Solr works!
> > > >
> > > > It would be good to figure out what's going on and why the JAR files
> in your
> > > > local repository do not have the required META-INF/services
> subfolder.
> > > >
> > > > Uwe
> > > >
> > > > -
> > > > Uwe Schindler
> > > > Achterdiek 19, D-28357 Bremen
> > > > https://www.thetaphi.de
> > > > eMail: u...@thetaphi.de
> > > >
> > > > > -Original Message-
> > > > > From: Michael Gibney
> > > > > Sent: Wednesday, June 2, 2021 6:05 AM
> > > > > To: dev@solr.apache.org
> > > > > Subject: Re: Solr/Lucene joint development workflow?
> > > > >
> > > > > Uwe,
> > > > > Thanks for this advice! I followed this as closely as I could
> > > > > (including the gradle `--refresh-dependencies` flag), and was again
> > > > > able to compile (and compileTest) against mavenLocal. But on
> actually
> > > > > running tests, afaict all tests fail with the same stacktrace
> (copied
> > > > > below). iirc this was the same error I was getting previously with
> the
> > > > > SNAPSHOT build (before the introduction of the prerelease
&
Re: Solr/Lucene joint development workflow?
Thanks for digging into this! I can confirm that my
lucene-*-9.0.0-SNAPSHOT.jar files have META-INF/services/ with all the
expected codecs. I did remove the lucene subfolder from the maven
repo, re-ran `gradlew mavenToLocalRepo`, manually verified that the
expected META-INF/services/ listed all codecs, and re-ran the Solr
build (with `--refresh-dependencies`, and after having added the
`mavenLocal()` repo, changed `versions.props` to point at
"9.0.0-SNAPSHO", and run `gradlew --write-locks` to update the
versions.lock file). I'm still seeing the same exception ...
Michael
On Wed, Jun 2, 2021 at 8:01 AM Uwe Schindler wrote:
>
> FYI,
>
> I did a recompile locally and checked the JAR files in my local Maven
> repository ("./gradlew mavenToLocalRepo"): All contain META-INF/services with
> all Codecs.
>
> I have no idea what goes wrong on your side. Next I will try to setup
> mavenLocal() and a SNAPSHOT dependency, too.
>
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> > -Original Message-
> > From: Uwe Schindler
> > Sent: Wednesday, June 2, 2021 12:40 PM
> > To: dev@solr.apache.org
> > Subject: RE: Solr/Lucene joint development workflow?
> >
> > I'd suggest to remove the whole Lucene subfolder from your local Maven repo.
> > E.g., rm -rf ~/.m2/repository/org/apache/lucene
> >
> > Then compile and deploy Lucene again.
> >
> > Uwe
> >
> > -
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> > > -Original Message-
> > > From: Uwe Schindler
> > > Sent: Wednesday, June 2, 2021 12:38 PM
> > > To: 'dev@solr.apache.org'
> > > Subject: RE: Solr/Lucene joint development workflow?
> > >
> > > Hi,
> > >
> > > this error helps. It looks like the lucene-core.jar file does not have
> > > the META-
> > > INF/services. I wonder how Solr works!
> > >
> > > It would be good to figure out what's going on and why the JAR files in
> > > your
> > > local repository do not have the required META-INF/services subfolder.
> > >
> > > Uwe
> > >
> > > -
> > > Uwe Schindler
> > > Achterdiek 19, D-28357 Bremen
> > > https://www.thetaphi.de
> > > eMail: u...@thetaphi.de
> > >
> > > > -Original Message-
> > > > From: Michael Gibney
> > > > Sent: Wednesday, June 2, 2021 6:05 AM
> > > > To: dev@solr.apache.org
> > > > Subject: Re: Solr/Lucene joint development workflow?
> > > >
> > > > Uwe,
> > > > Thanks for this advice! I followed this as closely as I could
> > > > (including the gradle `--refresh-dependencies` flag), and was again
> > > > able to compile (and compileTest) against mavenLocal. But on actually
> > > > running tests, afaict all tests fail with the same stacktrace (copied
> > > > below). iirc this was the same error I was getting previously with the
> > > > SNAPSHOT build (before the introduction of the prerelease dependency).
> > > > I'd be grateful for any advice on addressing this remaining error (and
> > > > having encountered the error, I continue to be curious whether anyone
> > > > has this workflow or something similar working in practice).
> > > >
> > > > David, your suggestion to add "mavenLocalI()" to Gradle's repo list
> > > > seems reasonable -- perhaps a more viable option now that the main
> > > > branch is pegged to specific builds (as opposed to a SNAPSHOT)? I'm
> > > > happy to help contribute to documentation of this process (once I have
> > > > it working!).
> > > >
> > > > One other challenge I encountered was determining which commit the
> > > > current prerelease build is based on. The desired workflow would be to
> > > > update the Lucene feature branch by merging upstream changes as far as
> > > > the commit of the prerelease build (for use as a dependency of the
> > > > Solr-side feature branch).
> > > >
> > > > The exception I'm getting on running tests is:
> > > >
> > > > java.lang.ExceptionInInitializerError
> > > > at org.apache.lucene.codecs.Codec.getDefault(Codec.java:141)
> > > > at
> > > >
> > >
> > org.apache.l
RE: Solr/Lucene joint development workflow?
FYI,
I did a recompile locally and checked the JAR files in my local Maven
repository ("./gradlew mavenToLocalRepo"): All contain META-INF/services with
all Codecs.
I have no idea what goes wrong on your side. Next I will try to setup
mavenLocal() and a SNAPSHOT dependency, too.
-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de
> -Original Message-
> From: Uwe Schindler
> Sent: Wednesday, June 2, 2021 12:40 PM
> To: dev@solr.apache.org
> Subject: RE: Solr/Lucene joint development workflow?
>
> I'd suggest to remove the whole Lucene subfolder from your local Maven repo.
> E.g., rm -rf ~/.m2/repository/org/apache/lucene
>
> Then compile and deploy Lucene again.
>
> Uwe
>
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> > -Original Message-
> > From: Uwe Schindler
> > Sent: Wednesday, June 2, 2021 12:38 PM
> > To: 'dev@solr.apache.org'
> > Subject: RE: Solr/Lucene joint development workflow?
> >
> > Hi,
> >
> > this error helps. It looks like the lucene-core.jar file does not have the
> > META-
> > INF/services. I wonder how Solr works!
> >
> > It would be good to figure out what's going on and why the JAR files in your
> > local repository do not have the required META-INF/services subfolder.
> >
> > Uwe
> >
> > -
> > Uwe Schindler
> > Achterdiek 19, D-28357 Bremen
> > https://www.thetaphi.de
> > eMail: u...@thetaphi.de
> >
> > > -Original Message-
> > > From: Michael Gibney
> > > Sent: Wednesday, June 2, 2021 6:05 AM
> > > To: dev@solr.apache.org
> > > Subject: Re: Solr/Lucene joint development workflow?
> > >
> > > Uwe,
> > > Thanks for this advice! I followed this as closely as I could
> > > (including the gradle `--refresh-dependencies` flag), and was again
> > > able to compile (and compileTest) against mavenLocal. But on actually
> > > running tests, afaict all tests fail with the same stacktrace (copied
> > > below). iirc this was the same error I was getting previously with the
> > > SNAPSHOT build (before the introduction of the prerelease dependency).
> > > I'd be grateful for any advice on addressing this remaining error (and
> > > having encountered the error, I continue to be curious whether anyone
> > > has this workflow or something similar working in practice).
> > >
> > > David, your suggestion to add "mavenLocalI()" to Gradle's repo list
> > > seems reasonable -- perhaps a more viable option now that the main
> > > branch is pegged to specific builds (as opposed to a SNAPSHOT)? I'm
> > > happy to help contribute to documentation of this process (once I have
> > > it working!).
> > >
> > > One other challenge I encountered was determining which commit the
> > > current prerelease build is based on. The desired workflow would be to
> > > update the Lucene feature branch by merging upstream changes as far as
> > > the commit of the prerelease build (for use as a dependency of the
> > > Solr-side feature branch).
> > >
> > > The exception I'm getting on running tests is:
> > >
> > > java.lang.ExceptionInInitializerError
> > > at org.apache.lucene.codecs.Codec.getDefault(Codec.java:141)
> > > at
> > >
> >
> org.apache.lucene.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetup
> > > AndRestoreClassEnv.java:133)
> > > at
> > >
> >
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter
> > > Rule.java:42)
> > > at
> > >
> >
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> > > ntAdapter.java:36)
> > > at
> > >
> >
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClass
> > > Name.java:38)
> > > at
> > >
> >
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod
> > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > > at
> > >
> >
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod
> > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > > at
> > >
> >
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> &
RE: Solr/Lucene joint development workflow?
I'd suggest to remove the whole Lucene subfolder from your local Maven repo. E.g., rm -rf ~/.m2/repository/org/apache/lucene Then compile and deploy Lucene again. Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Uwe Schindler > Sent: Wednesday, June 2, 2021 12:38 PM > To: 'dev@solr.apache.org' > Subject: RE: Solr/Lucene joint development workflow? > > Hi, > > this error helps. It looks like the lucene-core.jar file does not have the > META- > INF/services. I wonder how Solr works! > > It would be good to figure out what's going on and why the JAR files in your > local repository do not have the required META-INF/services subfolder. > > Uwe > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Michael Gibney > > Sent: Wednesday, June 2, 2021 6:05 AM > > To: dev@solr.apache.org > > Subject: Re: Solr/Lucene joint development workflow? > > > > Uwe, > > Thanks for this advice! I followed this as closely as I could > > (including the gradle `--refresh-dependencies` flag), and was again > > able to compile (and compileTest) against mavenLocal. But on actually > > running tests, afaict all tests fail with the same stacktrace (copied > > below). iirc this was the same error I was getting previously with the > > SNAPSHOT build (before the introduction of the prerelease dependency). > > I'd be grateful for any advice on addressing this remaining error (and > > having encountered the error, I continue to be curious whether anyone > > has this workflow or something similar working in practice). > > > > David, your suggestion to add "mavenLocalI()" to Gradle's repo list > > seems reasonable -- perhaps a more viable option now that the main > > branch is pegged to specific builds (as opposed to a SNAPSHOT)? I'm > > happy to help contribute to documentation of this process (once I have > > it working!). > > > > One other challenge I encountered was determining which commit the > > current prerelease build is based on. The desired workflow would be to > > update the Lucene feature branch by merging upstream changes as far as > > the commit of the prerelease build (for use as a dependency of the > > Solr-side feature branch). > > > > The exception I'm getting on running tests is: > > > > java.lang.ExceptionInInitializerError > > at org.apache.lucene.codecs.Codec.getDefault(Codec.java:141) > > at > > > org.apache.lucene.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetup > > AndRestoreClassEnv.java:133) > > at > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > Rule.java:42) > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > at > > > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClass > > Name.java:38) > > at > > > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod > > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > > at > > > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > > nsRequired.java:53) > > at > > > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > > Rule.java:43) > > at > > > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > > va:44) > > at > > > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > > oreAfterMaxFailures.java:60) > > at > > > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > > Suites.java:47) > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at > > > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > > ntAdapter.java:36) > >
RE: Solr/Lucene joint development workflow?
Hi, this error helps. It looks like the lucene-core.jar file does not have the META-INF/services. I wonder how Solr works! It would be good to figure out what's going on and why the JAR files in your local repository do not have the required META-INF/services subfolder. Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Michael Gibney > Sent: Wednesday, June 2, 2021 6:05 AM > To: dev@solr.apache.org > Subject: Re: Solr/Lucene joint development workflow? > > Uwe, > Thanks for this advice! I followed this as closely as I could > (including the gradle `--refresh-dependencies` flag), and was again > able to compile (and compileTest) against mavenLocal. But on actually > running tests, afaict all tests fail with the same stacktrace (copied > below). iirc this was the same error I was getting previously with the > SNAPSHOT build (before the introduction of the prerelease dependency). > I'd be grateful for any advice on addressing this remaining error (and > having encountered the error, I continue to be curious whether anyone > has this workflow or something similar working in practice). > > David, your suggestion to add "mavenLocalI()" to Gradle's repo list > seems reasonable -- perhaps a more viable option now that the main > branch is pegged to specific builds (as opposed to a SNAPSHOT)? I'm > happy to help contribute to documentation of this process (once I have > it working!). > > One other challenge I encountered was determining which commit the > current prerelease build is based on. The desired workflow would be to > update the Lucene feature branch by merging upstream changes as far as > the commit of the prerelease build (for use as a dependency of the > Solr-side feature branch). > > The exception I'm getting on running tests is: > > java.lang.ExceptionInInitializerError > at org.apache.lucene.codecs.Codec.getDefault(Codec.java:141) > at > org.apache.lucene.util.TestRuleSetupAndRestoreClassEnv.before(TestRuleSetup > AndRestoreClassEnv.java:133) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > Rule.java:42) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClass > Name.java:38) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethod > sRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertio > nsRequired.java:53) > at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfter > Rule.java:43) > at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.ja > va:44) > at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgn > oreAfterMaxFailures.java:60) > at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTest > Suites.java:47) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme > ntAdapter.java:36) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run( > ThreadLeakControl.java:370) > at > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutin > gTask$0(ThreadLeakControl.java:826) > at java.base/java.lang.Thread.run(Thread.java:834) > > Caused by: > java.lang.IllegalArgumentException: An SPI class of type > org.apache.lucene.codecs.Codec with name 'Lucene90' does not exist. > You need to add the corresponding JAR file supporting this SPI to your > classpath. The current classpath supports the following names: [] > at > org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:113) > at org.apache.lucene.codecs.Codec$Holder.(Codec.java:58) > ... 19 more > > > On Fri, May 28, 2021 at 4:03 PM David Smiley wrote: > > > > To whomever does this first, it would be great if the steps could be written > down caref
Re: Solr/Lucene joint development workflow?
is referring to the recent repository and picks a >> specific version (the build number of ASF jenkins). >> >> For joint development, I'd suggest to work like this: >> - Change the dependency in the gradle versions.props file to SNAPSHOT (only >> local, don't submit this as PR) >> - Add mavenLocal() as suggested in your mail (only local, don't submit this >> as PR) >> - develop changes in lucene and install them in local repo >> - use them from solr through the snapshot dependency. BUT: make sure the >> snapshot is updated, this can be enforced by passing a gradlew command line >> parameter to redownload all dependencies. >> - once you are done with joint development create pull requests in both >> repositories >> - Lucene's change should be merged first. Once this is done and all tests >> pass, ask a committer to trigger the Jenkins job to build a new prerelease >> - Update the Solr pull request and enter the new repository coordinates >> created by Jenkins (needs to be done in 2 files, see below). >> >> For more details how to update Lucene dependencies in Solr, run: >> $ gradlew :helpDeps >> >> Uwe >> >> - >> Uwe Schindler >> Achterdiek 19, D-28357 Bremen >> https://www.thetaphi.de >> eMail: u...@thetaphi.de >> >> > -Original Message- >> > From: Michael Gibney >> > Sent: Wednesday, May 26, 2021 8:02 PM >> > To: dev@solr.apache.org >> > Subject: Solr/Lucene joint development workflow? >> > >> > I'm working on some features that involve changes to both Lucene and >> > Solr. Post-TLP-split, I'm wondering whether anyone has recommended >> > techniques to handle this kind of situation. >> > >> > Ideally one would work on Lucene changes, get them merged, and then >> > proceed with Solr development; but realistically even if this were as >> > straightforward in practice as it sounds in principal, there are cases >> > where one would still want to develop in parallel. >> > >> > I haven't been able to find any documented recommendation on this >> > subject. It's possible to have a locally built Lucene snapshot (via >> > `gradlew mavenToLocalRepo`); but I was only able to actually _build_ >> > Solr against the local Lucene artifact by adding `mavenLocal()` to the >> > `allprojects/repositories` block in `gradle/defaults.gradle` -- and I >> > have yet to figure a way get the local Lucene artifact on the test >> > classpath (so I'm as yet unable to run Solr tests that depend on >> > unmerged upstream changes to Lucene). >> > >> > It's also possible that the partially-functional approach described >> > above will have to change now that Solr main depends on a specific >> > Lucene snapshot version. >> > >> > Is anybody doing something like this? Or perhaps I'm asking the wrong >> > question? I can think of solutions that involve setting up my own >> > maven repository, to which I publish my own pinned versions of Lucene, >> > and refer to such pinned versions/repo as part of a given Solr >> > "patch". But that feels both half-baked _and_ bloated, so I don't want >> > to go down that road unless I feel convinced there's no better >> > alternative. >> > >> > Michael >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> > For additional commands, e-mail: dev-h...@solr.apache.org >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> For additional commands, e-mail: dev-h...@solr.apache.org >> - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
Re: Solr/Lucene joint development workflow?
To whomever does this first, it would be great if the steps could be written down carefully in more detail so that it can be shared for its eventual inclusion in /dev-docs Maybe "mavenLocalI()" should simply be added to Gradle's repo list by default? This will speed up some first-time builds in some environments who already have lots of local Maven dependencies. ~ David Smiley Apache Lucene/Solr Search Developer http://www.linkedin.com/in/davidwsmiley On Thu, May 27, 2021 at 12:09 PM Uwe Schindler wrote: > Hi Michael, > > since yesterday, the Lucene dependency is no longer a snapshot one, so > parts of your mail are no longer valid. But the worksflow is very similar. > > If the Solr team wants to use a newer preview build of Lucene, there is a > workflow using ASF Jenkins that can build a new "prerelease" release of the > Lucene main branch and deploy it on some ASF data dump server as maven > repository. > > The Gradle build of Solr is referring to the recent repository and picks a > specific version (the build number of ASF jenkins). > > For joint development, I'd suggest to work like this: > - Change the dependency in the gradle versions.props file to SNAPSHOT > (only local, don't submit this as PR) > - Add mavenLocal() as suggested in your mail (only local, don't submit > this as PR) > - develop changes in lucene and install them in local repo > - use them from solr through the snapshot dependency. BUT: make sure the > snapshot is updated, this can be enforced by passing a gradlew command line > parameter to redownload all dependencies. > - once you are done with joint development create pull requests in both > repositories > - Lucene's change should be merged first. Once this is done and all tests > pass, ask a committer to trigger the Jenkins job to build a new prerelease > - Update the Solr pull request and enter the new repository coordinates > created by Jenkins (needs to be done in 2 files, see below). > > For more details how to update Lucene dependencies in Solr, run: > $ gradlew :helpDeps > > Uwe > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > > > -----Original Message- > > From: Michael Gibney > > Sent: Wednesday, May 26, 2021 8:02 PM > > To: dev@solr.apache.org > > Subject: Solr/Lucene joint development workflow? > > > > I'm working on some features that involve changes to both Lucene and > > Solr. Post-TLP-split, I'm wondering whether anyone has recommended > > techniques to handle this kind of situation. > > > > Ideally one would work on Lucene changes, get them merged, and then > > proceed with Solr development; but realistically even if this were as > > straightforward in practice as it sounds in principal, there are cases > > where one would still want to develop in parallel. > > > > I haven't been able to find any documented recommendation on this > > subject. It's possible to have a locally built Lucene snapshot (via > > `gradlew mavenToLocalRepo`); but I was only able to actually _build_ > > Solr against the local Lucene artifact by adding `mavenLocal()` to the > > `allprojects/repositories` block in `gradle/defaults.gradle` -- and I > > have yet to figure a way get the local Lucene artifact on the test > > classpath (so I'm as yet unable to run Solr tests that depend on > > unmerged upstream changes to Lucene). > > > > It's also possible that the partially-functional approach described > > above will have to change now that Solr main depends on a specific > > Lucene snapshot version. > > > > Is anybody doing something like this? Or perhaps I'm asking the wrong > > question? I can think of solutions that involve setting up my own > > maven repository, to which I publish my own pinned versions of Lucene, > > and refer to such pinned versions/repo as part of a given Solr > > "patch". But that feels both half-baked _and_ bloated, so I don't want > > to go down that road unless I feel convinced there's no better > > alternative. > > > > Michael > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > For additional commands, e-mail: dev-h...@solr.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org > >
RE: Solr/Lucene joint development workflow?
Hi Michael again, as a small addon: Due to the previously descibed changes, it is no longer needed to do the development in parallel (unless you want to check some compatibility or have a wider test-coverage). As Solr is now using a pinned "prerelease" (from ASF jenkins build number), you can add breaking changes to Lucene and Solr won't suddenly fail. Once you are ready to port the changes to Solr, you can trigger a new "prerelease" build on ASF jenkins and use it for development. So we are now completely decouped! Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Uwe Schindler > Sent: Thursday, May 27, 2021 6:09 PM > To: dev@solr.apache.org > Subject: RE: Solr/Lucene joint development workflow? > > Hi Michael, > > since yesterday, the Lucene dependency is no longer a snapshot one, so parts > of > your mail are no longer valid. But the worksflow is very similar. > > If the Solr team wants to use a newer preview build of Lucene, there is a > workflow using ASF Jenkins that can build a new "prerelease" release of the > Lucene main branch and deploy it on some ASF data dump server as maven > repository. > > The Gradle build of Solr is referring to the recent repository and picks a > specific > version (the build number of ASF jenkins). > > For joint development, I'd suggest to work like this: > - Change the dependency in the gradle versions.props file to SNAPSHOT (only > local, don't submit this as PR) > - Add mavenLocal() as suggested in your mail (only local, don't submit this as > PR) > - develop changes in lucene and install them in local repo > - use them from solr through the snapshot dependency. BUT: make sure the > snapshot is updated, this can be enforced by passing a gradlew command line > parameter to redownload all dependencies. > - once you are done with joint development create pull requests in both > repositories > - Lucene's change should be merged first. Once this is done and all tests > pass, > ask a committer to trigger the Jenkins job to build a new prerelease > - Update the Solr pull request and enter the new repository coordinates > created > by Jenkins (needs to be done in 2 files, see below). > > For more details how to update Lucene dependencies in Solr, run: > $ gradlew :helpDeps > > Uwe > > - > Uwe Schindler > Achterdiek 19, D-28357 Bremen > https://www.thetaphi.de > eMail: u...@thetaphi.de > > > -Original Message- > > From: Michael Gibney > > Sent: Wednesday, May 26, 2021 8:02 PM > > To: dev@solr.apache.org > > Subject: Solr/Lucene joint development workflow? > > > > I'm working on some features that involve changes to both Lucene and > > Solr. Post-TLP-split, I'm wondering whether anyone has recommended > > techniques to handle this kind of situation. > > > > Ideally one would work on Lucene changes, get them merged, and then > > proceed with Solr development; but realistically even if this were as > > straightforward in practice as it sounds in principal, there are cases > > where one would still want to develop in parallel. > > > > I haven't been able to find any documented recommendation on this > > subject. It's possible to have a locally built Lucene snapshot (via > > `gradlew mavenToLocalRepo`); but I was only able to actually _build_ > > Solr against the local Lucene artifact by adding `mavenLocal()` to the > > `allprojects/repositories` block in `gradle/defaults.gradle` -- and I > > have yet to figure a way get the local Lucene artifact on the test > > classpath (so I'm as yet unable to run Solr tests that depend on > > unmerged upstream changes to Lucene). > > > > It's also possible that the partially-functional approach described > > above will have to change now that Solr main depends on a specific > > Lucene snapshot version. > > > > Is anybody doing something like this? Or perhaps I'm asking the wrong > > question? I can think of solutions that involve setting up my own > > maven repository, to which I publish my own pinned versions of Lucene, > > and refer to such pinned versions/repo as part of a given Solr > > "patch". But that feels both half-baked _and_ bloated, so I don't want > > to go down that road unless I feel convinced there's no better > > alternative. > > > > Michael > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > For additional commands, e-mail: dev-h...@solr.apache.org > > > - > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
RE: Solr/Lucene joint development workflow?
Hi Michael, since yesterday, the Lucene dependency is no longer a snapshot one, so parts of your mail are no longer valid. But the worksflow is very similar. If the Solr team wants to use a newer preview build of Lucene, there is a workflow using ASF Jenkins that can build a new "prerelease" release of the Lucene main branch and deploy it on some ASF data dump server as maven repository. The Gradle build of Solr is referring to the recent repository and picks a specific version (the build number of ASF jenkins). For joint development, I'd suggest to work like this: - Change the dependency in the gradle versions.props file to SNAPSHOT (only local, don't submit this as PR) - Add mavenLocal() as suggested in your mail (only local, don't submit this as PR) - develop changes in lucene and install them in local repo - use them from solr through the snapshot dependency. BUT: make sure the snapshot is updated, this can be enforced by passing a gradlew command line parameter to redownload all dependencies. - once you are done with joint development create pull requests in both repositories - Lucene's change should be merged first. Once this is done and all tests pass, ask a committer to trigger the Jenkins job to build a new prerelease - Update the Solr pull request and enter the new repository coordinates created by Jenkins (needs to be done in 2 files, see below). For more details how to update Lucene dependencies in Solr, run: $ gradlew :helpDeps Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Michael Gibney > Sent: Wednesday, May 26, 2021 8:02 PM > To: dev@solr.apache.org > Subject: Solr/Lucene joint development workflow? > > I'm working on some features that involve changes to both Lucene and > Solr. Post-TLP-split, I'm wondering whether anyone has recommended > techniques to handle this kind of situation. > > Ideally one would work on Lucene changes, get them merged, and then > proceed with Solr development; but realistically even if this were as > straightforward in practice as it sounds in principal, there are cases > where one would still want to develop in parallel. > > I haven't been able to find any documented recommendation on this > subject. It's possible to have a locally built Lucene snapshot (via > `gradlew mavenToLocalRepo`); but I was only able to actually _build_ > Solr against the local Lucene artifact by adding `mavenLocal()` to the > `allprojects/repositories` block in `gradle/defaults.gradle` -- and I > have yet to figure a way get the local Lucene artifact on the test > classpath (so I'm as yet unable to run Solr tests that depend on > unmerged upstream changes to Lucene). > > It's also possible that the partially-functional approach described > above will have to change now that Solr main depends on a specific > Lucene snapshot version. > > Is anybody doing something like this? Or perhaps I'm asking the wrong > question? I can think of solutions that involve setting up my own > maven repository, to which I publish my own pinned versions of Lucene, > and refer to such pinned versions/repo as part of a given Solr > "patch". But that feels both half-baked _and_ bloated, so I don't want > to go down that road unless I feel convinced there's no better > alternative. > > Michael > > - > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > For additional commands, e-mail: dev-h...@solr.apache.org - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
Solr/Lucene joint development workflow?
I'm working on some features that involve changes to both Lucene and Solr. Post-TLP-split, I'm wondering whether anyone has recommended techniques to handle this kind of situation. Ideally one would work on Lucene changes, get them merged, and then proceed with Solr development; but realistically even if this were as straightforward in practice as it sounds in principal, there are cases where one would still want to develop in parallel. I haven't been able to find any documented recommendation on this subject. It's possible to have a locally built Lucene snapshot (via `gradlew mavenToLocalRepo`); but I was only able to actually _build_ Solr against the local Lucene artifact by adding `mavenLocal()` to the `allprojects/repositories` block in `gradle/defaults.gradle` -- and I have yet to figure a way get the local Lucene artifact on the test classpath (so I'm as yet unable to run Solr tests that depend on unmerged upstream changes to Lucene). It's also possible that the partially-functional approach described above will have to change now that Solr main depends on a specific Lucene snapshot version. Is anybody doing something like this? Or perhaps I'm asking the wrong question? I can think of solutions that involve setting up my own maven repository, to which I publish my own pinned versions of Lucene, and refer to such pinned versions/repo as part of a given Solr "patch". But that feels both half-baked _and_ bloated, so I don't want to go down that road unless I feel convinced there's no better alternative. Michael - To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
