[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17945731#comment-17945731
]
ASF subversion and git services commented on SOLR-17515:
Commit 5cfabe58da3251c89d30e52adc55c3494d07ee7b in solr's branch
refs/heads/add-pki-caching from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=5cfabe58da3 ]
SOLR-17515: Always init Http2SolrClient authStore (#2802)
Prior to this commit, Http2SolrClient only initialized its
'authenticationStore' instance var when creating a fresh Jetty
HttpClient. This causes NPE's later on when client code (e.g.
PreemptiveBasicAuthClientBuilderFactory) would attempt to access the
object.
This commit fixes this by making sure that 'authenticationStore' is
initialized when using an existing Jetty client (typically by reusing
the AuthenticationStore already used by the client).
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Fix For: 9.8
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using th
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894145#comment-17894145
]
Jason Gerlowski commented on SOLR-17515:
bq. Why wasn't CHANGES.txt updated?
Ah, yeah, I spoke to this [on the PR
yesterday|https://github.com/apache/solr/pull/2802#issuecomment-2445000209]. I
didn't include an entry for this on main or branch_9x because CHANGES.txt
doesn't have a 9.7.1 section on those branches. I did write a CHANGES.txt
entry, but it only exists on branch_9_7 where there's a section for it to go
into.
The CHANGES.txt entry on branch_9_7 reads:
bq. SOLR-17515: Http2SolrClient now sets its 'AuthenticationStore' in all
circumstances, preventing NPE's when used with a HttpClientBuilderFactory
(Jason Gerlowski)
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> git checkout branch_9_7
> ./gradlew clean assemble
> cd solr/packaging/build/solr-9.7.0-SNAPSHOT
> # At prompts, I cho
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894130#comment-17894130
]
ASF subversion and git services commented on SOLR-17515:
Commit 88ecadd2f546c3855c96e83bacc6ad61db71d9c2 in solr's branch
refs/heads/branch_9_7 from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=88ecadd2f54 ]
SOLR-17515: Always init Http2SolrClient authStore (#2802)
Prior to this commit, Http2SolrClient only initialized its
'authenticationStore' instance var when creating a fresh Jetty
HttpClient. This causes NPE's later on when client code (e.g.
PreemptiveBasicAuthClientBuilderFactory) would attempt to access the
object.
This commit fixes this by making sure that 'authenticationStore' is
initialized when using an existing Jetty client (typically by reusing
the AuthenticationStore already used by the client).
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> g
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17894052#comment-17894052
]
David Smiley commented on SOLR-17515:
-
Why wasn't CHANGES.txt updated?
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> git checkout branch_9_7
> ./gradlew clean assemble
> cd solr/packaging/build/solr-9.7.0-SNAPSHOT
> # At prompts, I chose: 4 nodes, "gettingstarted", 1 shard, 2 replicas,
> "_default" configset
> bin/solr start -e cloud
> bin/solr post -c gettingstarted example/exampledocs/books.json
> # Stop the node containing the non-leader replica
> bin/solr stop -p
> bin/solr post -c gettingstarted example/exampledocs/books.csv
> # Enable auth and trigger recovery by turning the node back on
> bin/solr auth enable -type basicAuth -credentials solr:solrRocks
> -blockUnknown true
> # This line will need tweaked based on which Solr node was previously stopped
> "bin/solr" start --cloud -p -s "exampl
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893959#comment-17893959
]
ASF subversion and git services commented on SOLR-17515:
Commit cae9f7c16f4a4fe20544dee5886cae2989a897c8 in solr's branch
refs/heads/branch_9x from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=cae9f7c16f4 ]
SOLR-17515: Always init Http2SolrClient authStore (#2802)
Prior to this commit, Http2SolrClient only initialized its
'authenticationStore' instance var when creating a fresh Jetty
HttpClient. This causes NPE's later on when client code (e.g.
PreemptiveBasicAuthClientBuilderFactory) would attempt to access the
object.
This commit fixes this by making sure that 'authenticationStore' is
initialized when using an existing Jetty client (typically by reusing
the AuthenticationStore already used by the client).
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> gi
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893950#comment-17893950
]
ASF subversion and git services commented on SOLR-17515:
Commit 700d2f2291030c0eade9b20004fdd428e633b7c1 in solr's branch
refs/heads/main from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=700d2f22910 ]
SOLR-17515: Always init Http2SolrClient authStore (#2802)
Prior to this commit, Http2SolrClient only initialized its
'authenticationStore' instance var when creating a fresh Jetty
HttpClient. This causes NPE's later on when client code (e.g.
PreemptiveBasicAuthClientBuilderFactory) would attempt to access the
object.
This commit fixes this by making sure that 'authenticationStore' is
initialized when using an existing Jetty client (typically by reusing
the AuthenticationStore already used by the client).
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> git chec
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893500#comment-17893500
]
Jason Gerlowski commented on SOLR-17515:
Alright, a few updates:
*First, a workaround!* The NPE reported above triggers when using
'PreemptiveBasicAuthClientBuilderFactory' to initialize a nascent
Http2SolrClient. PreemptiveBasicAuthClientBuilderFactory is frequently useful
on the client-side and in "standalone" Solr setups. But it's not strictly
needed on nodes running SolrCloud, which will either use PKI or forward
user-provided credentials on any internode requests.
So SolrCloud users at least can prevent this NPE by ensuring the
'solr.httpclient.builder.factory' system property is not set on their Solr
nodes. (This system property is set by the "bin/solr" startup script itself if
"SOLR_AUTH_TYPE" is specified in "solr.in.sh", so removing the SOLR_AUTH_TYPE
env-var will also help prevent Solr nodes from using
PremeptiveBasicAuthClientBuilderFactory.)
While this workaround seems effective in preliminary testing - the underlying
bug (i.e. that Http2SolrClient doesn't always initialize its
"authenticationStore") still remains on 9.7, so this workaround may not solve
all problems.
*Second, a note on testing*
Solr has a number of tests that cover basic-auth and recovery together, so this
bug has been a bit confounding: why didn't our existing tests catch this issue?
The workaround above points towards the answer: the bug relies on a sysprop
that's frequently set by the "bin/solr" scripts, but not set in our Java-based
tests. So this bug is another victim of the "fidelity gap" between how we test
Solr and how it is deployed by users.
*Third, a prospective fix*
I've created a PR [here|https://github.com/apache/solr/pull/2802], which aims
to fix the initialization bug in Http2SolrClient. It also strengthens some
existing tests so that they reproduce the NPE when this fix is not in place.
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892967#comment-17892967
]
Sanjay Dutt commented on SOLR-17515:
We both updated it same time. That's great! Yes go ahead and take it, and
meanwhile I will try to see If why my old test case were not able to caught
this one, and try to update them.
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> git checkout branch_9_7
> ./gradlew clean assemble
> cd solr/packaging/build/solr-9.7.0-SNAPSHOT
> # At prompts, I chose: 4 nodes, "gettingstarted", 1 shard, 2 replicas,
> "_default" configset
> bin/solr start -e cloud
> bin/solr post -c gettingstarted example/exampledocs/books.json
> # Stop the node containing the non-leader replica
> bin/solr stop -p
> bin/solr post -c gettingstarted example/exampledocs/books.csv
> # Enable auth and trigger recovery by turning the node back on
> bin/solr auth enable -type basicAuth -credentials solr:solrRocks
> -blockUnknown true
> # This line will need tweaked based on which Solr node was previo
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892962#comment-17892962
]
Sanjay Dutt commented on SOLR-17515:
Thank you so much [~gerlowskija] for reproducing it and providing all the
details. Though I am bit confused with all the different auth mechanism we have
in place. Even last time two auth cases found for which new test case were
added. Clearly, more test cases are required. Going to work on this one unless
you are already on it.
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Priority: Major
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> The issue can be reproduced locally in a source-checkout using the following
> steps:
> {code}
> git checkout branch_9_7
> ./gradlew clean assemble
> cd solr/packaging/build/solr-9.7.0-SNAPSHOT
> # At prompts, I chose: 4 nodes, "gettingstarted", 1 shard, 2 replicas,
> "_default" configset
> bin/solr start -e cloud
> bin/solr post -c gettingstarted example/exampledocs/books.json
> # Stop the node containing the non-leader replica
> bin/solr stop -p
> bin/solr post -c gettingstarted example/exampledocs/books.csv
> # Enable auth and trigger recove
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892821#comment-17892821
]
Jason Gerlowski commented on SOLR-17515:
Credit and thanks to Patrik Peng and Endika Posadas for [reporting this on the
users list|https://lists.apache.org/thread/jhs7lkg942nxg2hlb879k6tc832yhm06]!
This seems like a pretty serious bug: perhaps worth a 9.7.1?
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Priority: Major
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:212)
> ~[metrics-core-4.2.26.jar:4.2.26]
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> ~[?:?]
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> ~[?:?]
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[?:?]
> at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8
[jira] [Commented] (SOLR-17515) Recovery fails in Solr 9.7.0 if basic-auth is enabled
[
https://issues.apache.org/jira/browse/SOLR-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17892831#comment-17892831
]
Jason Gerlowski commented on SOLR-17515:
[~sanjaydutt] pointed me at the likely culprit:
At various points, the RecoveryStrategy code bootstraps a new Http2SolrClient
based on an existing one. But this bootstrapping overlooks the
'authenticationStore' object from the existing client, which results in a NPE
when code later on expects it to be set. The place to fix this is _probably_
in the "withHttpClient" builder method used by RecoveryStrategy (see the
calling snippet below):
{code:title=RecoveryStrategy#recoverySolrClientBuilder}
private Http2SolrClient.Builder recoverySolrClientBuilder(String baseUrl,
String leaderCoreName) {
final UpdateShardHandlerConfig cfg =
cc.getConfig().getUpdateShardHandlerConfig();
return new Http2SolrClient.Builder(baseUrl)
.withDefaultCollection(leaderCoreName)
.withHttpClient(cc.getUpdateShardHandler().getRecoveryOnlyHttpClient());
}
{code}
> Recovery fails in Solr 9.7.0 if basic-auth is enabled
> -
>
> Key: SOLR-17515
> URL: https://issues.apache.org/jira/browse/SOLR-17515
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 9.7
>Reporter: Jason Gerlowski
>Priority: Major
>
> Several reporters on the users@ list, recently shared a bug they noticed on
> upgrading to Solr 9.7. Replicas would try to recover, but fail with a
> NullPointerException:
> {code}
> 2024-09-18 09:36:31.238 ERROR
> (recoveryExecutor-12-thread-1-processing-fts06.host.internal:8983_solr
> dovecot_fts_shard5_replica_n61 dovecot_fts shard5 core_node62) [c:dovecot_fts
> s:shard5 r:core_node62 x:dovecot_fts_shard5_replica_n61 t:]
> o.a.s.c.RecoveryStrategy Error while trying to recover.
> core=dovecot_fts_shard5_replica_n61 => java.lang.NullPointerException: Cannot
> invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> java.lang.NullPointerException: Cannot invoke
> "org.apache.solr.client.solrj.impl.AuthenticationStoreHolder.updateAuthenticationStore(org.eclipse.jetty.client.api.AuthenticationStore)"
> because "this.authenticationStore" is null
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.setAuthenticationStore(Http2SolrClient.java:318)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:97)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory.setup(PreemptiveBasicAuthClientBuilderFactory.java:85)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.httpClientBuilderSetup(Http2SolrClient.java:1093)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:1062)
> ~[solr-solrj-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:907)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:633)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 -
> anshum - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:333)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:309)
> ~[solr-core-9.7.0.jar:9.7.0 675a41516e3f3bacfc975590773e7abdca444ff4 - anshum
> - 2024-09-03 15:05:20]
> ...
> {code}
> It turns out that the issue isn't specific to upgrading clusters: *any 9.7.0
> cluster (new or existing/upgrading) that uses basic-auth will hit this NPE on
> during replica recovery*. The result is that replicas will fail to recover,
> and sit marked as "recovering" indefinitely.
> T
