Re: [PR] SOLR-17243: CloudSolrClient support for req.getBasePath [solr]
dsmiley closed pull request #2414: SOLR-17243: CloudSolrClient support for req.getBasePath URL: https://github.com/apache/solr/pull/2414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-16974) Global Circuit Breakers
[ https://issues.apache.org/jira/browse/SOLR-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841005#comment-17841005 ] Jan Høydahl commented on SOLR-16974: Seems like a potential for optimization, please open another Jira. > Global Circuit Breakers > --- > > Key: SOLR-16974 > URL: https://issues.apache.org/jira/browse/SOLR-16974 > Project: Solr > Issue Type: New Feature > Components: Circuit Breakers >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 9.5 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Currently Circuit Breakers are configured per core in solrconfig.xml. > However, cores/collections do not live in isolation, and it could be that a > cluster administrator would like to enforce circuit breakers for the entire > cluster. > I'm not clear as to whether we need both cluster level and a core level > pluggability. And would core-level breakers add to any cluster-level ones or > override them for that core? > A potential design is to add this as a new plugin in solr.xml, and have them > added in a new static context of CircuitBreakerRegistry. Then the isTripped > logic would consult both the per-core list and the static/global list of > breakers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-14675) CloudSolrClient requestAsync API
[ https://issues.apache.org/jira/browse/SOLR-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-14675: Summary: CloudSolrClient requestAsync API (was: [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient) > CloudSolrClient requestAsync API > > > Key: SOLR-14675 > URL: https://issues.apache.org/jira/browse/SOLR-14675 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Rishi Sankar >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > In Solr 8.7, org.apache.solr.client.solrj.impl.Http2SolrClient has an > asyncRequest method which supports making async requests with a callback > parameter. However, this method is only used internally by the > MockingHttp2SolrClient and LBHttp2SolrClient. I'd like to contribute a method > to the CloudHttp2SolrClient that allows for making an asynchronous request > with a callback parameter that can be passed down to the Http2SolrClient > asyncRequest call. > I've been coordinating with [~dsmiley] about making this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-14675) [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient
[ https://issues.apache.org/jira/browse/SOLR-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-14675: Affects Version/s: (was: 8.7) > [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient > -- > > Key: SOLR-14675 > URL: https://issues.apache.org/jira/browse/SOLR-14675 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Rishi Sankar >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > In Solr 8.7, org.apache.solr.client.solrj.impl.Http2SolrClient has an > asyncRequest method which supports making async requests with a callback > parameter. However, this method is only used internally by the > MockingHttp2SolrClient and LBHttp2SolrClient. I'd like to contribute a method > to the CloudHttp2SolrClient that allows for making an asynchronous request > with a callback parameter that can be passed down to the Http2SolrClient > asyncRequest call. > I've been coordinating with [~dsmiley] about making this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-14675) [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient
[ https://issues.apache.org/jira/browse/SOLR-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840998#comment-17840998 ] David Smiley commented on SOLR-14675: - Now that SOLR-14763 (async API refactoring of Http2SolrClient) is done, this issue should be unblocked for having the same API (requestAsync() returning a CompletableFuture). Not sure if the initial PR here, long closed has any bits of interest; probably not as it was superseded by https://github.com/apache/lucene-solr/pull/1770 which *does* have the CloudHttp2SolrClient part. That part did not survive into the API refactor in order to keep the scope manageable. > [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient > -- > > Key: SOLR-14675 > URL: https://issues.apache.org/jira/browse/SOLR-14675 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 8.7 >Reporter: Rishi Sankar >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > In Solr 8.7, org.apache.solr.client.solrj.impl.Http2SolrClient has an > asyncRequest method which supports making async requests with a callback > parameter. However, this method is only used internally by the > MockingHttp2SolrClient and LBHttp2SolrClient. I'd like to contribute a method > to the CloudHttp2SolrClient that allows for making an asynchronous request > with a callback parameter that can be passed down to the Http2SolrClient > asyncRequest call. > I've been coordinating with [~dsmiley] about making this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-14675) [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient
[ https://issues.apache.org/jira/browse/SOLR-14675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-14675: Priority: Major (was: Minor) > [SolrJ] Http2SolrClient async request through CloudHttp2SolrClient > -- > > Key: SOLR-14675 > URL: https://issues.apache.org/jira/browse/SOLR-14675 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 8.7 >Reporter: Rishi Sankar >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > In Solr 8.7, org.apache.solr.client.solrj.impl.Http2SolrClient has an > asyncRequest method which supports making async requests with a callback > parameter. However, this method is only used internally by the > MockingHttp2SolrClient and LBHttp2SolrClient. I'd like to contribute a method > to the CloudHttp2SolrClient that allows for making an asynchronous request > with a callback parameter that can be passed down to the Http2SolrClient > asyncRequest call. > I've been coordinating with [~dsmiley] about making this change. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-12813: subqueries should respect basic auth [solr]
rseitz commented on code in PR #2404:
URL: https://github.com/apache/solr/pull/2404#discussion_r1580195804
##
solr/core/src/java/org/apache/solr/response/transform/SubQueryAugmenterFactory.java:
##
@@ -340,7 +346,9 @@ public void transform(SolrDocument doc, int docid) {
final SolrParams docWithDeprefixed =
SolrParams.wrapDefaults(new DocRowParams(doc, prefix, separator),
baseSubParams);
try {
- QueryResponse rsp = server.query(coreName, docWithDeprefixed);
+ QueryRequest req = new QueryRequest(docWithDeprefixed);
+ req.setUserPrincipal(principal);
+ QueryResponse rsp = req.process(server, coreName);
Review Comment:
I commented re: SolrRequestInfo separately but will add detail there. The
question is: what can we assume about the contents of SolrRequestInfo? In the
context of subquery execution, what we find at the top of the stack inside
SolrRequestInfo is the _new_ subquery request that we've constructed, not the
enclosing/original request that contained the authentication info. The
following excerpt from EmbeddedSolrServer shows how SolrRequestInfo gets set
before a subquery is actually executed:
```
req = _parser.buildRequestFrom(core, params,
getContentStreams(request));
SolrQueryResponse rsp = new SolrQueryResponse();
SolrRequestInfo.setRequestInfo(new SolrRequestInfo(req, rsp));
core.execute(handler, req, rsp);
```
My basic intuition is that if we're creating a brand new QueryRequest, we
really _should_ be copying any authentication info onto it that needs to be
used in its execution. I don't see any notion in the codebase of a parent/child
relationship between queries, where a child query could somehow know who its
parent is and derive auth info from the parent. Therefore, it really seems to
me like copying the auth info is the logical option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Updated] (SOLR-14763) SolrJ Client Async HTTP/2 Requests
[ https://issues.apache.org/jira/browse/SOLR-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated SOLR-14763: Description: In SOLR-14354, [~caomanhdat] created an API to use Jetty async API to make more thread efficient HttpShardHandler requests. This added public async request APIs to Http2SolrClient and LBHttp2SolrClient. There are a few ways this API can be improved, that I will track in this issue: 1) Using a CompletableFuture-based async API signature, instead of using internal custom interfaces (Cancellable, AsyncListener) - based on [this discussion|https://lists.apache.org/thread.html/r548f318d9176c84ad1a4ed49ff182eeea9f82f26cb23e372244c8a23%40%3Cdev.lucene.apache.org%3E]. --- The below was removed from the scope of what was delivered in 9.6; a linked JIRA will address it. 2) An async API is also useful in other HTTP/2 Solr clients as well, particularly CloudHttp2SolrClient (SOLR-14675). I will add a requestAsync method to the SolrClient class, with a default method that initially throws an unsupported operation exception (maybe this can be later updated to use an executor to handle the async request as a default impl). For now, I'll override the default implementation in the Http2SolrClient and CloudHttp2SolrClient. was: In SOLR-14354, [~caomanhdat] created an API to use Jetty async API to make more thread efficient HttpShardHandler requests. This added public async request APIs to Http2SolrClient and LBHttp2SolrClient. There are a few ways this API can be improved, that I will track in this issue: 1) Using a CompletableFuture-based async API signature, instead of using internal custom interfaces (Cancellable, AsyncListener) - based on [this discussion|https://lists.apache.org/thread.html/r548f318d9176c84ad1a4ed49ff182eeea9f82f26cb23e372244c8a23%40%3Cdev.lucene.apache.org%3E]. 2) An async API is also useful in other HTTP/2 Solr clients as well, particularly CloudHttp2SolrClient (SOLR-14675). I will add a requestAsync method to the SolrClient class, with a default method that initially throws an unsupported operation exception (maybe this can be later updated to use an executor to handle the async request as a default impl). For now, I'll override the default implementation in the Http2SolrClient and CloudHttp2SolrClient. > SolrJ Client Async HTTP/2 Requests > -- > > Key: SOLR-14763 > URL: https://issues.apache.org/jira/browse/SOLR-14763 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Affects Versions: 8.7 >Reporter: Rishi Sankar >Assignee: James Dyer >Priority: Major > Fix For: main (10.0), 9.6 > > Time Spent: 7h 40m > Remaining Estimate: 0h > > In SOLR-14354, [~caomanhdat] created an API to use Jetty async API to make > more thread efficient HttpShardHandler requests. This added public async > request APIs to Http2SolrClient and LBHttp2SolrClient. There are a few ways > this API can be improved, that I will track in this issue: > 1) Using a CompletableFuture-based async API signature, instead of using > internal custom interfaces (Cancellable, AsyncListener) - based on [this > discussion|https://lists.apache.org/thread.html/r548f318d9176c84ad1a4ed49ff182eeea9f82f26cb23e372244c8a23%40%3Cdev.lucene.apache.org%3E]. > --- > The below was removed from the scope of what was delivered in 9.6; a linked > JIRA will address it. > 2) An async API is also useful in other HTTP/2 Solr clients as well, > particularly CloudHttp2SolrClient (SOLR-14675). I will add a requestAsync > method to the SolrClient class, with a default method that initially throws > an unsupported operation exception (maybe this can be later updated to use an > executor to handle the async request as a default impl). For now, I'll > override the default implementation in the Http2SolrClient and > CloudHttp2SolrClient. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-12813: subqueries should respect basic auth [solr]
rseitz commented on code in PR #2404:
URL: https://github.com/apache/solr/pull/2404#discussion_r1580121540
##
solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java:
##
@@ -252,7 +260,11 @@ private SolrQueryRequest buildRequestFrom(
new SolrQueryRequestBase(core, params, requestTimer) {
@Override
public Principal getUserPrincipal() {
-return req == null ? null : req.getUserPrincipal();
+if (principal != null) {
+ return principal;
+} else {
+ return req == null ? null : req.getUserPrincipal();
+}
}
Review Comment:
I tried it and got a stack overflow. Here is what happens:
`EmbeddedSolrServer#request ` uses `SolrRequestParsers#buildRequestFrom` to
obtain a`SolrQueryRequest req`. It then calls
`SolrRequestInfo.setRequestInfo(new SolrRequestInfo(req, rsp))` to designate
that request as the current one (see
[here](https://github.com/apache/solr/blob/9ad70743a218665254f8aff15952ab4859e65b75/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java#L224)).
The `getUserPrincipal` method that I'm changing here belongs to an anonymous
inner class that extends `SolrQueryRequestBase`. At the time this
`getUserPrincipal` method is actually called, `SolrRequestInfo.getRequestInfo`
would return a `SolrRequestInfo` containing the same `SolrQueryRequest`
instance that this `getUserPrincipal` method is being called on in the first
place.
Another way to say this is that when we try to call
`SolrRequestInfo#getUserPrincipal` in this scenario, we don't interact with
original or enclosing request that included the original user query and the
auth info. Instead, we interact with the new request that is being created as
part of the subquery execution process.
I haven't noticed a way around this but am certainly eager for suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-12813: subqueries should respect basic auth [solr]
rseitz commented on code in PR #2404:
URL: https://github.com/apache/solr/pull/2404#discussion_r1580121540
##
solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java:
##
@@ -252,7 +260,11 @@ private SolrQueryRequest buildRequestFrom(
new SolrQueryRequestBase(core, params, requestTimer) {
@Override
public Principal getUserPrincipal() {
-return req == null ? null : req.getUserPrincipal();
+if (principal != null) {
+ return principal;
+} else {
+ return req == null ? null : req.getUserPrincipal();
+}
}
Review Comment:
I tried it and got a stack overflow. Here is what happens:
`EmbeddedSolrServer#request ` uses `SolrRequestParsers#buildRequestFrom` to
obtain a`SolrQueryRequest req`. It then calls
`SolrRequestInfo.setRequestInfo(new SolrRequestInfo(req, rsp))` to designate
that request as the current one (see
[here](https://github.com/apache/solr/blob/9ad70743a218665254f8aff15952ab4859e65b75/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java#L224)).
The `getUserPrincipal` method that I'm changing here belongs to an anonymous
inner class that extends `SolrQueryRequestBase`. At the time this
`getUserPrincipal` method is actually called, `SolrRequestInfo.getRequestInfo`
would return a `SolrRequestInfo` containing the same `SolrQueryRequest`
instance that this `getUserPrincipal` method is being called on in the first
place.
Another way to say this is that when we try to call
`SolrRequestInfo#getUserPrincipal` in this scenario, we don't interact with
original or enclosing request that included the original user query and the
auth info. Instead, we interact with the new request that is being created as
part of the subquery execution process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-12813: subqueries should respect basic auth [solr]
rseitz commented on code in PR #2404:
URL: https://github.com/apache/solr/pull/2404#discussion_r1580121540
##
solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java:
##
@@ -252,7 +260,11 @@ private SolrQueryRequest buildRequestFrom(
new SolrQueryRequestBase(core, params, requestTimer) {
@Override
public Principal getUserPrincipal() {
-return req == null ? null : req.getUserPrincipal();
+if (principal != null) {
+ return principal;
+} else {
+ return req == null ? null : req.getUserPrincipal();
+}
}
Review Comment:
I tried it and got a stack overflow. Here is what happens:
`EmbeddedSolrServer#request ` uses `SolrRequestParsers#buildRequestFrom` to
obtain a`SolrQueryRequest req`. It then calls
`SolrRequestInfo.setRequestInfo(new SolrRequestInfo(req, rsp))` to designate
that request as the current one (see
[here](https://github.com/apache/solr/blob/9ad70743a218665254f8aff15952ab4859e65b75/solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java#L224)).
The `getUserPrincipal` method that I'm changing here belongs to an anonymous
inner class that extends `SolrQueryRequestBase`. At the time this
`getUserPrincipal` method is actually called, `SolrRequestInfo.getRequestInfo`
would return a `SolrRequestInfo` containing the same `SolrQueryRequest`
instance that `getUserPrincipal` is being called on in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-4587: integrate lucene-monitor into solr [solr]
kotman12 commented on PR #2382:
URL: https://github.com/apache/solr/pull/2382#issuecomment-2078142747
> So I was trying to learn how the main configuration bits fit together here
and high-level the reverse search idea and my _solr-monitor-naive-dinner-demo_
branch (or #2421 diff) off this pull request's branch is a side effect of that
and my understanding so far based on it is that:
>
> * the in-memory state is in the `Presearcher` object in the
`ReverseQueryParserPlugin` class object (and in the
_solr-monitor-naive-dinner-demo_ i just used a simple `Monitor` object instead
of the `Presearcher` object)
> * the state is updated via the `MonitorUpdateRequestProcessor` i.e. saved
searches are added as `MonitorQuery` objects to the `Monitor` object (and
updating of the `Presearcher` object is a bit different)
> * the state is accessed via the `ReverseSearchComponent` component
(currently non-distributed but conceptually distributed would work too?)
>
> Is that basic understanding correct? As a next step I might go learn more
about the `Presearcher` itself.
I'll give the PR a look but when I first looked at this my main concerns
wiring a Monitor straight into solr were:
1. Handling commit/rollback and what to update the tlog with if you also
writing to a "sidecar" monitor object?
2. Handling persistence. Currently the Monitor has its own tightly sealed
index. It can be configured for persistence but if you want to peek at the
segments a monitor is writing to disk it might not be easy, especially to
handle configurations like tlog+pull. The alternative is to use only the
in-memory Monitor configurations but that has limitations and takes away
precious resources from the {cacheId -> deserialized query} cache.
3. Bringing me to my final point that the cache a Monitor object wraps is a
simple concurrentHashMap which is updated with a very coarse-grained lock that
can block reads for a long time. It just doesn't feel like it "jives" with the
solr approach to concurrency that is much more sophisticated (it is a fully
fledged db after all). We could make the Monitor cache more configurable in the
upstream lucene monitor repo but in my opinion lucene monitor tries to do too
much state-management that its not that good at but the most valuable thing to
take advantage of is the sophisticated reverse search methods (query
decomposition for faster matching, query tokenization for pre-search, term
weighting, optimized document-to-query conversion with term-acceptor, and
probably something else I am forgetting).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17248: Refactor ZK related SolrCli tools to separate SolrZkClient and CloudSolrClient instantiation/usage [solr]
laminelam commented on code in PR #2417:
URL: https://github.com/apache/solr/pull/2417#discussion_r1578173456
##
solr/core/src/java/org/apache/solr/cli/SolrCLI.java:
##
@@ -607,6 +611,46 @@ public static String getZkHost(CommandLine cli) throws
Exception {
return zkHost;
}
+ public static SolrZkClient getSolrZkClient(CommandLine cli) throws Exception
{
+return getSolrZkClient(cli, getZkHost(cli));
+ }
+
+ public static SolrZkClient getSolrZkClient(CommandLine cli, String zkHost)
throws Exception {
+if (zkHost == null) {
+ throw new IllegalStateException(
+ "Solr at "
+ + cli.getOptionValue("solrUrl")
+ + " is running in standalone server mode, this command can only
be used when running in SolrCloud mode.\n");
+}
+return new SolrZkClient.Builder()
+.withUrl(zkHost)
+.withTimeout(SolrZkClientTimeout.DEFAULT_ZK_CLIENT_TIMEOUT,
TimeUnit.MILLISECONDS)
+.withSolrClassLoader(getSolrResourceLoader())
+.build();
+ }
+
+ public static CloudHttp2SolrClient getCloudHttp2SolrClient(String zkHost) {
+return getCloudHttp2SolrClient(zkHost, null);
+ }
+
+ public static CloudHttp2SolrClient getCloudHttp2SolrClient(
+ String zkHost, Http2SolrClient.Builder builder) {
+return new CloudHttp2SolrClient.Builder(Collections.singletonList(zkHost),
Optional.empty())
+.withSolrClassLoader(getSolrResourceLoader())
+.withInternalClientBuilder(builder)
+.build();
+ }
+
+ private static SolrResourceLoader getSolrResourceLoader() {
+String dir = System.getProperty("solr.solr.home");
+if (StrUtils.isNullOrEmpty(dir)) {
+ dir = System.getProperty("solr.install.dir");
+}
+final SolrResourceLoader loader = new SolrResourceLoader(Paths.get(dir));
+NodeConfig.initModules(loader, null);
Review Comment:
Some context
[here](https://github.com/apache/solr/pull/1994#discussion_r1534863816)
If SolrZKClient needs to instantiate a class part of a module, we need to
pass a SolrClassLoader that "contains" the class/dependency.
Now, the only way to load modules/dependencies into SolrClassLoader is to
use the static method
[NodeConfig.initModules](https://github.com/apache/solr/blob/59744b0625d6c9b7b461ad6da9f44a7c6a7313dc/solr/core/src/java/org/apache/solr/core/NodeConfig.java#L523)
I know it's kind if weird but this is, to my knowledge, the only way Solr
has to load modules' dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17248: Refactor ZK related SolrCli tools to separate SolrZkClient and CloudSolrClient instantiation/usage [solr]
laminelam commented on code in PR #2417:
URL: https://github.com/apache/solr/pull/2417#discussion_r1579968386
##
solr/core/src/java/org/apache/solr/cli/SolrCLI.java:
##
@@ -607,6 +611,46 @@ public static String getZkHost(CommandLine cli) throws
Exception {
return zkHost;
}
+ public static SolrZkClient getSolrZkClient(CommandLine cli) throws Exception
{
+return getSolrZkClient(cli, getZkHost(cli));
+ }
+
+ public static SolrZkClient getSolrZkClient(CommandLine cli, String zkHost)
throws Exception {
+if (zkHost == null) {
+ throw new IllegalStateException(
+ "Solr at "
+ + cli.getOptionValue("solrUrl")
+ + " is running in standalone server mode, this command can only
be used when running in SolrCloud mode.\n");
+}
+return new SolrZkClient.Builder()
+.withUrl(zkHost)
+.withTimeout(SolrZkClientTimeout.DEFAULT_ZK_CLIENT_TIMEOUT,
TimeUnit.MILLISECONDS)
+.withSolrClassLoader(getSolrResourceLoader())
+.build();
+ }
+
+ public static CloudHttp2SolrClient getCloudHttp2SolrClient(String zkHost) {
+return getCloudHttp2SolrClient(zkHost, null);
+ }
+
+ public static CloudHttp2SolrClient getCloudHttp2SolrClient(
+ String zkHost, Http2SolrClient.Builder builder) {
+return new CloudHttp2SolrClient.Builder(Collections.singletonList(zkHost),
Optional.empty())
+.withSolrClassLoader(getSolrResourceLoader())
+.withInternalClientBuilder(builder)
+.build();
+ }
+
+ private static SolrResourceLoader getSolrResourceLoader() {
+String dir = System.getProperty("solr.solr.home");
+if (StrUtils.isNullOrEmpty(dir)) {
+ dir = System.getProperty("solr.install.dir");
+}
+final SolrResourceLoader loader = new SolrResourceLoader(Paths.get(dir));
+NodeConfig.initModules(loader, null);
Review Comment:
Actually I am one the who made that comment. I feel there is a lack of
separation between NodeConfig, modules and SolrResourceLoader logic. Which may
have contributed to the situation we're facing here.
As for _SolrResourceLoader_, the main problem is that _solrj_ doesn't have
access to classes part of _modules_ or _core_. As _SolrZKClient_ is part of
_solrj_, it needs a way to get access to those classes.
There's some discussion
[here](https://github.com/apache/solr/pull/826#discussion_r866976673) about
this issue. That discussion led to the new feature of passing a
SolrResourceLoader to SolrZKclient after loading the modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17248: Refactor ZK related SolrCli tools to separate SolrZkClient and CloudSolrClient instantiation/usage [solr]
laminelam commented on code in PR #2417:
URL: https://github.com/apache/solr/pull/2417#discussion_r1579968386
##
solr/core/src/java/org/apache/solr/cli/SolrCLI.java:
##
@@ -607,6 +611,46 @@ public static String getZkHost(CommandLine cli) throws
Exception {
return zkHost;
}
+ public static SolrZkClient getSolrZkClient(CommandLine cli) throws Exception
{
+return getSolrZkClient(cli, getZkHost(cli));
+ }
+
+ public static SolrZkClient getSolrZkClient(CommandLine cli, String zkHost)
throws Exception {
+if (zkHost == null) {
+ throw new IllegalStateException(
+ "Solr at "
+ + cli.getOptionValue("solrUrl")
+ + " is running in standalone server mode, this command can only
be used when running in SolrCloud mode.\n");
+}
+return new SolrZkClient.Builder()
+.withUrl(zkHost)
+.withTimeout(SolrZkClientTimeout.DEFAULT_ZK_CLIENT_TIMEOUT,
TimeUnit.MILLISECONDS)
+.withSolrClassLoader(getSolrResourceLoader())
+.build();
+ }
+
+ public static CloudHttp2SolrClient getCloudHttp2SolrClient(String zkHost) {
+return getCloudHttp2SolrClient(zkHost, null);
+ }
+
+ public static CloudHttp2SolrClient getCloudHttp2SolrClient(
+ String zkHost, Http2SolrClient.Builder builder) {
+return new CloudHttp2SolrClient.Builder(Collections.singletonList(zkHost),
Optional.empty())
+.withSolrClassLoader(getSolrResourceLoader())
+.withInternalClientBuilder(builder)
+.build();
+ }
+
+ private static SolrResourceLoader getSolrResourceLoader() {
+String dir = System.getProperty("solr.solr.home");
+if (StrUtils.isNullOrEmpty(dir)) {
+ dir = System.getProperty("solr.install.dir");
+}
+final SolrResourceLoader loader = new SolrResourceLoader(Paths.get(dir));
+NodeConfig.initModules(loader, null);
Review Comment:
Actually I am one the who made that comment. I feel there is a lack of
separation between NodeConfig, modules and SolrResourceLoader logic. Which may
have contributed to the situation we facing here.
As for _SolrResourceLoader_, the main problem is that _solrj_ doesn't have
access to classes part of _modules_ or _core_. As _SolrZKClient_ is part of
_solrj_, it needs a way to get access to those classes.
There's some discussion
[here](https://github.com/apache/solr/pull/826#discussion_r866976673) about
this issue. That discussion led to the new feature of passing a
SolrResourceLoader to SolrZKclient after loading the modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-16116: Use apache curator to manage the Solr Zookeeper interactions [solr]
dsmiley commented on PR #760: URL: https://github.com/apache/solr/pull/760#issuecomment-2077904166 This is an important initiative. If you've given up on it @risdenk, let us know so someone can take over. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17247: Fix bug - 'WWW-Authenticate' headers missing in MultiAuthPlugin [solr]
laminelam commented on PR #2416: URL: https://github.com/apache/solr/pull/2416#issuecomment-2077823343 Hi @janhoy If you have some time, would you please take a look at this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-4587: solr-monitor-naive-dinner-demo [solr]
cpoerschke closed pull request #2421: SOLR-4587: solr-monitor-naive-dinner-demo URL: https://github.com/apache/solr/pull/2421 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-4587: integrate lucene-monitor into solr [solr]
cpoerschke commented on PR #2382: URL: https://github.com/apache/solr/pull/2382#issuecomment-2077786562 So I was trying to learn how the main configuration bits fit together here and high-level the reverse search idea and my _solr-monitor-naive-dinner-demo_ branch (or #link-to-follow diff) off this pull request's branch is a side effect of that and my understanding so far based on it is that: * the in-memory state is in the `Presearcher` object in the `ReverseQueryParserPlugin` class object (and in the _solr-monitor-naive-dinner-demo_ i just used a simple `Monitor` object instead of the `Presearcher` object) * the state is updated via the `MonitorUpdateRequestProcessor` i.e. saved searches are added as `MonitorQuery` objects to the `Monitor` object (and updating of the `Presearcher` object is a bit different) * the state is accessed via the `ReverseSearchComponent` component (currently non-distributed but conceptually distributed would work too?) Is that basic understanding correct? As a next step I might go learn more about the `Presearcher` itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-16974) Global Circuit Breakers
[ https://issues.apache.org/jira/browse/SOLR-16974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840872#comment-17840872 ] Ilan Ginzburg commented on SOLR-16974: -- Looking at the [SolrCore constructor call|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/core/SolrCore.java#L1076] to build a [CircuitBreakerRegistry|https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/util/circuitbreaker/CircuitBreakerRegistry.java#L65] it seems we redo the static configuration again for each core (even though the configuration is not core specific) > Global Circuit Breakers > --- > > Key: SOLR-16974 > URL: https://issues.apache.org/jira/browse/SOLR-16974 > Project: Solr > Issue Type: New Feature > Components: Circuit Breakers >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Fix For: 9.5 > > Time Spent: 6h 50m > Remaining Estimate: 0h > > Currently Circuit Breakers are configured per core in solrconfig.xml. > However, cores/collections do not live in isolation, and it could be that a > cluster administrator would like to enforce circuit breakers for the entire > cluster. > I'm not clear as to whether we need both cluster level and a core level > pluggability. And would core-level breakers add to any cluster-level ones or > override them for that core? > A potential design is to add this as a new plugin in solr.xml, and have them > added in a new static context of CircuitBreakerRegistry. Then the isTripped > logic would consult both the per-core list and the static/global list of > breakers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [I] Update the default solr.xml to include maxBooleanClauses [solr-operator]
coolstim closed issue #526: Update the default solr.xml to include maxBooleanClauses URL: https://github.com/apache/solr-operator/issues/526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [I] Update the default solr.xml to include maxBooleanClauses [solr-operator]
coolstim commented on issue #526: URL: https://github.com/apache/solr-operator/issues/526#issuecomment-2077507752 issue is solved in https://github.com/apache/solr-operator/issues/630 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-17243: CloudSolrClient support for req.getBasePath [solr]
dsmiley commented on PR #2414: URL: https://github.com/apache/solr/pull/2414#issuecomment-2077367304 As I said in my last comment, I'd prefer to abandon QueryRequest.getBasePath and setBasePath. https://issues.apache.org/jira/browse/SOLR-17256 It'd still be nice to expose an HttpSolrClient out of CloudSolrClient but my motivation for doing so may be handled by SOLR-17256 so this PR here may not be needed anyway. I'll take the javadocs though :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Resolved] (SOLR-17243) CloudSolrClient should support SolrRequest.getBasePath (a URL)
[ https://issues.apache.org/jira/browse/SOLR-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley resolved SOLR-17243. - Resolution: Won't Fix > CloudSolrClient should support SolrRequest.getBasePath (a URL) > -- > > Key: SOLR-17243 > URL: https://issues.apache.org/jira/browse/SOLR-17243 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: David Smiley >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Sometimes you have a CloudSolrClient and you have a use-case where you want > to send a request to a specific node / URL. Perhaps uncommon for typical > users but it's common internally (within Solr) to encounter this. > SolrRequest.setBasePath / getBasePath (which is actually a URL!) is already > here for this but it only works on an Http SolrClient. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] Avoid instantiating NodeStateProvider several times when computing replica placement [solr]
ycallea opened a new pull request, #2420:
URL: https://github.com/apache/solr/pull/2420
https://issues.apache.org/jira/browse/SOLR-X
# Description
When using either the Minimize Cores or the Affinity placement strategy with
Solr 9, positioning a new replica becomes very inefficient past a certain
collection count.
At Salesforce, we are operating clusters packed with tens of thousands of
collection, and each replica placement operation takes several seconds to
complete at that scale.
This pull request aims at bringing a first simple change to improve the
performance of replica placement when operating with a large number of
collections. It will eventually be followed by additional changes as we are
progressing towards solutions for addressing such scalability issues.
# Solution
When fetching the cluster's metrics and information needed to compute the
optimal position of the replica to be positioned, we are calling a seemingly
harmless `cloudManager.getNodeStateProvider()` method `2 * number_of_nodes`
times in
[AttributeFetcherImpl.java#L144-L209](https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/cluster/placement/impl/AttributeFetcherImpl.java#L144-L209).
However, this method instantiates a
[SolrClientNodeStateProvider](https://github.com/apache/solr/blob/main/solr/solrj-zookeeper/src/java/org/apache/solr/client/solrj/impl/SolrClientNodeStateProvider.java)
instance every time it is called, which performs an expensive operation within
its constructor to build a `nodeVsCollectionVsShardVsReplicaInfo` data
structure, filled with information about all existing collections and replicas.
```java
protected final Map>>>
nodeVsCollectionVsShardVsReplicaInfo = new HashMap<>();
public SolrClientNodeStateProvider(CloudLegacySolrClient solrClient) {
this.solrClient = solrClient;
try {
readReplicaDetails();
} catch (IOException e) {
throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e);
}
}
protected void readReplicaDetails() throws IOException {
ClusterStateProvider clusterStateProvider = getClusterStateProvider();
ClusterState clusterState = clusterStateProvider.getClusterState();
if (clusterState == null) { // zkStateReader still initializing
return;
}
Map all =
clusterStateProvider.getClusterState().getCollectionStates();
all.forEach(
(collName, ref) -> {
DocCollection coll = ref.get();
if (coll == null) return;
coll.forEachReplica(
(shard, replica) -> {
Map>> nodeData =
nodeVsCollectionVsShardVsReplicaInfo.computeIfAbsent(
replica.getNodeName(), k -> new HashMap<>());
Map> collData =
nodeData.computeIfAbsent(collName, k -> new HashMap<>());
List replicas = collData.computeIfAbsent(shard, k
-> new ArrayList<>());
replicas.add((Replica) replica.clone());
});
});
}
```
This operation has been observed to take several seconds, even on clusters
with a very moderate amount of small collections (~1000).
The simple change proposed in this pull request will reduce the number of
times this expensive operation is performed for each replica placement
operation from `2 * number_of_nodes` to only once.
While the underlying algorithm will in all likelihood require further
optimizations to provide acceptable execution times on large clusters with tens
of thousands collections, we will already observe a noticeable improvement with
this first fix.
# Tests
Changes have been validated as part of pre-production tests at Salesforce.
# Checklist
Please review the following and check all that apply:
- [X] I have reviewed the guidelines for [How to
Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my
code conforms to the standards described there to the best of my ability.
- [ ] I have created a Jira issue and added the issue ID to my pull request
title.
- [X] I have given Solr maintainers
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
to contribute to my PR branch. (optional but recommended)
- [X] I have developed this patch against the `main` branch.
- [X] I have run `./gradlew check`.
- [ ] I have added tests for my changes.
- [ ] I have added documentation for the [Reference
Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@solr
Re: [PR] SOLR-17243: CloudSolrClient support for req.getBasePath [solr]
gerlowskija commented on PR #2414: URL: https://github.com/apache/solr/pull/2414#issuecomment-2077004881 > I forgot to share I have the same concern. But I'd like a convenience method like CloudHttpSolrClient.getHttpSolrClient(String url). How does that sound? Ah, glad it's not just me. I think a convenience method is a solid approach here - if you're not familiar with the LB client, `cloudClient.getLBClient().getClient(...)` is pretty unintuitive. I could quibble about the method name - I'd opt for something that focuses on conveying the usecase instead of just telling me what the method returns. e.g. `getClientForUrl(...)`. But that's just my 2c, and I don't have strong opinions there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
