Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 merged PR #4149: URL: https://github.com/apache/solr/pull/4149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3104148944
##
changelog/unreleased/SOLR-8127-distributed-luke.yml:
##
@@ -0,0 +1,8 @@
+# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc
+title: Distributed Luke
Review Comment:
expand on that please ;-)
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (multiple shards)
+
+When running in SolrCloud, the Luke handler automatically distributes requests
across all shards in the collection, the same as search requests.
+To inspect only the local shard's index set `distrib=false`.
+In non-cloud mode, you can still distribute across shards by passing the
`shards` parameter with explicit shard URLs.
Review Comment:
```suggestion
In user-managed clusters, you can distribute across shards by passing the
`shards` parameter with explicit shard URLs.
```
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (multiple shards)
+
+When running in SolrCloud, the Luke handler automatically distributes requests
across all shards in the collection, the same as search requests.
+To inspect only the local shard's index set `distrib=false`.
Review Comment:
```suggestion
To inspect only the receiving shard's index set `distrib=false`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on PR #4149: URL: https://github.com/apache/solr/pull/4149#issuecomment-4270526127 I plan on merging this in a few days unless anyone has objections. cc @dsmiley -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3029970968
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import java.util.Map;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends
BaseDistributedSearchTestCase {
+
+ private static final Long NUM_DOCS = 20L;
+
+ public LukeRequestHandlerDistribTest() {
+fixShardCount(2);
+ }
+
+ private LukeResponse requestLuke() throws Exception {
+return requestLuke(new ModifiableSolrParams());
+ }
+
+ private LukeResponse requestLuke(ModifiableSolrParams extra) throws
Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.add(extra);
+
+// query() sends to control and a random shard with shards param, compares
responses
+handle.put("QTime", SKIPVAL);
+handle.put("index", SKIP);
+handle.put("shards", SKIP);
+// Detailed per-field stats (distinct, topTerms, histogram) are kept
per-shard in
+// distributed mode and intentionally excluded from the aggregated
top-level fields.
+// Local mode includes them inline, so skip them in the comparison.
+handle.put("distinct", SKIP);
+handle.put("topTerms", SKIP);
+handle.put("histogram", SKIP);
+QueryResponse qr = query(params);
+LukeResponse rsp = new LukeResponse();
+rsp.setResponse(qr.getResponse());
+
+return rsp;
+ }
+
+ private void assertLukeXPath(ModifiableSolrParams extra, String... xpaths)
throws Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.set("wt", "xml");
+params.set("shards", shards);
+params.add(extra);
+QueryRequest req = new QueryRequest(params);
Review Comment:
I tweaked `LukeRequest` to handle extra params which allows us to just use
that. I feel this is the best approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3025720178
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import java.util.Map;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends
BaseDistributedSearchTestCase {
+
+ private static final Long NUM_DOCS = 20L;
+
+ public LukeRequestHandlerDistribTest() {
+fixShardCount(2);
+ }
+
+ private LukeResponse requestLuke() throws Exception {
+return requestLuke(new ModifiableSolrParams());
+ }
+
+ private LukeResponse requestLuke(ModifiableSolrParams extra) throws
Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.add(extra);
+
+// query() sends to control and a random shard with shards param, compares
responses
+handle.put("QTime", SKIPVAL);
+handle.put("index", SKIP);
+handle.put("shards", SKIP);
+// Detailed per-field stats (distinct, topTerms, histogram) are kept
per-shard in
+// distributed mode and intentionally excluded from the aggregated
top-level fields.
+// Local mode includes them inline, so skip them in the comparison.
+handle.put("distinct", SKIP);
+handle.put("topTerms", SKIP);
+handle.put("histogram", SKIP);
+QueryResponse qr = query(params);
+LukeResponse rsp = new LukeResponse();
+rsp.setResponse(qr.getResponse());
+
+return rsp;
+ }
+
+ private void assertLukeXPath(ModifiableSolrParams extra, String... xpaths)
throws Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.set("wt", "xml");
+params.set("shards", shards);
+params.add(extra);
+QueryRequest req = new QueryRequest(params);
+req.setResponseParser(new InputStreamResponseParser("xml"));
+NamedList raw = controlClient.request(req);
+String xml = InputStreamResponseParser.consumeResponseToString(raw);
+String failedXpath = BaseTestHarness.validateXPath(xml, xpaths);
+assertNull("XPath validation failed: " + failedXpath + "\nResponse:\n" +
xml, failedXpath);
+ }
+
+ private void indexTestData() throws Exception {
+for (int i = 0; i < NUM_DOCS; i++) {
+ index("id", String.valueOf(i), "name", "name_" + i, "subject", "subject
value " + (i % 5));
+}
+commit();
+ }
+
+ @Test
+ @ShardsFixed(num = 2)
+ public void testDistributedAggregate() throws Exception {
+indexTestData();
+
+LukeResponse rsp = requestLuke();
+
+assertEquals("aggregated numDocs should equal total docs", NUM_DOCS,
rsp.getNumDocs());
+assertTrue("aggregated maxDoc should be > 0", rsp.getMaxDoc() > 0);
+assertNotNull("deletedDocs should be present", rsp.getDeletedDocs());
+
+Map shardResponses = rsp.getShardResponses();
Review Comment:
I appreciate it _could_ be useful but I suspect many callers don't care
about it. I just wrote a program days ago to find out what fields have
docValues & terms index. I definitely didn't want per-shard details.
I suspect the admin UI wouldn't want it either, at least not by default. It
would add complexity to a UI to display.
It's also kind of noise even if it's "cheap".
--
This is
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3025624341
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import java.util.Map;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends
BaseDistributedSearchTestCase {
+
+ private static final Long NUM_DOCS = 20L;
+
+ public LukeRequestHandlerDistribTest() {
+fixShardCount(2);
+ }
+
+ private LukeResponse requestLuke() throws Exception {
+return requestLuke(new ModifiableSolrParams());
+ }
+
+ private LukeResponse requestLuke(ModifiableSolrParams extra) throws
Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.add(extra);
+
+// query() sends to control and a random shard with shards param, compares
responses
+handle.put("QTime", SKIPVAL);
+handle.put("index", SKIP);
+handle.put("shards", SKIP);
+// Detailed per-field stats (distinct, topTerms, histogram) are kept
per-shard in
+// distributed mode and intentionally excluded from the aggregated
top-level fields.
+// Local mode includes them inline, so skip them in the comparison.
+handle.put("distinct", SKIP);
+handle.put("topTerms", SKIP);
+handle.put("histogram", SKIP);
+QueryResponse qr = query(params);
+LukeResponse rsp = new LukeResponse();
+rsp.setResponse(qr.getResponse());
+
+return rsp;
+ }
+
+ private void assertLukeXPath(ModifiableSolrParams extra, String... xpaths)
throws Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.set("wt", "xml");
+params.set("shards", shards);
+params.add(extra);
+QueryRequest req = new QueryRequest(params);
+req.setResponseParser(new InputStreamResponseParser("xml"));
+NamedList raw = controlClient.request(req);
+String xml = InputStreamResponseParser.consumeResponseToString(raw);
+String failedXpath = BaseTestHarness.validateXPath(xml, xpaths);
+assertNull("XPath validation failed: " + failedXpath + "\nResponse:\n" +
xml, failedXpath);
+ }
+
+ private void indexTestData() throws Exception {
+for (int i = 0; i < NUM_DOCS; i++) {
+ index("id", String.valueOf(i), "name", "name_" + i, "subject", "subject
value " + (i % 5));
+}
+commit();
+ }
+
+ @Test
+ @ShardsFixed(num = 2)
+ public void testDistributedAggregate() throws Exception {
+indexTestData();
+
+LukeResponse rsp = requestLuke();
+
+assertEquals("aggregated numDocs should equal total docs", NUM_DOCS,
rsp.getNumDocs());
+assertTrue("aggregated maxDoc should be > 0", rsp.getMaxDoc() > 0);
+assertNotNull("deletedDocs should be present", rsp.getDeletedDocs());
+
+Map shardResponses = rsp.getShardResponses();
Review Comment:
This includes index information that comes with the default luke response
(as well as with show=index). I didn't want to just omit all those fields that
couldn't be easily merged in the distributed mode (version, indexCommit,
isCurrent, segmentsFile, directory, ...). I suppose you could but it didn't
feel intuitive and that info could be seen as useful.
I think the initial motivation fo
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3025624341
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import java.util.Map;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends
BaseDistributedSearchTestCase {
+
+ private static final Long NUM_DOCS = 20L;
+
+ public LukeRequestHandlerDistribTest() {
+fixShardCount(2);
+ }
+
+ private LukeResponse requestLuke() throws Exception {
+return requestLuke(new ModifiableSolrParams());
+ }
+
+ private LukeResponse requestLuke(ModifiableSolrParams extra) throws
Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.add(extra);
+
+// query() sends to control and a random shard with shards param, compares
responses
+handle.put("QTime", SKIPVAL);
+handle.put("index", SKIP);
+handle.put("shards", SKIP);
+// Detailed per-field stats (distinct, topTerms, histogram) are kept
per-shard in
+// distributed mode and intentionally excluded from the aggregated
top-level fields.
+// Local mode includes them inline, so skip them in the comparison.
+handle.put("distinct", SKIP);
+handle.put("topTerms", SKIP);
+handle.put("histogram", SKIP);
+QueryResponse qr = query(params);
+LukeResponse rsp = new LukeResponse();
+rsp.setResponse(qr.getResponse());
+
+return rsp;
+ }
+
+ private void assertLukeXPath(ModifiableSolrParams extra, String... xpaths)
throws Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.set("wt", "xml");
+params.set("shards", shards);
+params.add(extra);
+QueryRequest req = new QueryRequest(params);
+req.setResponseParser(new InputStreamResponseParser("xml"));
+NamedList raw = controlClient.request(req);
+String xml = InputStreamResponseParser.consumeResponseToString(raw);
+String failedXpath = BaseTestHarness.validateXPath(xml, xpaths);
+assertNull("XPath validation failed: " + failedXpath + "\nResponse:\n" +
xml, failedXpath);
+ }
+
+ private void indexTestData() throws Exception {
+for (int i = 0; i < NUM_DOCS; i++) {
+ index("id", String.valueOf(i), "name", "name_" + i, "subject", "subject
value " + (i % 5));
+}
+commit();
+ }
+
+ @Test
+ @ShardsFixed(num = 2)
+ public void testDistributedAggregate() throws Exception {
+indexTestData();
+
+LukeResponse rsp = requestLuke();
+
+assertEquals("aggregated numDocs should equal total docs", NUM_DOCS,
rsp.getNumDocs());
+assertTrue("aggregated maxDoc should be > 0", rsp.getMaxDoc() > 0);
+assertNotNull("deletedDocs should be present", rsp.getDeletedDocs());
+
+Map shardResponses = rsp.getShardResponses();
Review Comment:
This includes index information that comes with the default luke response
(as well as with show=index). I didn't want to just omit all those fields that
couldn't be easily merged in the distributed mode (version, indexCommit,
segmentsFile, directory, ...). I suppose you could but it didn't feel intuitive
and that info could be seen as useful.
I think the initial motivation for SHARDS_IN
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3025624341
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import java.util.Map;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends
BaseDistributedSearchTestCase {
+
+ private static final Long NUM_DOCS = 20L;
+
+ public LukeRequestHandlerDistribTest() {
+fixShardCount(2);
+ }
+
+ private LukeResponse requestLuke() throws Exception {
+return requestLuke(new ModifiableSolrParams());
+ }
+
+ private LukeResponse requestLuke(ModifiableSolrParams extra) throws
Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.add(extra);
+
+// query() sends to control and a random shard with shards param, compares
responses
+handle.put("QTime", SKIPVAL);
+handle.put("index", SKIP);
+handle.put("shards", SKIP);
+// Detailed per-field stats (distinct, topTerms, histogram) are kept
per-shard in
+// distributed mode and intentionally excluded from the aggregated
top-level fields.
+// Local mode includes them inline, so skip them in the comparison.
+handle.put("distinct", SKIP);
+handle.put("topTerms", SKIP);
+handle.put("histogram", SKIP);
+QueryResponse qr = query(params);
+LukeResponse rsp = new LukeResponse();
+rsp.setResponse(qr.getResponse());
+
+return rsp;
+ }
+
+ private void assertLukeXPath(ModifiableSolrParams extra, String... xpaths)
throws Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.set("wt", "xml");
+params.set("shards", shards);
+params.add(extra);
+QueryRequest req = new QueryRequest(params);
+req.setResponseParser(new InputStreamResponseParser("xml"));
+NamedList raw = controlClient.request(req);
+String xml = InputStreamResponseParser.consumeResponseToString(raw);
+String failedXpath = BaseTestHarness.validateXPath(xml, xpaths);
+assertNull("XPath validation failed: " + failedXpath + "\nResponse:\n" +
xml, failedXpath);
+ }
+
+ private void indexTestData() throws Exception {
+for (int i = 0; i < NUM_DOCS; i++) {
+ index("id", String.valueOf(i), "name", "name_" + i, "subject", "subject
value " + (i % 5));
+}
+commit();
+ }
+
+ @Test
+ @ShardsFixed(num = 2)
+ public void testDistributedAggregate() throws Exception {
+indexTestData();
+
+LukeResponse rsp = requestLuke();
+
+assertEquals("aggregated numDocs should equal total docs", NUM_DOCS,
rsp.getNumDocs());
+assertTrue("aggregated maxDoc should be > 0", rsp.getMaxDoc() > 0);
+assertNotNull("deletedDocs should be present", rsp.getDeletedDocs());
+
+Map shardResponses = rsp.getShardResponses();
Review Comment:
This includes index information that comes with the default luke response
(no special flags). I didn't want to just omit all those fields that couldn't
be easily merged in the distributed mode (version, indexCommit, segmentsFile,
directory, ...). I suppose you could but it didn't feel intuitive and that info
could be seen as useful.
I think the initial motivation for SHARDS_INFO is a li
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3025301863
##
solr/solrj/src/java/org/apache/solr/client/solrj/response/LukeResponse.java:
##
@@ -113,11 +115,12 @@ public static class FieldInfo implements Serializable {
String name;
String type;
String schema;
-int docs;
+long docs;
int distinct;
EnumSet flags;
boolean cacheableFaceting;
NamedList topTerms;
+Map extras = new HashMap<>();
Review Comment:
maybe use TreeMap for consistent ordering?
##
solr/solrj/src/java/org/apache/solr/client/solrj/response/LukeResponse.java:
##
@@ -256,16 +277,25 @@ public String getIndexDirectory() {
return (String) indexInfo.get("directory");
}
- public Integer getNumDocs() {
+ private Long getIndexLong(String key) {
Review Comment:
the word "Index" here confuses me.
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,522 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import java.util.Map;
+import org.apache.solr.BaseDistributedSearchTestCase;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends
BaseDistributedSearchTestCase {
+
+ private static final Long NUM_DOCS = 20L;
+
+ public LukeRequestHandlerDistribTest() {
+fixShardCount(2);
+ }
+
+ private LukeResponse requestLuke() throws Exception {
+return requestLuke(new ModifiableSolrParams());
+ }
+
+ private LukeResponse requestLuke(ModifiableSolrParams extra) throws
Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.add(extra);
+
+// query() sends to control and a random shard with shards param, compares
responses
+handle.put("QTime", SKIPVAL);
+handle.put("index", SKIP);
+handle.put("shards", SKIP);
+// Detailed per-field stats (distinct, topTerms, histogram) are kept
per-shard in
+// distributed mode and intentionally excluded from the aggregated
top-level fields.
+// Local mode includes them inline, so skip them in the comparison.
+handle.put("distinct", SKIP);
+handle.put("topTerms", SKIP);
+handle.put("histogram", SKIP);
+QueryResponse qr = query(params);
+LukeResponse rsp = new LukeResponse();
+rsp.setResponse(qr.getResponse());
+
+return rsp;
+ }
+
+ private void assertLukeXPath(ModifiableSolrParams extra, String... xpaths)
throws Exception {
+ModifiableSolrParams params = new ModifiableSolrParams();
+params.set("qt", "/admin/luke");
+params.set("numTerms", "0");
+params.set("wt", "xml");
+params.set("shards", shards);
+params.add(extra);
+QueryRequest req = new QueryRequest(params);
+req.setResponseParser(new InputStreamResponseParser("xml"));
+NamedList raw = controlClient.request(req);
+String xml = InputStreamResponseParser.consumeResponseToString(raw);
+String failedXpath = BaseTestHarness.validateXPath(xml, xpaths);
+assertNull("XPath validation failed: " + failedXpath + "\nResponse:\n" +
xml, failedXpath);
+ }
+
+ private void indexTestData() throws Exception {
+for (int i = 0; i < NUM_DOCS; i++) {
+ index("id", String.valueOf(i), "name", "name_" + i, "subject", "subject
value " + (i % 5));
+}
+commit();
+ }
+
+ @Test
+ @ShardsFixed(num = 2)
+ public void testDistributedAg
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3024754188
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import static org.apache.solr.common.params.CommonParams.DISTRIB;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends SolrCloudTestCase {
Review Comment:
Also to leverage the `BaseDistributedSearchTestCase` machinery I had to read
the luke response as a `QueryResponse` first which felt weird but probably ok.
Since this is doc-search centric it makes sense why it returns a
`QueryResponse` which I use as a conduit for creating a `LukeResponse`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3024743920
##
solr/core/src/test/org/apache/solr/handler/admin/LukeRequestHandlerDistribTest.java:
##
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import static org.apache.solr.common.params.CommonParams.DISTRIB;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.InputStreamResponseParser;
+import org.apache.solr.client.solrj.response.LukeResponse;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.core.SolrCore;
+import org.apache.solr.embedded.JettySolrRunner;
+import org.apache.solr.request.SolrQueryRequestBase;
+import org.apache.solr.update.AddUpdateCommand;
+import org.apache.solr.update.CommitUpdateCommand;
+import org.apache.solr.util.BaseTestHarness;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class LukeRequestHandlerDistribTest extends SolrCloudTestCase {
Review Comment:
As noted earlier, some things are not the same in local vs distributed mode.
Thus we skip those sections when running the implicit comparisons within
`BaseDistributedSearchTestCase:query(...)`. Just noting this for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3018556574
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
Admittedly this is out of scope; lets table this idea of mine for another
time. I like the idea of making this a V2 thing in terms of migration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3018327740
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
You can run [this
test](https://github.com/kotman12/solr/commit/c729a424000d8a3ab601ebb8a004f448ecb7299c)
to see what I am getting at re the problem with `CloudSolrClient`.
Edit:
Also nothing is stopping you from trying to access a [specific
core](https://github.com/kotman12/solr/commit/0c2dca4b446fdde09a5ae3c278288752c92ff5f7)
with `CloudSolrClient` so my proposed workaround isn't really good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3018327740
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
You can run [this
test](https://github.com/kotman12/solr/commit/c729a424000d8a3ab601ebb8a004f448ecb7299c)
to see what I am getting at re the problem with `CloudSolrClient`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3018229614
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
@dsmiley here is a really [rough
attempt](https://github.com/kotman12/solr/commit/b86054dd3db26e3eb933983f6615df8d4fbfb2c9)
with using the request path to determine distrib. I am not really sure what
you had in mind so I figured we could hash out the details over code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3018229614
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
@dsmiley here is a really [rough
attempt](https://github.com/kotman12/solr/commit/b86054dd3db26e3eb933983f6615df8d4fbfb2c9)
with using the request path to determine distrib. I am not really sure what
you had in mind so I figured we could hash out the details over code. One of
the flies in the ointment is the CloudSolrClient which hits the cores directly
so if you are using it client-side the solr server now will incorrectly assume
you are `distrib=false`. The way I worked around this for now is changing
CloudSolrClient to set distrib explicitly when it's hitting a collection. Ofc
there may be other approaches to consider here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149: URL: https://github.com/apache/solr/pull/4149#discussion_r3012872093 ## changelog/unreleased/PR#4149-distributed-luke: ## Review Comment: when there's a JIRA issue, the file name & links should soley reflect the JIRA. If you create feature branches starting with a SOLR-1234 then you'll get this handled for you automatically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3012804156
##
solr/core/src/test/org/apache/solr/handler/admin/LukeHandlerCloudTest.java:
##
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import static org.apache.solr.common.params.CommonParams.DISTRIB;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Cloud-specific Luke tests that require SolrCloud features like managed
schema and Schema API. */
+public class LukeHandlerCloudTest extends SolrCloudTestCase {
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+configureCluster(2).addConfig("managed",
configset("cloud-managed")).configure();
+ }
+
+ @AfterClass
+ public static void afterClass() throws Exception {
+shutdownCluster();
Review Comment:
is that really necessary?
##
solr/core/src/test/org/apache/solr/handler/admin/LukeHandlerCloudTest.java:
##
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.handler.admin;
+
+import static org.apache.solr.common.params.CommonParams.DISTRIB;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.SolrQuery;
+import org.apache.solr.client.solrj.request.schema.SchemaRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.cloud.DocCollection;
+import org.apache.solr.common.cloud.Replica;
+import org.apache.solr.common.cloud.Slice;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Cloud-specific Luke tests that require SolrCloud features like managed
schema and Schema API. */
+public class LukeHandlerCloudTest extends SolrCloudTestCase {
Review Comment:
nice!
##
solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java:
##
@@ -544,10 +561,35 @@ private void aggregateShardField(
}
}
-Long docsAsLong = fi.getDocsAsLong();
-if (docsAsLong != null) {
- fieldData.aggregated.compute(
- KEY_DOCS_AS_LONG, (key, v
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3012372289
##
solr/solrj/src/java/org/apache/solr/client/solrj/response/LukeResponse.java:
##
@@ -257,19 +285,42 @@ public String getIndexDirectory() {
return (String) indexInfo.get("directory");
}
- public Integer getNumDocs() {
+ private Long getIndexLong(String key) {
if (indexInfo == null) return null;
-return (Integer) indexInfo.get("numDocs");
+Number n = (Number) indexInfo.get(key);
+return n != null ? n.longValue() : null;
+ }
+
+ public Integer getNumDocs() {
Review Comment:
@dsmiley I took your advice from another comment to gate the response based
on user agent's solr version. When I detect that the major version is less
than, say, 10.2.0 I narrow the long `docs` to int whenever possible so as to
break as few people as possible. Otherwise I really think this could be a
problem assuming we make `distrib=true` the default (at least when addressing a
collection as per your other comment).
Edit:
One extra thing to consider here is if the narrowing function belongs in
some util class assuming of course it even makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3012372289
##
solr/solrj/src/java/org/apache/solr/client/solrj/response/LukeResponse.java:
##
@@ -257,19 +285,42 @@ public String getIndexDirectory() {
return (String) indexInfo.get("directory");
}
- public Integer getNumDocs() {
+ private Long getIndexLong(String key) {
if (indexInfo == null) return null;
-return (Integer) indexInfo.get("numDocs");
+Number n = (Number) indexInfo.get(key);
+return n != null ? n.longValue() : null;
+ }
+
+ public Integer getNumDocs() {
Review Comment:
@dsmiley I took your advice from another comment to gate the response based
on user agent's solr version. When I detect that the major version is less
than, say, 10.2.0 I narrow the long docs to int whenever possible so as to
break as few people as possible. Otherwise I really think this could be a
problem assuming we make `distrib=true` the default (at least when addressing a
collection as per your other comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3012372289
##
solr/solrj/src/java/org/apache/solr/client/solrj/response/LukeResponse.java:
##
@@ -257,19 +285,42 @@ public String getIndexDirectory() {
return (String) indexInfo.get("directory");
}
- public Integer getNumDocs() {
+ private Long getIndexLong(String key) {
if (indexInfo == null) return null;
-return (Integer) indexInfo.get("numDocs");
+Number n = (Number) indexInfo.get(key);
+return n != null ? n.longValue() : null;
+ }
+
+ public Integer getNumDocs() {
Review Comment:
@dsmiley I took your advice from another comment to gate the response based
on user agent's solr version. When I detect that the major version is less
than, say, 10.2.0 I narrow the long `docs` to int whenever possible so as to
break as few people as possible. Otherwise I really think this could be a
problem assuming we make `distrib=true` the default (at least when addressing a
collection as per your other comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r3010932063
##
solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java:
##
@@ -192,8 +238,346 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
info.add(
"NOTE",
"Document Frequency (df) is not updated when a document is marked for
deletion. df values include deleted documents.");
-rsp.add("info", info);
+rsp.add(RSP_INFO, info);
+rsp.setHttpCaching(false);
+ }
+
+ /** Per-field accumulation state across shards: aggregated response data and
field validation. */
+ private static class AggregatedFieldData {
+final SimpleOrderedMap aggregated = new SimpleOrderedMap<>();
+final String originalShardAddr;
+final LukeResponse.FieldInfo originalFieldInfo;
+private Object indexFlags;
+private String indexFlagsShardAddr;
+
+AggregatedFieldData(String shardAddr, LukeResponse.FieldInfo fieldInfo) {
+ this.originalShardAddr = shardAddr;
+ this.originalFieldInfo = fieldInfo;
+ Object flags = fieldInfo.getExtras().get(KEY_INDEX_FLAGS);
+ if (flags != null) {
+this.indexFlags = flags;
+this.indexFlagsShardAddr = shardAddr;
+ }
+}
+ }
+
+ private static class ShardData {
+final String shardAddr;
+final Map shardFieldInfo;
+private NamedList indexInfo;
+private SimpleOrderedMap detailedFields;
+
+ShardData(String shardAddr, Map
shardFieldInfo) {
+ this.shardAddr = shardAddr;
+ this.shardFieldInfo = shardFieldInfo;
+}
+
+void setIndexInfo(NamedList indexInfo) {
+ this.indexInfo = indexInfo;
+}
+
+void addDetailedFieldInfo(String fieldName, SimpleOrderedMap
fieldStats) {
+ if (detailedFields == null) {
+detailedFields = new SimpleOrderedMap<>();
+ }
+ detailedFields.add(fieldName, fieldStats);
+}
+
+SimpleOrderedMap toResponseEntry() {
+ SimpleOrderedMap entry = new SimpleOrderedMap<>();
+ if (indexInfo != null) {
+entry.add(RSP_INDEX, indexInfo);
+ }
+ if (detailedFields != null) {
+entry.add(RSP_FIELDS, detailedFields);
+ }
+ return entry;
+}
+ }
+
+ /**
+ * @return true if the request was handled in distributed mode, false if
prepDistributed
+ * short-circuited (e.g. single-shard collection) and the caller should
fall through to local
+ * logic.
+ */
+ private boolean handleDistributed(SolrQueryRequest req, SolrQueryResponse
rsp) {
+SolrParams reqParams = req.getParams();
+
+// docId is a Lucene-internal integer, not meaningful across shards
+if (reqParams.getInt(DOC_ID) != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "docId parameter is not supported in distributed mode."
+ + " Use the id parameter to look up documents by their Solr
unique key.");
+}
+
+ShardHandler shardHandler = shardHandlerFactory.getShardHandler();
+ResponseBuilder rb = new ResponseBuilder(req, rsp,
Collections.emptyList());
+shardHandler.prepDistributed(rb);
+
+String[] shards = rb.shards;
+if (shards == null || shards.length == 0) {
+ return false;
+}
+
+ShardRequest sreq = new ShardRequest();
+sreq.shards = shards;
+sreq.actualShards = shards;
+sreq.responses = new ArrayList<>(shards.length);
+
+String reqPath = (String) req.getContext().get(PATH);
+
+for (String shard : shards) {
+ ModifiableSolrParams params = new ModifiableSolrParams(reqParams);
+ params.set(CommonParams.QT, reqPath);
+ ShardHandler.setShardAttributesToParams(params, sreq.purpose);
+ shardHandler.submit(sreq, shard, params);
+}
+
+ShardResponse lastSrsp = shardHandler.takeCompletedOrError();
+if (lastSrsp == null) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, "No responses received
from shards");
+}
+List responses = sreq.responses;
+for (ShardResponse srsp : responses) {
+ if (srsp.getException() != null) {
+shardHandler.cancelAll();
+if (srsp.getException() instanceof SolrException) {
+ throw (SolrException) srsp.getException();
+}
+throw new SolrException(ErrorCode.SERVER_ERROR, srsp.getException());
+ }
+}
+
+aggregateDistributedResponses(rsp, responses);
rsp.setHttpCaching(false);
+return true;
+ }
+
+ private static String shardAddress(ShardResponse srsp) {
+return srsp.getShardAddress() != null ? srsp.getShardAddress() :
srsp.getShard();
+ }
+
+ private void aggregateDistributedResponses(SolrQueryResponse rsp,
List responses) {
+
+if (!responses.isEmpty()) {
+ ShardResponse firstRsp = responses.getFirst();
+ NamedList firstShardRsp =
firstRsp.getSolrResponse().getResponse();
+ if (firstShardRsp == null) {
+throw new SolrException(
+ErrorCode.SERVER_ER
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2968040137
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
Ok I do see there is a `prefix` in `V2HttpSolrCall` that could be used for
this. But I am not sure I see the v1 equivalent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2967996069
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
Is there an existing pattern of what you are describing or is this something
new? I couldn't find anything quite like this in existing solr handlers. Also,
is this something that is easier to do specifically with `V2HttpCall`? I wasn't
able to find anything the request context that could easily distinguish what
the original path was (in v1 at least). I'm kind of assuming this would require
a routing layer change but before I dive in that direction I wanted to see if
there was something I was missing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2967444380
##
solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java:
##
@@ -192,8 +238,346 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
info.add(
"NOTE",
"Document Frequency (df) is not updated when a document is marked for
deletion. df values include deleted documents.");
-rsp.add("info", info);
+rsp.add(RSP_INFO, info);
+rsp.setHttpCaching(false);
+ }
+
+ /** Per-field accumulation state across shards: aggregated response data and
field validation. */
+ private static class AggregatedFieldData {
+final SimpleOrderedMap aggregated = new SimpleOrderedMap<>();
+final String originalShardAddr;
+final LukeResponse.FieldInfo originalFieldInfo;
+private Object indexFlags;
+private String indexFlagsShardAddr;
Review Comment:
I did a little more testing on this and my initial theory that this
distinction only exists because we get the first *live* doc rather than the
first doc is _incorrect_. It turns out a fully deleted field will lack terms
even though it _does_ still have schema flags. If a field lacks terms then we
can't even get _any_ doc for it (live or not). A workaround for this would be
to ignore all fields that don't have terms (that _should_ have terms) but this
can get complicated for other reasons. As such, I am leaning toward keeping it
the way it is currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2957019085
##
solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java:
##
@@ -192,8 +238,346 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
info.add(
"NOTE",
"Document Frequency (df) is not updated when a document is marked for
deletion. df values include deleted documents.");
-rsp.add("info", info);
+rsp.add(RSP_INFO, info);
+rsp.setHttpCaching(false);
+ }
+
+ /** Per-field accumulation state across shards: aggregated response data and
field validation. */
+ private static class AggregatedFieldData {
+final SimpleOrderedMap aggregated = new SimpleOrderedMap<>();
+final String originalShardAddr;
+final LukeResponse.FieldInfo originalFieldInfo;
+private Object indexFlags;
+private String indexFlagsShardAddr;
Review Comment:
If we're not sure it matters, let's not over-complicate this. Do the simple
thing ignore "live docs". We're not boxed in from realizing later we'd like
something more complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2956992015
##
solr/solr-ref-guide/modules/indexing-guide/pages/luke-request-handler.adoc:
##
@@ -118,3 +130,43 @@ Alternatively, to work through the Lucene native id:
http://localhost:8983/solr/techproducts/admin/luke?fl=manu&docId=0
From SolrJ, you can access /luke using the
{solr-javadocs}/solrj/org/apache/solr/client/solrj/request/LukeRequest.html[`LukeRequest`]
object.
+
+== Distributed Mode (SolrCloud)
+
+When running in SolrCloud, the Luke handler can aggregate results from all
shards in a collection by setting `distrib=true`.
+By default, `distrib` is `false` and the handler inspects only the local
shard's index.
+
+To get a collection-wide view:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true
+
+To get detailed field statistics across all shards for a specific field:
+
+[source,text]
+http://localhost:8983/solr/techproducts/admin/luke?distrib=true&fl=manu
Review Comment:
ehh... I think a 10.1 we can change something of this nature. It's a matter
of defaults with a clear way a user to get the old behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2956981246
##
solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java:
##
@@ -192,8 +238,346 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
info.add(
"NOTE",
"Document Frequency (df) is not updated when a document is marked for
deletion. df values include deleted documents.");
-rsp.add("info", info);
+rsp.add(RSP_INFO, info);
+rsp.setHttpCaching(false);
+ }
+
+ /** Per-field accumulation state across shards: aggregated response data and
field validation. */
+ private static class AggregatedFieldData {
+final SimpleOrderedMap aggregated = new SimpleOrderedMap<>();
+final String originalShardAddr;
+final LukeResponse.FieldInfo originalFieldInfo;
+private Object indexFlags;
+private String indexFlagsShardAddr;
+
+AggregatedFieldData(String shardAddr, LukeResponse.FieldInfo fieldInfo) {
+ this.originalShardAddr = shardAddr;
+ this.originalFieldInfo = fieldInfo;
+ Object flags = fieldInfo.getExtras().get(KEY_INDEX_FLAGS);
+ if (flags != null) {
+this.indexFlags = flags;
+this.indexFlagsShardAddr = shardAddr;
+ }
+}
+ }
+
+ private static class ShardData {
+final String shardAddr;
+final Map shardFieldInfo;
+private NamedList indexInfo;
+private SimpleOrderedMap detailedFields;
+
+ShardData(String shardAddr, Map
shardFieldInfo) {
+ this.shardAddr = shardAddr;
+ this.shardFieldInfo = shardFieldInfo;
+}
+
+void setIndexInfo(NamedList indexInfo) {
+ this.indexInfo = indexInfo;
+}
+
+void addDetailedFieldInfo(String fieldName, SimpleOrderedMap
fieldStats) {
+ if (detailedFields == null) {
+detailedFields = new SimpleOrderedMap<>();
+ }
+ detailedFields.add(fieldName, fieldStats);
+}
+
+SimpleOrderedMap toResponseEntry() {
+ SimpleOrderedMap entry = new SimpleOrderedMap<>();
+ if (indexInfo != null) {
+entry.add(RSP_INDEX, indexInfo);
+ }
+ if (detailedFields != null) {
+entry.add(RSP_FIELDS, detailedFields);
+ }
+ return entry;
+}
+ }
+
+ /**
+ * @return true if the request was handled in distributed mode, false if
prepDistributed
+ * short-circuited (e.g. single-shard collection) and the caller should
fall through to local
+ * logic.
+ */
+ private boolean handleDistributed(SolrQueryRequest req, SolrQueryResponse
rsp) {
+SolrParams reqParams = req.getParams();
+
+// docId is a Lucene-internal integer, not meaningful across shards
+if (reqParams.getInt(DOC_ID) != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "docId parameter is not supported in distributed mode."
+ + " Use the id parameter to look up documents by their Solr
unique key.");
+}
+
+ShardHandler shardHandler = shardHandlerFactory.getShardHandler();
+ResponseBuilder rb = new ResponseBuilder(req, rsp,
Collections.emptyList());
+shardHandler.prepDistributed(rb);
+
+String[] shards = rb.shards;
+if (shards == null || shards.length == 0) {
+ return false;
+}
+
+ShardRequest sreq = new ShardRequest();
+sreq.shards = shards;
+sreq.actualShards = shards;
+sreq.responses = new ArrayList<>(shards.length);
+
+String reqPath = (String) req.getContext().get(PATH);
+
+for (String shard : shards) {
+ ModifiableSolrParams params = new ModifiableSolrParams(reqParams);
+ params.set(CommonParams.QT, reqPath);
+ ShardHandler.setShardAttributesToParams(params, sreq.purpose);
+ shardHandler.submit(sreq, shard, params);
+}
+
+ShardResponse lastSrsp = shardHandler.takeCompletedOrError();
+if (lastSrsp == null) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, "No responses received
from shards");
+}
+List responses = sreq.responses;
+for (ShardResponse srsp : responses) {
+ if (srsp.getException() != null) {
+shardHandler.cancelAll();
+if (srsp.getException() instanceof SolrException) {
+ throw (SolrException) srsp.getException();
+}
+throw new SolrException(ErrorCode.SERVER_ERROR, srsp.getException());
+ }
+}
+
+aggregateDistributedResponses(rsp, responses);
rsp.setHttpCaching(false);
+return true;
+ }
+
+ private static String shardAddress(ShardResponse srsp) {
+return srsp.getShardAddress() != null ? srsp.getShardAddress() :
srsp.getShard();
+ }
+
+ private void aggregateDistributedResponses(SolrQueryResponse rsp,
List responses) {
+
+if (!responses.isEmpty()) {
+ ShardResponse firstRsp = responses.getFirst();
+ NamedList firstShardRsp =
firstRsp.getSolrResponse().getResponse();
+ if (firstShardRsp == null) {
+throw new SolrException(
+ErrorCode.SERVER_ERR
Re: [PR] SOLR-8127 Distributed Luke [solr]
dsmiley commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2956972195
##
solr/solrj/src/java/org/apache/solr/client/solrj/response/LukeResponse.java:
##
@@ -257,19 +285,42 @@ public String getIndexDirectory() {
return (String) indexInfo.get("directory");
}
- public Integer getNumDocs() {
+ private Long getIndexLong(String key) {
if (indexInfo == null) return null;
-return (Integer) indexInfo.get("numDocs");
+Number n = (Number) indexInfo.get(key);
+return n != null ? n.longValue() : null;
+ }
+
+ public Integer getNumDocs() {
Review Comment:
That's minor enough for a 10.1 change. What I'd like to retain is a SolrJ
10.0 JAR using default javabin to talk to Solr 10.1. XML & JSON consistency as
well but that is not expected to change with Integer vs Long... it's javabin
that's the risk (if I recall).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-8127 Distributed Luke [solr]
kotman12 commented on code in PR #4149:
URL: https://github.com/apache/solr/pull/4149#discussion_r2898200193
##
solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java:
##
@@ -192,8 +238,346 @@ public void handleRequestBody(SolrQueryRequest req,
SolrQueryResponse rsp) throw
info.add(
"NOTE",
"Document Frequency (df) is not updated when a document is marked for
deletion. df values include deleted documents.");
-rsp.add("info", info);
+rsp.add(RSP_INFO, info);
+rsp.setHttpCaching(false);
+ }
+
+ /** Per-field accumulation state across shards: aggregated response data and
field validation. */
+ private static class AggregatedFieldData {
+final SimpleOrderedMap aggregated = new SimpleOrderedMap<>();
+final String originalShardAddr;
+final LukeResponse.FieldInfo originalFieldInfo;
+private Object indexFlags;
+private String indexFlagsShardAddr;
Review Comment:
A different approach might be to simply get "first doc" from the index
instead of "first _live_ doc". I don't know if this has other implications. But
it would eliminate this edge case. It's funny that we already touched first
live doc logic here https://github.com/apache/solr/pull/4157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
