Re: [PR] SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param [solr]
gaojiabao1991 commented on PR #3181: URL: https://github.com/apache/solr/pull/3181#issuecomment-2676365101 > I'll be merging this nice fix into 9.8.1 Thanks so much, David! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param [solr]
dsmiley merged PR #3181: URL: https://github.com/apache/solr/pull/3181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param [solr]
dsmiley commented on PR #3181: URL: https://github.com/apache/solr/pull/3181#issuecomment-2676315964 I'll be merging this nice fix into 9.8.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param [solr]
gaojiabao1991 commented on code in PR #3181:
URL: https://github.com/apache/solr/pull/3181#discussion_r1955480881
##
solr/core/src/test/org/apache/solr/search/TestRerankOnSolrCloud.java:
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search;
+
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.SolrParams;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Test for ReRankQParserPlugin functionality */
+public class TestRerankOnSolrCloud extends SolrCloudTestCase {
+
+ private static final String COLLECTION_NAME = "rerank_test";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+System.setProperty("managed.schema.mutable", "true");
+configureCluster(2)
+.addConfig(
+"conf1",
TEST_PATH().resolve("configsets").resolve("cloud-managed").resolve("conf"))
+.configure();
+
+// Create collection
+CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf1", 2, 1)
+.process(cluster.getSolrClient());
+
+cluster.waitForActiveCollection(COLLECTION_NAME, 2, 2);
+
+// Prepare test data
+CloudSolrClient client = cluster.getSolrClient();
+UpdateRequest updateRequest = new UpdateRequest();
+
+// Add test documents
+for (int i = 1; i <= 5; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField(CommonParams.ID, String.valueOf(i));
+ doc.addField("test_s", "hello");
+ doc.addField("test_i", i);
+ updateRequest.add(doc);
+}
+
+updateRequest.process(client, COLLECTION_NAME);
+client.commit(COLLECTION_NAME);
+ }
+
+ @Test
+ public void testLargeReRankDocsParamWithoutSort() throws Exception {
+// Build query parameters
+SolrParams params =
+params(
+CommonParams.Q, "*:*",
+CommonParams.FL, "id,test_s,test_i,score",
+CommonParams.RQ,
+"{!rerank reRankQuery='{!func} 100' reRankDocs=10
reRankWeight=2}");
+
+QueryRequest req = new QueryRequest(params);
+QueryResponse rsp = req.process(cluster.getSolrClient(), COLLECTION_NAME);
+
+// Test passes if execution completes without OOM
+Assert.assertTrue(true);
Review Comment:
Agreed. My unit tests are non-standard and were only intended to help others
confirm the issue. They should not be included in the PR as code to be merged.
I also agree with David's point @dsmiley that this modification is
relatively straightforward, so once the issue is confirmed, we can simply
decide on the logic and easily fix it. Trying to specifically test this small
point might actually complicate things.
Therefore, I have simply removed the unit tests from the PR and added them
to a separate branch:
https://github.com/apache/solr/compare/main...gaojiabao1991:solr:17670-extra-test
The tests in this branch are only for reviewers to easily confirm the issue
and will not be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param [solr]
dsmiley commented on code in PR #3181:
URL: https://github.com/apache/solr/pull/3181#discussion_r1955109776
##
solr/core/src/test/org/apache/solr/search/TestRerankOnSolrCloud.java:
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search;
+
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.SolrParams;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Test for ReRankQParserPlugin functionality */
+public class TestRerankOnSolrCloud extends SolrCloudTestCase {
+
+ private static final String COLLECTION_NAME = "rerank_test";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+System.setProperty("managed.schema.mutable", "true");
+configureCluster(2)
+.addConfig(
+"conf1",
TEST_PATH().resolve("configsets").resolve("cloud-managed").resolve("conf"))
+.configure();
+
+// Create collection
+CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf1", 2, 1)
+.process(cluster.getSolrClient());
+
+cluster.waitForActiveCollection(COLLECTION_NAME, 2, 2);
+
+// Prepare test data
+CloudSolrClient client = cluster.getSolrClient();
+UpdateRequest updateRequest = new UpdateRequest();
+
+// Add test documents
+for (int i = 1; i <= 5; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField(CommonParams.ID, String.valueOf(i));
+ doc.addField("test_s", "hello");
+ doc.addField("test_i", i);
+ updateRequest.add(doc);
+}
+
+updateRequest.process(client, COLLECTION_NAME);
+client.commit(COLLECTION_NAME);
+ }
+
+ @Test
+ public void testLargeReRankDocsParamWithoutSort() throws Exception {
+// Build query parameters
+SolrParams params =
+params(
+CommonParams.Q, "*:*",
+CommonParams.FL, "id,test_s,test_i,score",
+CommonParams.RQ,
+"{!rerank reRankQuery='{!func} 100' reRankDocs=10
reRankWeight=2}");
+
+QueryRequest req = new QueryRequest(params);
+QueryResponse rsp = req.process(cluster.getSolrClient(), COLLECTION_NAME);
+
+// Test passes if execution completes without OOM
+Assert.assertTrue(true);
Review Comment:
IMO no test is needed.
Furthermore this functionality has nothing to do with SolrCloud (or even
distributed search?), so doesn't need the overhead of a SolrCloud test (or even
Jetty for that matter) if we did want to test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17670: Fix unnecessary memory allocation caused by a large reRankDocs param [solr]
risdenk commented on code in PR #3181:
URL: https://github.com/apache/solr/pull/3181#discussion_r1954932905
##
solr/core/src/test/org/apache/solr/search/TestRerankOnSolrCloud.java:
##
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.search;
+
+import org.apache.solr.client.solrj.impl.CloudSolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.client.solrj.response.QueryResponse;
+import org.apache.solr.cloud.SolrCloudTestCase;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.SolrParams;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+/** Test for ReRankQParserPlugin functionality */
+public class TestRerankOnSolrCloud extends SolrCloudTestCase {
+
+ private static final String COLLECTION_NAME = "rerank_test";
+
+ @BeforeClass
+ public static void setupCluster() throws Exception {
+System.setProperty("managed.schema.mutable", "true");
+configureCluster(2)
+.addConfig(
+"conf1",
TEST_PATH().resolve("configsets").resolve("cloud-managed").resolve("conf"))
+.configure();
+
+// Create collection
+CollectionAdminRequest.createCollection(COLLECTION_NAME, "conf1", 2, 1)
+.process(cluster.getSolrClient());
+
+cluster.waitForActiveCollection(COLLECTION_NAME, 2, 2);
+
+// Prepare test data
+CloudSolrClient client = cluster.getSolrClient();
+UpdateRequest updateRequest = new UpdateRequest();
+
+// Add test documents
+for (int i = 1; i <= 5; i++) {
+ SolrInputDocument doc = new SolrInputDocument();
+ doc.addField(CommonParams.ID, String.valueOf(i));
+ doc.addField("test_s", "hello");
+ doc.addField("test_i", i);
+ updateRequest.add(doc);
+}
+
+updateRequest.process(client, COLLECTION_NAME);
+client.commit(COLLECTION_NAME);
+ }
+
+ @Test
+ public void testLargeReRankDocsParamWithoutSort() throws Exception {
+// Build query parameters
+SolrParams params =
+params(
+CommonParams.Q, "*:*",
+CommonParams.FL, "id,test_s,test_i,score",
+CommonParams.RQ,
+"{!rerank reRankQuery='{!func} 100' reRankDocs=10
reRankWeight=2}");
+
+QueryRequest req = new QueryRequest(params);
+QueryResponse rsp = req.process(cluster.getSolrClient(), COLLECTION_NAME);
+
+// Test passes if execution completes without OOM
+Assert.assertTrue(true);
Review Comment:
I don't like this test. We shouldn't be trying to cause an OOM. There has to
be a better way to test this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
