[jira] [Updated] (SOLR-10317) Solr Nightly Benchmarks
[ https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-10317: Description: The benchmarking suite is now here: [https://github.com/thesearchstack/solr-bench] Actual datasets and queries are TBD yet. --- Original description --- Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found here, [https://home.apache.org/~mikemccand/lucenebench/]. Preferably, we need: # A suite of benchmarks that build Solr from a commit point, start Solr nodes, both in SolrCloud and standalone mode, and record timing information of various operations like indexing, querying, faceting, grouping, replication etc. # It should be possible to run them either as an independent suite or as a Jenkins job, and we should be able to report timings as graphs (Jenkins has some charting plugins). # The code should eventually be integrated in the Solr codebase, so that it never goes out of date. There is some prior work / discussion: # [https://github.com/shalinmangar/solr-perf-tools] (Shalin) # [https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md] (Ishan/Vivek) # SOLR-2646 & SOLR-9863 (Mark Miller) # [https://home.apache.org/~mikemccand/lucenebench/] (Mike McCandless) # [https://github.com/lucidworks/solr-scale-tk] (Tim Potter) There is support for building, starting, indexing/querying and stopping Solr in some of these frameworks above. However, the benchmarks run are very limited. Any of these can be a starting point, or a new framework can as well be used. The motivation is to be able to cover every functionality of Solr with a corresponding benchmark that is run every night. Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure [~shalinmangar] and [~[markrmil...@gmail.com|mailto:markrmil...@gmail.com]] would help here. was: The benchmarking suite is now here: https://github.com/thesearchstack/solr-bench Actual datasets and queries are TBD yet. > Solr Nightly Benchmarks > --- > > Key: SOLR-10317 > URL: https://issues.apache.org/jira/browse/SOLR-10317 > Project: Solr > Issue Type: Task >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: gsoc2017, mentor > Attachments: > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf, > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, SOLR-10317.patch, > SOLR-10317.patch, Screenshot from 2017-07-30 20-30-05.png, > changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema, > solrconfig.xml > > > The benchmarking suite is now here: > [https://github.com/thesearchstack/solr-bench] > Actual datasets and queries are TBD yet. > > --- Original description --- > Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be > found here, [https://home.apache.org/~mikemccand/lucenebench/]. > > Preferably, we need: > # A suite of benchmarks that build Solr from a commit point, start Solr > nodes, both in SolrCloud and standalone mode, and record timing information > of various operations like indexing, querying, faceting, grouping, > replication etc. > # It should be possible to run them either as an independent suite or as a > Jenkins job, and we should be able to report timings as graphs (Jenkins has > some charting plugins). > # The code should eventually be integrated in the Solr codebase, so that it > never goes out of date. > > There is some prior work / discussion: > # [https://github.com/shalinmangar/solr-perf-tools] (Shalin) > # [https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md] > (Ishan/Vivek) > # SOLR-2646 & SOLR-9863 (Mark Miller) > # [https://home.apache.org/~mikemccand/lucenebench/] (Mike McCandless) > # [https://github.com/lucidworks/solr-scale-tk] (Tim Potter) > > There is support for building, starting, indexing/querying and stopping Solr > in some of these frameworks above. However, the benchmarks run are very > limited. Any of these can be a starting point, or a new framework can as well > be used. The motivation is to be able to cover every functionality of Solr > with a corresponding benchmark that is run every night. > > Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure > [~shalinmangar] and [~[markrmil...@gmail.com|mailto:markrmil...@gmail.com]] > would help here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-10317) Solr Nightly Benchmarks
[ https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151078#comment-17151078 ] Ishan Chattopadhyaya edited comment on SOLR-10317 at 7/4/20, 5:55 AM: -- This is now done and in a good shape. This is currently hosted at https://github.com/thesearchstack/solr-bench (forked from https://github.com/fullstorydev/solr-bench). This now has the original local mode as discussed here. It has an additional GCP mode, where GCP nodes will be spun up and the tests will run on them. Plenty of scope for improvement. Moreover, we need to craft datasets and queries that we can run continuously. -I shall address that in a separate issue.- Thanks to everyone who contributed. Thanks to [~vivek.nar...@uga.edu] for his work on this, to Google Summer of Code program for helping us start the project and to FullStory for supporting part of the development. was (Author: ichattopadhyaya): This is now done and in a good shape. This is currently hosted at https://github.com/thesearchstack/solr-bench (forked from https://github.com/fullstorydev/solr-bench). This now has the original local mode as discussed here. It has an additional GCP mode, where GCP nodes will be spun up and the tests will run on them. Plenty of scope for improvement. Moreover, we need to craft datasets and queries that we can run continuously. I shall address that in a separate issue. Thanks to everyone who contributed. Thanks to [~vivek.nar...@uga.edu] for his work on this, to Google Summer of Code program for helping us start the project and to FullStory for supporting part of the development. > Solr Nightly Benchmarks > --- > > Key: SOLR-10317 > URL: https://issues.apache.org/jira/browse/SOLR-10317 > Project: Solr > Issue Type: Task >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: gsoc2017, mentor > Attachments: > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf, > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, SOLR-10317.patch, > SOLR-10317.patch, Screenshot from 2017-07-30 20-30-05.png, > changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema, > solrconfig.xml > > > The benchmarking suite is now here: > https://github.com/thesearchstack/solr-bench > Actual datasets and queries are TBD yet. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Reopened] (SOLR-10317) Solr Nightly Benchmarks
[ https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya reopened SOLR-10317: - Sure, David. I'm re-opening, so we can discuss datasets and queries here itself. Initially I thought it would be better to open a new issue and discuss that there (and reserve this issue solely for the development effort that has gone in). I shall set this up soon with Wikipedia dataset (that powers lucene benchmarks) and host it somewhere for running repeatably. However, it would be immensely beneficial to have some e-commerce centric dataset and usecases, since that will cover more of Solr functionality. Do you have something in mind that we can use? > Solr Nightly Benchmarks > --- > > Key: SOLR-10317 > URL: https://issues.apache.org/jira/browse/SOLR-10317 > Project: Solr > Issue Type: Task >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: gsoc2017, mentor > Attachments: > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf, > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, SOLR-10317.patch, > SOLR-10317.patch, Screenshot from 2017-07-30 20-30-05.png, > changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema, > solrconfig.xml > > > The benchmarking suite is now here: > https://github.com/thesearchstack/solr-bench > Actual datasets and queries are TBD yet. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-10317) Solr Nightly Benchmarks
[ https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya updated SOLR-10317: Description: The benchmarking suite is now here: https://github.com/thesearchstack/solr-bench Actual datasets and queries are TBD yet. was: Currently hosted at: http://212.47.242.214/MergedViewCloud.html Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be found here, https://home.apache.org/~mikemccand/lucenebench/. Preferably, we need: # A suite of benchmarks that build Solr from a commit point, start Solr nodes, both in SolrCloud and standalone mode, and record timing information of various operations like indexing, querying, faceting, grouping, replication etc. # It should be possible to run them either as an independent suite or as a Jenkins job, and we should be able to report timings as graphs (Jenkins has some charting plugins). # The code should eventually be integrated in the Solr codebase, so that it never goes out of date. There is some prior work / discussion: # https://github.com/shalinmangar/solr-perf-tools (Shalin) # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md (Ishan/Vivek) # SOLR-2646 & SOLR-9863 (Mark Miller) # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless) # https://github.com/lucidworks/solr-scale-tk (Tim Potter) There is support for building, starting, indexing/querying and stopping Solr in some of these frameworks above. However, the benchmarks run are very limited. Any of these can be a starting point, or a new framework can as well be used. The motivation is to be able to cover every functionality of Solr with a corresponding benchmark that is run every night. Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure [~shalinmangar] and [~markrmil...@gmail.com] would help here. > Solr Nightly Benchmarks > --- > > Key: SOLR-10317 > URL: https://issues.apache.org/jira/browse/SOLR-10317 > Project: Solr > Issue Type: Task >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: gsoc2017, mentor > Attachments: > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf, > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, SOLR-10317.patch, > SOLR-10317.patch, Screenshot from 2017-07-30 20-30-05.png, > changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema, > solrconfig.xml > > > The benchmarking suite is now here: > https://github.com/thesearchstack/solr-bench > Actual datasets and queries are TBD yet. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-10317) Solr Nightly Benchmarks
[ https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151183#comment-17151183 ] David Smiley commented on SOLR-10317: - Closing this seems premature if there are no benchmarks being run automatically for us to simply view at a URL. Am I missing something? Perhaps the issue description should be updated to have the end-product(s) / URLs, and then a horizontal line for the current description. The current description starts with a link to something that doesn't exist. > Solr Nightly Benchmarks > --- > > Key: SOLR-10317 > URL: https://issues.apache.org/jira/browse/SOLR-10317 > Project: Solr > Issue Type: Task >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: gsoc2017, mentor > Attachments: > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf, > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, SOLR-10317.patch, > SOLR-10317.patch, Screenshot from 2017-07-30 20-30-05.png, > changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema, > solrconfig.xml > > > Currently hosted at: http://212.47.242.214/MergedViewCloud.html > > Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be > found here, https://home.apache.org/~mikemccand/lucenebench/. > Preferably, we need: > # A suite of benchmarks that build Solr from a commit point, start Solr > nodes, both in SolrCloud and standalone mode, and record timing information > of various operations like indexing, querying, faceting, grouping, > replication etc. > # It should be possible to run them either as an independent suite or as a > Jenkins job, and we should be able to report timings as graphs (Jenkins has > some charting plugins). > # The code should eventually be integrated in the Solr codebase, so that it > never goes out of date. > There is some prior work / discussion: > # https://github.com/shalinmangar/solr-perf-tools (Shalin) > # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md > (Ishan/Vivek) > # SOLR-2646 & SOLR-9863 (Mark Miller) > # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless) > # https://github.com/lucidworks/solr-scale-tk (Tim Potter) > There is support for building, starting, indexing/querying and stopping Solr > in some of these frameworks above. However, the benchmarks run are very > limited. Any of these can be a starting point, or a new framework can as well > be used. The motivation is to be able to cover every functionality of Solr > with a corresponding benchmark that is run every night. > Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure > [~shalinmangar] and [~markrmil...@gmail.com] would help here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14021) Remove HDFS support from Solr
[ https://issues.apache.org/jira/browse/SOLR-14021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151180#comment-17151180 ] Ishan Chattopadhyaya commented on SOLR-14021: - +1, Joel. Thanks a lot for raising the issue. I propose we deprecate all HDFS support in 8x (8.6 or 8.7) and remove from 9.0. On a best effort basis, we can see if it can be factored out into a package. > Remove HDFS support from Solr > - > > Key: SOLR-14021 > URL: https://issues.apache.org/jira/browse/SOLR-14021 > Project: Solr > Issue Type: Improvement > Components: hdfs >Reporter: Joel Bernstein >Priority: Major > > This ticket will remove HDFS support from the Solr project. > There appears to be growing consensus among committers that it's time to > start removing features so committers can have a manageable system to > maintain. HDFS has come up a number of times as needing to be removed. The > HDFS tests have not been maintained over the years and fail frequently. We > need to start removing features that no one cares about enough to even > maintain the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14621) Deprecate and eventually remove HDFS support
[ https://issues.apache.org/jira/browse/SOLR-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya resolved SOLR-14621. - Resolution: Duplicate Oh goodness, how did I miss that! Thanks David. Lets move over there for discussion. > Deprecate and eventually remove HDFS support > > > Key: SOLR-14621 > URL: https://issues.apache.org/jira/browse/SOLR-14621 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Ishan Chattopadhyaya >Priority: Major > > This issue is to deprecate HDFS support from Solr. HDFS clutter should not be > part of Solr core and should either be removed completely or be packaged up > as an optional module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14621) Deprecate and eventually remove HDFS support
[ https://issues.apache.org/jira/browse/SOLR-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151168#comment-17151168 ] David Smiley commented on SOLR-14621: - +1 to package-ify it and thus have it not be in solr-core. This issue duplicates SOLR-14021 by [~jbernste] > Deprecate and eventually remove HDFS support > > > Key: SOLR-14621 > URL: https://issues.apache.org/jira/browse/SOLR-14621 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Ishan Chattopadhyaya >Priority: Major > > This issue is to deprecate HDFS support from Solr. HDFS clutter should not be > part of Solr core and should either be removed completely or be packaged up > as an optional module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] shalinmangar commented on pull request #1647: SOLR-14626: General improvements to coreContainer shutdown
shalinmangar commented on pull request #1647: URL: https://github.com/apache/lucene-solr/pull/1647#issuecomment-653714746 Please be specific about the changes. Why do you think preClose is unnecessary? I see that invocation of `removeEphemeralLiveNode` method is removed entirely. Why do you think that is a safe change to make? With this PR, the live node is not removed until the zk client is closed which happens after pretty much everything is closed. This will cause other clients/replicas to keep sending requests to this node and erroring out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul closed pull request #1649: SOLR-14610: ReflectMapWriter to use VarHandle instead of reflection for branch_8x
noblepaul closed pull request #1649: URL: https://github.com/apache/lucene-solr/pull/1649 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] shalinmangar commented on pull request #1646: SOLR-14625: Leader election improvements
shalinmangar commented on pull request #1646: URL: https://github.com/apache/lucene-solr/pull/1646#issuecomment-653713178 This is doing much more than removing a 2500ms wait. It seems you have completely removed `leaderVoteWait`? Can you please accurately describe what you intend to do here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul opened a new pull request #1649: SOLR-14610: ReflectMapWriter to use VarHandle instead of reflection for branch_8x
noblepaul opened a new pull request #1649: URL: https://github.com/apache/lucene-solr/pull/1649 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14022) Deprecate CDCR from Solr in 8.x
[ https://issues.apache.org/jira/browse/SOLR-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151131#comment-17151131 ] Erick Erickson commented on SOLR-14022: --- [~sarkaramr...@gmail.com] What Ishan said. For that matter, I'm the one that originally put it in, admittedly others did the heavy lifting. But this is how my career has gone. Work/fix/work/fix. At some point you have to stop beating your head against a wall and move on to a better solution ;). I once saw M. Night Shyamalan talking about movie school. He was told "you always have to be ready to cut the scene you love most". Analogously, you always have to be ready to let go of code ;) I'm certain that CDCR could be made better. I'm also certain that we'll _never_ be able to put as much effort into it as apps out there that are built for this problem rather than bolted on the side of Solr. > Deprecate CDCR from Solr in 8.x > --- > > Key: SOLR-14022 > URL: https://issues.apache.org/jira/browse/SOLR-14022 > Project: Solr > Issue Type: Improvement > Components: CDCR >Reporter: Joel Bernstein >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > This ticket will deprecate CDCR in Solr 8x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] tflobbe commented on a change in pull request #1644: SOLR-14623: Minor Improvements to SolrCloud tests
tflobbe commented on a change in pull request #1644:
URL: https://github.com/apache/lucene-solr/pull/1644#discussion_r449712037
##
File path: solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java
##
@@ -207,18 +208,38 @@ private void addDocToWhenOtherReplicasAreDown(String
collection, Replica leader,
* Leader should be on node - 0
*/
private void addDocWhenOtherReplicasAreNetworkPartitioned(String collection,
Replica leader, int docId) throws Exception {
-for (int i = 0; i < 3; i++) {
- proxies.get(cluster.getJettySolrRunner(i)).close();
+DocCollection col =
cluster.getSolrClient().getZkStateReader().getClusterState().getCollection(collection);
+Replica shard1Leader = col.getLeader("shard1");
+String baseUrl = shard1Leader.getBaseUrl();
+JettySolrRunner j1 = null; // This will be the jetty hosting shard1's
leader
+for (JettySolrRunner j : cluster.getJettySolrRunners()) {
+ System.out.println("cmp:" + j.getProxyBaseUrl() + " " + baseUrl);
+ if (j.getProxyBaseUrl().toString().equals(baseUrl)) {
+j1 = j;
+break;
+ }
+}
+
+assertNotNull(baseUrl, j1);
Review comment:
Maybe you can improve this message? A failure would look confusing
##
File path: solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java
##
@@ -207,18 +208,38 @@ private void addDocToWhenOtherReplicasAreDown(String
collection, Replica leader,
* Leader should be on node - 0
*/
private void addDocWhenOtherReplicasAreNetworkPartitioned(String collection,
Replica leader, int docId) throws Exception {
-for (int i = 0; i < 3; i++) {
- proxies.get(cluster.getJettySolrRunner(i)).close();
+DocCollection col =
cluster.getSolrClient().getZkStateReader().getClusterState().getCollection(collection);
+Replica shard1Leader = col.getLeader("shard1");
+String baseUrl = shard1Leader.getBaseUrl();
+JettySolrRunner j1 = null; // This will be the jetty hosting shard1's
leader
+for (JettySolrRunner j : cluster.getJettySolrRunners()) {
+ System.out.println("cmp:" + j.getProxyBaseUrl() + " " + baseUrl);
+ if (j.getProxyBaseUrl().toString().equals(baseUrl)) {
+j1 = j;
+break;
+ }
+}
+
+assertNotNull(baseUrl, j1);
+
+for (JettySolrRunner j : cluster.getJettySolrRunners()) {
+ if (j != j1) {
+proxies.get(j).close();
+ }
}
-addDoc(collection, docId, cluster.getJettySolrRunner(0));
-JettySolrRunner j1 = cluster.getJettySolrRunner(0);
+
+addDoc(collection, docId, j1);
+
j1.stop();
cluster.waitForJettyToStop(j1);
-for (int i = 1; i < 3; i++) {
- proxies.get(cluster.getJettySolrRunner(i)).reopen();
+for (JettySolrRunner j : cluster.getJettySolrRunners()) {
+ if (j != j1) {
+proxies.get(j).reopen();
+ }
}
waitForState("Timeout waiting for leader goes DOWN", collection,
(liveNodes, collectionState)
--> collectionState.getReplica(leader.getName()).getState() ==
Replica.State.DOWN);
+-> collectionState.getReplica(leader.getName()).getState() ==
Replica.State.DOWN);
+Thread.sleep(1000);
Review comment:
Is this sleep here to wait for the stop? Why is it now needed?
##
File path: solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java
##
@@ -207,18 +208,38 @@ private void addDocToWhenOtherReplicasAreDown(String
collection, Replica leader,
* Leader should be on node - 0
*/
private void addDocWhenOtherReplicasAreNetworkPartitioned(String collection,
Replica leader, int docId) throws Exception {
-for (int i = 0; i < 3; i++) {
- proxies.get(cluster.getJettySolrRunner(i)).close();
+DocCollection col =
cluster.getSolrClient().getZkStateReader().getClusterState().getCollection(collection);
+Replica shard1Leader = col.getLeader("shard1");
+String baseUrl = shard1Leader.getBaseUrl();
+JettySolrRunner j1 = null; // This will be the jetty hosting shard1's
leader
+for (JettySolrRunner j : cluster.getJettySolrRunners()) {
+ System.out.println("cmp:" + j.getProxyBaseUrl() + " " + baseUrl);
Review comment:
Is this intentionally left here?
##
File path: solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java
##
@@ -207,18 +208,38 @@ private void addDocToWhenOtherReplicasAreDown(String
collection, Replica leader,
* Leader should be on node - 0
Review comment:
I guess this is no longer true?
##
File path: solr/core/src/test/org/apache/solr/cloud/TestCloudConsistency.java
##
@@ -207,18 +208,38 @@ private void addDocToWhenOtherReplicasAreDown(String
collection, Replica leader,
* Leader should be on node - 0
*/
private void addDocWhenOtherReplicasAreNetworkPartitioned(String collection,
Replica leader, int docId) throws Exception {
-for (int i = 0; i < 3; i++) {
- proxies.get(cluster.getJettySo
[jira] [Commented] (SOLR-14022) Deprecate CDCR from Solr in 8.x
[ https://issues.apache.org/jira/browse/SOLR-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151129#comment-17151129 ] Ishan Chattopadhyaya commented on SOLR-14022: - [~sarkaramr...@gmail.com], you did a great job trying to revive something that was broken by design. I look forward to your help in building ourselves the next scalable CDCR solution that users will actually use and love. > Deprecate CDCR from Solr in 8.x > --- > > Key: SOLR-14022 > URL: https://issues.apache.org/jira/browse/SOLR-14022 > Project: Solr > Issue Type: Improvement > Components: CDCR >Reporter: Joel Bernstein >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > This ticket will deprecate CDCR in Solr 8x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14022) Deprecate CDCR from Solr in 8.x
[ https://issues.apache.org/jira/browse/SOLR-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151128#comment-17151128 ] Amrit Sarkar commented on SOLR-14022: - FWIW I apologize for not able to fix issues and improve the CDCR module in Apache Solr. I vouch to contribute and assist respected committers to build and implement the best possible design with the capabilities I possess. > Deprecate CDCR from Solr in 8.x > --- > > Key: SOLR-14022 > URL: https://issues.apache.org/jira/browse/SOLR-14022 > Project: Solr > Issue Type: Improvement > Components: CDCR >Reporter: Joel Bernstein >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.6 > > Time Spent: 10m > Remaining Estimate: 0h > > This ticket will deprecate CDCR in Solr 8x. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14523) Enhance gradle logging calls validation: eliminate getMessage()
[
https://issues.apache.org/jira/browse/SOLR-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151123#comment-17151123
]
Erick Erickson commented on SOLR-14523:
---
Oops, yeah I'll take care of it. Thanks for pointing that out.
> Enhance gradle logging calls validation: eliminate getMessage()
> ---
>
> Key: SOLR-14523
> URL: https://issues.apache.org/jira/browse/SOLR-14523
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andras Salamon
>Assignee: Erick Erickson
>Priority: Minor
> Fix For: 8.7
>
> Attachments: SOLR-14523.patch, validation.patch
>
>
> SOLR-14280 fixed a logging problem in SolrConfig by removing a few
> getMessage() calls. We could enhance this solution by modifying gradle's
> logging calls validation and forbid getMessage() calls during logging. We
> should check the existing code and eliminate such calls.
> It is possible to suppress the warning using {{//logok}}.
> [~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14620) Preference checks in Autoscaling framework can go wrong
Ilan Ginzburg created SOLR-14620:
Summary: Preference checks in Autoscaling framework can go wrong
Key: SOLR-14620
URL: https://issues.apache.org/jira/browse/SOLR-14620
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: AutoScaling, SolrCloud
Reporter: Ilan Ginzburg
In the Autoscaling framework, preferences are values to maximize or minimize
when doing replica placement decisions and are used to guide the framework
towards the optimal placement decision. The default cluster preferences are
minimizing the number of cores on each node then maximizing the free disk space
on each node. In order to evaluate a {{Preference}}, the corresponding
parameter value for a node (for example number of cores or free disk space)
must be known.
Such parameters for a node (and their values) are stored in a {{Cell}} array in
the {{Row}} object. Parameter names come from multiple sources (not 100% sure
about the list below, hard to really trace legit values):
* {{From Preference}} definitions: {{freedisk}}, {{cores}}, {{heapUsage}} and
{{sysLoadAvg}}
* {{From Policy}} definitions: {{withCollection}}, {{port}}, {{ip_1}},
{{ip_2}}, {{ip_3}}, {{ip_4}}, {{freedisk}}, {{nodeRole}}, {{cores}},
{{sysLoadAvg}}, {{heapUsage}}, {{host}}, {{node}}, {{nodeset}}, {{metrics:*}}
({{*}} meaning any string)
* Possibly from other places...
{{"cores"}} and {{"freedisk"}} are always added to the {{Cell}} array in
{{Row}} (see {{Policy.DEFAULT_PARAMS_OF_INTEREST}}). The {{Cell}} array is
*sorted* by natural ordering of the parameter names. This causes {{"cores"}} to
be first and {{"freedisk"}} second, as you'll notice all other parameter names
listed above are lexicographically greater than these two.
When comparing rows in {{Preference.compare()}} (used for sorting them), the
value of the {{Preference}} is obtained from the {{Cell}} with array index
equal to the {{Preference}} index (starts at 0, in the order declared).
This obviously only makes sense if the {{Cell}} array order is identical to
{{Preference}} list order. {{Preferences}} therefore would have to be provided
by increasing parameter name and no parameters should exist in the {{Cell}}
array that are lexicographically smaller than the "highest" {{Preference}}
without having a matching {{Preference}}.
This basically means that when preferences are the default minimize number of
cores first then maximize freedisk second, the check works. But if for example
cluster preferences are explicitly defined to maximize freedisk first then
minimize number of cores, the check is broken. This will be more apparent when
parameters to maximize are swapped with parameters to minimize (which would be
the case here).
Unclear to me what's the real impact of this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14616) Remove CDCR from 9.0
[ https://issues.apache.org/jira/browse/SOLR-14616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151112#comment-17151112 ] Ishan Chattopadhyaya commented on SOLR-14616: - Can someone please review? [~erickerickson], would you please take a look? > Remove CDCR from 9.0 > > > Key: SOLR-14616 > URL: https://issues.apache.org/jira/browse/SOLR-14616 > Project: Solr > Issue Type: Sub-task >Affects Versions: master (9.0) >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Blocker > Time Spent: 10m > Remaining Estimate: 0h > > This was deprecated in SOLR-14022 and should be removed in 9.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman opened a new pull request #1648: SOLR-14616: Remove CDCR from Solr 9.x
chatman opened a new pull request #1648: URL: https://github.com/apache/lucene-solr/pull/1648 We already deprecated CDCR in 8.6. We need to remove it from master/9.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14624) Improvements to waitFor* handling
Ishan Chattopadhyaya created SOLR-14624: --- Summary: Improvements to waitFor* handling Key: SOLR-14624 URL: https://issues.apache.org/jira/browse/SOLR-14624 Project: Solr Issue Type: Sub-task Security Level: Public (Default Security Level. Issues are Public) Reporter: Ishan Chattopadhyaya Remove polling from OCMH#waitForCoreNodeName(), OCMH#waitForNewShard(), OCMH#waitToSeeReplicasInState(), CollectionsHandler#waitForActiveCollection() to only check replica counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14537) Improve performance of ExportWriter
[
https://issues.apache.org/jira/browse/SOLR-14537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki resolved SOLR-14537.
-
Resolution: Fixed
> Improve performance of ExportWriter
> ---
>
> Key: SOLR-14537
> URL: https://issues.apache.org/jira/browse/SOLR-14537
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Export Writer
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.7
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Retrieving, sorting and writing out documents in {{ExportWriter}} are three
> aspects of the /export handler that can be further optimized.
> SOLR-14470 introduced some level of caching in {{StringValue}}. Further
> options for caching and speedups should be explored.
> Currently the sort/retrieve and write operations are done sequentially, but
> they could be parallelized, considering that they block on different channels
> - the first is index reading & CPU bound, the other is bound by the receiving
> end because it uses blocking IO. The sorting and retrieving of values could
> be done in parallel with the operation of writing out the current batch of
> results.
> One possible approach here would be to use "double buffering" where one
> buffered batch that is ready (already sorted and retrieved) is being written
> out, while the other batch is being prepared in a background thread, and when
> both are done the buffers are swapped. This wouldn't complicate the current
> code too much but it should instantly give up to 2x higher throughput.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14621) Deprecate and eventually remove HDFS support
[ https://issues.apache.org/jira/browse/SOLR-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151058#comment-17151058 ] Atri Sharma commented on SOLR-14621: FWIW, I am +1 to this. I suggest that we move this to a a plugin and remove it in a major version > Deprecate and eventually remove HDFS support > > > Key: SOLR-14621 > URL: https://issues.apache.org/jira/browse/SOLR-14621 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Ishan Chattopadhyaya >Priority: Major > > This issue is to deprecate HDFS support from Solr. HDFS clutter should not be > part of Solr core and should either be removed completely or be packaged up > as an optional module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14537) Improve performance of ExportWriter
[
https://issues.apache.org/jira/browse/SOLR-14537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150955#comment-17150955
]
ASF subversion and git services commented on SOLR-14537:
Commit dc2de7418cbb2f700103d8149e6d1416e24ff109 in lucene-solr's branch
refs/heads/branch_8x from Andrzej Bialecki
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=dc2de74 ]
SOLR-14537: Improve performance of ExportWriter.
> Improve performance of ExportWriter
> ---
>
> Key: SOLR-14537
> URL: https://issues.apache.org/jira/browse/SOLR-14537
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Export Writer
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Retrieving, sorting and writing out documents in {{ExportWriter}} are three
> aspects of the /export handler that can be further optimized.
> SOLR-14470 introduced some level of caching in {{StringValue}}. Further
> options for caching and speedups should be explored.
> Currently the sort/retrieve and write operations are done sequentially, but
> they could be parallelized, considering that they block on different channels
> - the first is index reading & CPU bound, the other is bound by the receiving
> end because it uses blocking IO. The sorting and retrieving of values could
> be done in parallel with the operation of writing out the current batch of
> results.
> One possible approach here would be to use "double buffering" where one
> buffered batch that is ready (already sorted and retrieved) is being written
> out, while the other batch is being prepared in a background thread, and when
> both are done the buffers are swapped. This wouldn't complicate the current
> code too much but it should instantly give up to 2x higher throughput.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14537) Improve performance of ExportWriter
[
https://issues.apache.org/jira/browse/SOLR-14537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated SOLR-14537:
Fix Version/s: 8.7
> Improve performance of ExportWriter
> ---
>
> Key: SOLR-14537
> URL: https://issues.apache.org/jira/browse/SOLR-14537
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Export Writer
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.7
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Retrieving, sorting and writing out documents in {{ExportWriter}} are three
> aspects of the /export handler that can be further optimized.
> SOLR-14470 introduced some level of caching in {{StringValue}}. Further
> options for caching and speedups should be explored.
> Currently the sort/retrieve and write operations are done sequentially, but
> they could be parallelized, considering that they block on different channels
> - the first is index reading & CPU bound, the other is bound by the receiving
> end because it uses blocking IO. The sorting and retrieving of values could
> be done in parallel with the operation of writing out the current batch of
> results.
> One possible approach here would be to use "double buffering" where one
> buffered batch that is ready (already sorted and retrieved) is being written
> out, while the other batch is being prepared in a background thread, and when
> both are done the buffers are swapped. This wouldn't complicate the current
> code too much but it should instantly give up to 2x higher throughput.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14608) Faster sorting for the /export handler
[ https://issues.apache.org/jira/browse/SOLR-14608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Bernstein updated SOLR-14608: -- Description: The largest cost of the export handler is the sorting. This ticket will implement an improved algorithm for sorting that should greatly increase overall throughput for the export handler. *The current algorithm is as follows:* Collect a bitset of matching docs. Iterate over that bitset and materialize the top level oridinals for the sort fields in the document and add them to priority queue of size 3. Then export the top 3 docs, turn off the bits in the bit set and iterate again until all docs are sorted and sent. There are two performance bottlenecks with this approach: 1) Materializing the top level ordinals adds a huge amount of overhead to the sorting process. 2) The size of priority queue, 30,000, adds significant overhead to sorting operations. *The new algorithm:* Has a top level *merge sort iterator* that wraps segment level iterators that perform segment level priority queue sorts. *Segment level:* The segment level docset will be iterated and the segment level ordinals for the sort fields will be materialized and added to a segment level priority queue. As the segment level iterator pops docs from the priority queue the top level ordinals for the sort fields are materialized. Because the top level ordinals are materialized AFTER the sort, they only need to be looked up when the segment level ordinal changes. This takes advantage of the sort to limit the lookups into the top level ordinal structures. This also eliminates redundant lookups of top level ordinals that occur during the multiple passes over the matching docset. The segment level priority queues can be kept smaller than 30,000 to improve performance of the sorting operations because the overall batch size will still be 30,000 or greater when all the segment priority queue sizes are added up. This allows for batch sizes much larger then 30,000 without using a single large priority queue. The increased batch size means fewer iterations over the matching docset and the decreased priority queue size means faster sorting operations. *Top level:* A top level iterator does a merge sort over the segment level iterators by comparing the top level ordinals materialized when the segment level docs are popped from the segment level priority queues. This requires no extra memory and will be very performant. was: The largest cost of the export handler is the sorting. This ticket will implement an improved algorithm for sorting that should greatly increase overall throughput for the export handler. *The current algorithm is as follows:* Collect a bitset of matching docs. Iterate over that bitset and materialize the top level oridinals for the sort fields in the document and add them to priority queue of size 3. Then export the top 3 docs, turn off the bits in the bit set and iterate again until all docs are sorted and sent. There are two performance bottlenecks with this approach: 1) Materializing the top level ordinals adds a huge amount of overhead to the sorting process. 2) The size of priority queue, 30,000, adds significant overhead to sorting operations. *The new algorithm:* Has a top level *merge sort iterator* that wraps segment level iterators that perform segment level priority queue sorts. *Segment level:* The segment level docset will be iterated and the segment level ordinals for the sort fields will be materialized and added to a segment level priority queue. As the segment level iterator pops docs from the priority queue the top level ordinals for the sort fields are materialized. Because the top level ordinals are materialized AFTER the sort, they only need to be looked up when the segment level ordinal changes. This takes advantage of the sort to limit the lookups into the top level ordinal structures. This also eliminates redundant lookups of top level ordinals that occur during the multiple passes over the matching docset. The segment level priority queues can be kept smaller than 30,000 to improve performance of the sorting operations because the overall batch size will still be 30,000 or greater when all the segment priority queue sizes are added up. This allows for batch sizes much larger then 30,000 without using a single large priority queue. The increased batchsize means fewer iterations over the matching docset and the decreased priority queue size means faster sorting operations. *Top level:* A top level iterator does a merge sort over the segment level iterators by comparing the top level ordinals materialized when the segment level docs are popped from the segment level priority queues. This requires no extra memory and will be very performant. > Faster sorting for the /export handler > --
[jira] [Updated] (SOLR-14608) Faster sorting for the /export handler
[ https://issues.apache.org/jira/browse/SOLR-14608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Bernstein updated SOLR-14608: -- Description: The largest cost of the export handler is the sorting. This ticket will implement an improved algorithm for sorting that should greatly increase overall throughput for the export handler. *The current algorithm is as follows:* Collect a bitset of matching docs. Iterate over that bitset and materialize the top level oridinals for the sort fields in the document and add them to priority queue of size 3. Then export the top 3 docs, turn off the bits in the bit set and iterate again until all docs are sorted and sent. There are two performance bottlenecks with this approach: 1) Materializing the top level ordinals adds a huge amount of overhead to the sorting process. 2) The size of priority queue, 30,000, adds significant overhead to sorting operations. *The new algorithm:* Has a top level *merge sort iterator* that wraps segment level iterators that perform segment level priority queue sorts. *Segment level:* The segment level docset will be iterated and the segment level ordinals for the sort fields will be materialized and added to a segment level priority queue. As the segment level iterator pops docs from the queue the top level ordinals for the sort fields are materialized. Because the top level ordinals are materialized AFTER the sort, they only need to be looked up when the segment level ordinal changes. This takes advantage of the sort to limit the lookups into the top level ordinal structures. This also eliminates redundant lookups of top level ordinals that occur during the multiple passes over the matching docset. The segment level priority queues can be kept smaller than 30,000 to improve performance of the sorting operations because the overall batch size will still be 30,000 or greater when all the segment priority queue sizes are added up. This allows for batch sizes much larger then 30,000 without using a single large priority queue. The increased batchsize means fewer iterations over the matching docset and the decreased priority queue size means faster sorting operations. *Top level:* A top level iterator does a merge sort over the segment level iterators by comparing the top level ordinals materialized when the segment level docs are popped from the segment level priority queues. This requires no extra memory and will be very performant. was: The largest cost of the export handler is the sorting. This ticket will implement an improved algorithm for sorting that should greatly increase overall throughput for the export handler. *The current algorithm is as follows:* Collect a bitset of matching docs. Iterate over that bitset and materialize the top level oridinals for the sort fields in the document and add them to priority queue of size 3. Then export the top 3 docs, turn off the bits in the bit set and iterate again until all docs are sorted and sent. There are two performance bottlenecks with this approach: 1) Materializing the top level ordinals adds a huge amount of overhead to the sorting process. 2) The size of priority queue, 30,000, adds significant overhead to sorting operations. *The new algorithm:* Has a top level *merge sort iterator* that wraps segment level iterators that perform segment level priority queue sorts. *Segment level:* The segment level docset will be iterated and the segment level ordinals for the sort fields will be materialized and added to a segment level priority queue. As the segment level iterator pops docs from the queue the top level ordinals for the sort fields are materialized. Because the top level ordinals are materialized AFTER the sort, they only need to be looked up when the segment level ordinal changes. This takes advantage of the sort to limit the lookups into the top level ordinal structures. This also eliminates redundant lookups of top level ordinals that occur during the multiple passes over the matching docset. The segment level priority queues can be kept smaller than 30,000 to improve performance of the sorting operations because the overall batchsize will still be 30,000 or greater when all the segment priority queue sizes are added up. This allows for batchsize much larger then 30,000 without using a single large priority queue. The increased batchsize means fewer iterations over the matching docset and the decreased priority queue size means faster sorting operations. *Top level:* A top level iterator does a merge sort over the segment level iterators by comparing the top level ordinals materialized when the segment level docs are popped from the segment level priority queues. This requires no extra memory and will be very performant. > Faster sorting for the /export handler > -- > >
[jira] [Commented] (SOLR-14523) Enhance gradle logging calls validation: eliminate getMessage()
[
https://issues.apache.org/jira/browse/SOLR-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151055#comment-17151055
]
Andras Salamon commented on SOLR-14523:
---
[~erickerickson] My patch did not contain your original {{validation.patch}}. I
mistakenly thought that you want to add it as a separate commit. Can you please
commit that as well.
> Enhance gradle logging calls validation: eliminate getMessage()
> ---
>
> Key: SOLR-14523
> URL: https://issues.apache.org/jira/browse/SOLR-14523
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andras Salamon
>Assignee: Erick Erickson
>Priority: Minor
> Fix For: 8.7
>
> Attachments: SOLR-14523.patch, validation.patch
>
>
> SOLR-14280 fixed a logging problem in SolrConfig by removing a few
> getMessage() calls. We could enhance this solution by modifying gradle's
> logging calls validation and forbid getMessage() calls during logging. We
> should check the existing code and eliminate such calls.
> It is possible to suppress the warning using {{//logok}}.
> [~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14608) Faster sorting for the /export handler
[ https://issues.apache.org/jira/browse/SOLR-14608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Bernstein updated SOLR-14608: -- Description: The largest cost of the export handler is the sorting. This ticket will implement an improved algorithm for sorting that should greatly increase overall throughput for the export handler. *The current algorithm is as follows:* Collect a bitset of matching docs. Iterate over that bitset and materialize the top level oridinals for the sort fields in the document and add them to priority queue of size 3. Then export the top 3 docs, turn off the bits in the bit set and iterate again until all docs are sorted and sent. There are two performance bottlenecks with this approach: 1) Materializing the top level ordinals adds a huge amount of overhead to the sorting process. 2) The size of priority queue, 30,000, adds significant overhead to sorting operations. *The new algorithm:* Has a top level *merge sort iterator* that wraps segment level iterators that perform segment level priority queue sorts. *Segment level:* The segment level docset will be iterated and the segment level ordinals for the sort fields will be materialized and added to a segment level priority queue. As the segment level iterator pops docs from the priority queue the top level ordinals for the sort fields are materialized. Because the top level ordinals are materialized AFTER the sort, they only need to be looked up when the segment level ordinal changes. This takes advantage of the sort to limit the lookups into the top level ordinal structures. This also eliminates redundant lookups of top level ordinals that occur during the multiple passes over the matching docset. The segment level priority queues can be kept smaller than 30,000 to improve performance of the sorting operations because the overall batch size will still be 30,000 or greater when all the segment priority queue sizes are added up. This allows for batch sizes much larger then 30,000 without using a single large priority queue. The increased batchsize means fewer iterations over the matching docset and the decreased priority queue size means faster sorting operations. *Top level:* A top level iterator does a merge sort over the segment level iterators by comparing the top level ordinals materialized when the segment level docs are popped from the segment level priority queues. This requires no extra memory and will be very performant. was: The largest cost of the export handler is the sorting. This ticket will implement an improved algorithm for sorting that should greatly increase overall throughput for the export handler. *The current algorithm is as follows:* Collect a bitset of matching docs. Iterate over that bitset and materialize the top level oridinals for the sort fields in the document and add them to priority queue of size 3. Then export the top 3 docs, turn off the bits in the bit set and iterate again until all docs are sorted and sent. There are two performance bottlenecks with this approach: 1) Materializing the top level ordinals adds a huge amount of overhead to the sorting process. 2) The size of priority queue, 30,000, adds significant overhead to sorting operations. *The new algorithm:* Has a top level *merge sort iterator* that wraps segment level iterators that perform segment level priority queue sorts. *Segment level:* The segment level docset will be iterated and the segment level ordinals for the sort fields will be materialized and added to a segment level priority queue. As the segment level iterator pops docs from the queue the top level ordinals for the sort fields are materialized. Because the top level ordinals are materialized AFTER the sort, they only need to be looked up when the segment level ordinal changes. This takes advantage of the sort to limit the lookups into the top level ordinal structures. This also eliminates redundant lookups of top level ordinals that occur during the multiple passes over the matching docset. The segment level priority queues can be kept smaller than 30,000 to improve performance of the sorting operations because the overall batch size will still be 30,000 or greater when all the segment priority queue sizes are added up. This allows for batch sizes much larger then 30,000 without using a single large priority queue. The increased batchsize means fewer iterations over the matching docset and the decreased priority queue size means faster sorting operations. *Top level:* A top level iterator does a merge sort over the segment level iterators by comparing the top level ordinals materialized when the segment level docs are popped from the segment level priority queues. This requires no extra memory and will be very performant. > Faster sorting for the /export handler >
[jira] [Created] (SOLR-14623) Minor improvements to SolrCloud tests
Ishan Chattopadhyaya created SOLR-14623: --- Summary: Minor improvements to SolrCloud tests Key: SOLR-14623 URL: https://issues.apache.org/jira/browse/SOLR-14623 Project: Solr Issue Type: Sub-task Security Level: Public (Default Security Level. Issues are Public) Reporter: Ishan Chattopadhyaya -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14621) Deprecate and eventually remove HDFS support
[ https://issues.apache.org/jira/browse/SOLR-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151056#comment-17151056 ] Ishan Chattopadhyaya commented on SOLR-14621: - [~markrmil...@gmail.com], [~hgadre], [~gchanan], [~krisden], would like to have your thoughts, please. > Deprecate and eventually remove HDFS support > > > Key: SOLR-14621 > URL: https://issues.apache.org/jira/browse/SOLR-14621 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Ishan Chattopadhyaya >Priority: Major > > This issue is to deprecate HDFS support from Solr. HDFS clutter should not be > part of Solr core and should either be removed completely or be packaged up > as an optional module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14622) Miscellaneous improvements to SolrCloud functionality, tests
Ishan Chattopadhyaya created SOLR-14622: --- Summary: Miscellaneous improvements to SolrCloud functionality, tests Key: SOLR-14622 URL: https://issues.apache.org/jira/browse/SOLR-14622 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Reporter: Ishan Chattopadhyaya Assignee: Ishan Chattopadhyaya Opening this issue to track some sub-tasks and issues that I wish to open as separate JIRAs. The goal to improve reliability of SolrCloud APIs (esp. collection APIs), speed up various parts of Solr, improve (speed up / harden) tests. I am working with [~noble.paul], and much of the thoughts/ideas/work is due to [~markrmil...@gmail.com]. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14404) CoreContainer level custom requesthandlers
[
https://issues.apache.org/jira/browse/SOLR-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151049#comment-17151049
]
ASF subversion and git services commented on SOLR-14404:
Commit c61b6a00fc08a066b167a93c890379c24d36363a in lucene-solr's branch
refs/heads/branch_8x from Erick Erickson
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c61b6a0 ]
SOLR-14404: Fix 8x precommit problem from
093c992fd208bbcd1cc3b7738b202c6737207a4f
> CoreContainer level custom requesthandlers
> --
>
> Key: SOLR-14404
> URL: https://issues.apache.org/jira/browse/SOLR-14404
> Project: Solr
> Issue Type: New Feature
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> caveats:
> * The class should be annotated with {{org.apache.solr.api.EndPoint}}.
> Which means only V2 APIs are supported
> * The path should have prefix {{/api/plugin}}
> add a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin", "class": "full.ClassName"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> add a plugin from a package
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin", "class": "pkgName:full.ClassName" ,
> "version: "1.0"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> remove a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "remove": "myplugin"
> }' http://localhost:8983/api/cluster/plugins
> {code}
> The configuration will be stored in the {{clusterprops.json}}
> as
> {code:java}
> {
> "plugins" : {
> "myplugin" : {"class": "full.ClassName" }
> }
> }
> {code}
> example plugin
> {code:java}
> public class MyPlugin {
> private final CoreContainer coreContainer;
> public MyPlugin(CoreContainer coreContainer) {
> this.coreContainer = coreContainer;
> }
> @EndPoint(path = "/myplugin/path1",
> method = METHOD.GET,
> permission = READ)
> public void call(SolrQueryRequest req, SolrQueryResponse rsp){
> rsp.add("myplugin.version", "2.0");
> }
> }
> {code}
> This plugin will be accessible on all nodes at {{/api/myplugin/path1}}. It's
> possible to add more methods at different paths. Ensure that all paths start
> with {{myplugin}} because that is the name in which the plugin is registered
> with. So {{/myplugin/path2}} , {{/myplugin/my/deeply/nested/path}} are all
> valid paths.
> It's possible that the user chooses to register the plugin with a different
> name. In that case , use a template variable as follows in paths
> {{$plugin-name/path1}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14626) Improvements to core container shutdown
Ishan Chattopadhyaya created SOLR-14626: --- Summary: Improvements to core container shutdown Key: SOLR-14626 URL: https://issues.apache.org/jira/browse/SOLR-14626 Project: Solr Issue Type: Sub-task Security Level: Public (Default Security Level. Issues are Public) Reporter: Ishan Chattopadhyaya Some general improvements to shutdown of a core container. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14621) Deprecate and eventually remove HDFS support
Ishan Chattopadhyaya created SOLR-14621: --- Summary: Deprecate and eventually remove HDFS support Key: SOLR-14621 URL: https://issues.apache.org/jira/browse/SOLR-14621 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Reporter: Ishan Chattopadhyaya This issue is to deprecate HDFS support from Solr. HDFS clutter should not be part of Solr core and should either be removed completely or be packaged up as an optional module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-10317) Solr Nightly Benchmarks
[ https://issues.apache.org/jira/browse/SOLR-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishan Chattopadhyaya resolved SOLR-10317. - Resolution: Fixed This is now done and in a good shape. This is currently hosted at https://github.com/thesearchstack/solr-bench (forked from https://github.com/fullstorydev/solr-bench). This now has the original local mode as discussed here. It has an additional GCP mode, where GCP nodes will be spun up and the tests will run on them. Plenty of scope for improvement. Moreover, we need to craft datasets and queries that we can run continuously. I shall address that in a separate issue. Thanks to everyone who contributed. Thanks to [~vivek.nar...@uga.edu] for his work on this, to Google Summer of Code program for helping us start the project and to FullStory for supporting part of the development. > Solr Nightly Benchmarks > --- > > Key: SOLR-10317 > URL: https://issues.apache.org/jira/browse/SOLR-10317 > Project: Solr > Issue Type: Task >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Labels: gsoc2017, mentor > Attachments: > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks-FINAL-PROPOSAL.pdf, > Narang-Vivek-SOLR-10317-Solr-Nightly-Benchmarks.docx, SOLR-10317.patch, > SOLR-10317.patch, Screenshot from 2017-07-30 20-30-05.png, > changes-lucene-20160907.json, changes-solr-20160907.json, managed-schema, > solrconfig.xml > > > Currently hosted at: http://212.47.242.214/MergedViewCloud.html > > Solr needs nightly benchmarks reporting. Similar Lucene benchmarks can be > found here, https://home.apache.org/~mikemccand/lucenebench/. > Preferably, we need: > # A suite of benchmarks that build Solr from a commit point, start Solr > nodes, both in SolrCloud and standalone mode, and record timing information > of various operations like indexing, querying, faceting, grouping, > replication etc. > # It should be possible to run them either as an independent suite or as a > Jenkins job, and we should be able to report timings as graphs (Jenkins has > some charting plugins). > # The code should eventually be integrated in the Solr codebase, so that it > never goes out of date. > There is some prior work / discussion: > # https://github.com/shalinmangar/solr-perf-tools (Shalin) > # https://github.com/chatman/solr-upgrade-tests/blob/master/BENCHMARKS.md > (Ishan/Vivek) > # SOLR-2646 & SOLR-9863 (Mark Miller) > # https://home.apache.org/~mikemccand/lucenebench/ (Mike McCandless) > # https://github.com/lucidworks/solr-scale-tk (Tim Potter) > There is support for building, starting, indexing/querying and stopping Solr > in some of these frameworks above. However, the benchmarks run are very > limited. Any of these can be a starting point, or a new framework can as well > be used. The motivation is to be able to cover every functionality of Solr > with a corresponding benchmark that is run every night. > Proposing this as a GSoC 2017 project. I'm willing to mentor, and I'm sure > [~shalinmangar] and [~markrmil...@gmail.com] would help here. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14620) Preference checks in Autoscaling framework can go wrong
[
https://issues.apache.org/jira/browse/SOLR-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilan Ginzburg updated SOLR-14620:
-
Description:
In the Autoscaling framework, preferences are values to maximize or minimize
when doing replica placement decisions and are used to guide the framework
towards the optimal placement decision. The default cluster preferences are
minimizing the number of cores on each node then maximizing the free disk space
on each node. In order to evaluate a {{Preference}}, the corresponding
parameter value for a node (for example number of cores or free disk space)
must be known.
Such parameters for a node (and their values) are stored in a {{Cell}} array in
the {{Row}} object. Parameter names come from multiple sources (not 100% sure
about the list below, hard to really trace legit values):
* {{From Preference}} definitions: {{freedisk}}, {{cores}}, {{heapUsage}} and
{{sysLoadAvg}}
* {{From Policy}} definitions: {{withCollection}}, {{port}}, {{ip_1}},
{{ip_2}}, {{ip_3}}, {{ip_4}}, {{freedisk}}, {{nodeRole}}, {{cores}},
{{sysLoadAvg}}, {{heapUsage}}, {{host}}, {{node}}, {{nodeset}}, {{metrics:***}}
({{*}} meaning any string)
* Possibly from other places...
{{"cores"}} and {{"freedisk"}} are always added to the {{Cell}} array in
{{Row}} (see {{Policy.DEFAULT_PARAMS_OF_INTEREST}}). The {{Cell}} array is
*sorted* by natural ordering of the parameter names. This causes {{"cores"}} to
be first and {{"freedisk"}} second, as you'll notice all other parameter names
listed above are lexicographically greater than these two.
When comparing rows in {{Preference.compare()}} (used for sorting them), the
value of the {{Preference}} is obtained from the {{Cell}} with array index
equal to the {{Preference}} index (starts at 0, in the order declared).
This obviously only makes sense if the {{Cell}} array order is identical to
{{Preference}} list order. {{Preferences}} therefore would have to be provided
by increasing parameter name and no parameters should exist in the {{Cell}}
array that are lexicographically smaller than the "highest" {{Preference}}
without having a matching {{Preference}}.
This basically means that when preferences are the default minimize number of
cores first then maximize freedisk second, the check works. But if for example
cluster preferences are explicitly defined to maximize freedisk first then
minimize number of cores, the check is broken. This will be more apparent when
parameters to maximize are swapped with parameters to minimize (which would be
the case here).
Unclear to me what's the real impact of this issue.
was:
In the Autoscaling framework, preferences are values to maximize or minimize
when doing replica placement decisions and are used to guide the framework
towards the optimal placement decision. The default cluster preferences are
minimizing the number of cores on each node then maximizing the free disk space
on each node. In order to evaluate a {{Preference}}, the corresponding
parameter value for a node (for example number of cores or free disk space)
must be known.
Such parameters for a node (and their values) are stored in a {{Cell}} array in
the {{Row}} object. Parameter names come from multiple sources (not 100% sure
about the list below, hard to really trace legit values):
* {{From Preference}} definitions: {{freedisk}}, {{cores}}, {{heapUsage}} and
{{sysLoadAvg}}
* {{From Policy}} definitions: {{withCollection}}, {{port}}, {{ip_1}},
{{ip_2}}, {{ip_3}}, {{ip_4}}, {{freedisk}}, {{nodeRole}}, {{cores}},
{{sysLoadAvg}}, {{heapUsage}}, {{host}}, {{node}}, {{nodeset}}, {{metrics:*}}
({{*}} meaning any string)
* Possibly from other places...
{{"cores"}} and {{"freedisk"}} are always added to the {{Cell}} array in
{{Row}} (see {{Policy.DEFAULT_PARAMS_OF_INTEREST}}). The {{Cell}} array is
*sorted* by natural ordering of the parameter names. This causes {{"cores"}} to
be first and {{"freedisk"}} second, as you'll notice all other parameter names
listed above are lexicographically greater than these two.
When comparing rows in {{Preference.compare()}} (used for sorting them), the
value of the {{Preference}} is obtained from the {{Cell}} with array index
equal to the {{Preference}} index (starts at 0, in the order declared).
This obviously only makes sense if the {{Cell}} array order is identical to
{{Preference}} list order. {{Preferences}} therefore would have to be provided
by increasing parameter name and no parameters should exist in the {{Cell}}
array that are lexicographically smaller than the "highest" {{Preference}}
without having a matching {{Preference}}.
This basically means that when preferences are the default minimize number of
cores first then maximize freedisk second, the check works. But if for example
cluster preferences are explicitly defined to maximize freedisk first then
minimize number of cores, th
[jira] [Created] (SOLR-14625) Leader election: getting rid of a 2500ms wait
Ishan Chattopadhyaya created SOLR-14625: --- Summary: Leader election: getting rid of a 2500ms wait Key: SOLR-14625 URL: https://issues.apache.org/jira/browse/SOLR-14625 Project: Solr Issue Type: Sub-task Security Level: Public (Default Security Level. Issues are Public) Reporter: Ishan Chattopadhyaya Some improvements to leader election. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14620) Preference checks in Autoscaling framework can go wrong
[
https://issues.apache.org/jira/browse/SOLR-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ilan Ginzburg updated SOLR-14620:
-
Description:
In the Autoscaling framework, preferences are values to maximize or minimize
when doing replica placement decisions and are used to guide the framework
towards the optimal placement decision. The default cluster preferences are
minimizing the number of cores on each node then maximizing the free disk space
on each node. In order to evaluate a {{Preference}}, the corresponding
parameter value for a node (for example number of cores or free disk space)
must be known.
Such parameters for a node (and their values) are stored in a {{Cell}} array in
the {{Row}} object. Parameter names come from multiple sources (not 100% sure
about the list below, hard to really trace legit values):
* {{From Preference}} definitions: {{freedisk}}, {{cores}}, {{heapUsage}} and
{{sysLoadAvg}}
* {{From Policy}} definitions: {{withCollection}}, {{port}}, {{ip_1}},
{{ip_2}}, {{ip_3}}, {{ip_4}}, {{freedisk}}, {{nodeRole}}, {{cores}},
{{sysLoadAvg}}, {{heapUsage}}, {{host}}, {{node}}, {{nodeset}}, {{metrics:\*}}
(\* meaning any string)
* Possibly from other places...
{{"cores"}} and {{"freedisk"}} are always added to the {{Cell}} array in
{{Row}} (see {{Policy.DEFAULT_PARAMS_OF_INTEREST}}). The {{Cell}} array is
*sorted* by natural ordering of the parameter names. This causes {{"cores"}} to
be first and {{"freedisk"}} second, as you'll notice all other parameter names
listed above are lexicographically greater than these two.
When comparing rows in {{Preference.compare()}} (used for sorting them), the
value of the {{Preference}} is obtained from the {{Cell}} with array index
equal to the {{Preference}} index (starts at 0, in the order declared).
This obviously only makes sense if the {{Cell}} array order is identical to
{{Preference}} list order. {{Preferences}} therefore would have to be provided
by increasing parameter name and no parameters should exist in the {{Cell}}
array that are lexicographically smaller than the "highest" {{Preference}}
without having a matching {{Preference}}.
This basically means that when preferences are the default minimize number of
cores first then maximize freedisk second, the check works. But if for example
cluster preferences are explicitly defined to maximize freedisk first then
minimize number of cores, the check is broken. This will be more apparent when
parameters to maximize are swapped with parameters to minimize (which would be
the case here).
Unclear to me what's the real impact of this issue.
was:
In the Autoscaling framework, preferences are values to maximize or minimize
when doing replica placement decisions and are used to guide the framework
towards the optimal placement decision. The default cluster preferences are
minimizing the number of cores on each node then maximizing the free disk space
on each node. In order to evaluate a {{Preference}}, the corresponding
parameter value for a node (for example number of cores or free disk space)
must be known.
Such parameters for a node (and their values) are stored in a {{Cell}} array in
the {{Row}} object. Parameter names come from multiple sources (not 100% sure
about the list below, hard to really trace legit values):
* {{From Preference}} definitions: {{freedisk}}, {{cores}}, {{heapUsage}} and
{{sysLoadAvg}}
* {{From Policy}} definitions: {{withCollection}}, {{port}}, {{ip_1}},
{{ip_2}}, {{ip_3}}, {{ip_4}}, {{freedisk}}, {{nodeRole}}, {{cores}},
{{sysLoadAvg}}, {{heapUsage}}, {{host}}, {{node}}, {{nodeset}}, {{metrics:***}}
({{*}} meaning any string)
* Possibly from other places...
{{"cores"}} and {{"freedisk"}} are always added to the {{Cell}} array in
{{Row}} (see {{Policy.DEFAULT_PARAMS_OF_INTEREST}}). The {{Cell}} array is
*sorted* by natural ordering of the parameter names. This causes {{"cores"}} to
be first and {{"freedisk"}} second, as you'll notice all other parameter names
listed above are lexicographically greater than these two.
When comparing rows in {{Preference.compare()}} (used for sorting them), the
value of the {{Preference}} is obtained from the {{Cell}} with array index
equal to the {{Preference}} index (starts at 0, in the order declared).
This obviously only makes sense if the {{Cell}} array order is identical to
{{Preference}} list order. {{Preferences}} therefore would have to be provided
by increasing parameter name and no parameters should exist in the {{Cell}}
array that are lexicographically smaller than the "highest" {{Preference}}
without having a matching {{Preference}}.
This basically means that when preferences are the default minimize number of
cores first then maximize freedisk second, the check works. But if for example
cluster preferences are explicitly defined to maximize freedisk first then
minimize number of cores, the
[GitHub] [lucene-solr] chatman opened a new pull request #1647: General improvements to coreContainer shutdown
chatman opened a new pull request #1647: URL: https://github.com/apache/lucene-solr/pull/1647 * Removing unnecessary "preClose()" * Other changes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman opened a new pull request #1646: SOLR-14625: Leader election improvements
chatman opened a new pull request #1646: URL: https://github.com/apache/lucene-solr/pull/1646 * Removing a 2500ms wait. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman opened a new pull request #1645: SOLR-14624: Improvements to waitFor* handling
chatman opened a new pull request #1645: URL: https://github.com/apache/lucene-solr/pull/1645 1. CollectionsHandler#waitForActiveCollection() to only check replica counts 2. Remove polling from OCMH#waitForCoreNodeName(), OCMH#waitForNewShard(), OCMH#waitToSeeReplicasInState() This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman opened a new pull request #1644: SOLR-14623: Minor Improvements to SolrCloud tests
chatman opened a new pull request #1644: URL: https://github.com/apache/lucene-solr/pull/1644 Here are some of the changes herein: 1. MiniSolrCloudCluster: Changing the polling to waitForState (based on watch/latch) 2. Improve waiting for Jetty stop, wait until node is not in liveNodes (JettySolrRunner) 3. TestCloudConsistency: Explicitly pick the jetty hosting shard1's leader 4. TestWaitForStateWithJettyShutdowns: Wait until replica becomes active 5. TestCollectionsAPIViaSolrCloudCluster: Don't wait for the collection to be deleted fully at the end 6. TestRecovery: Unset the UpdateLog hooks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-site] msokolov merged pull request #21: publish commits adding sokolov to PMC
msokolov merged pull request #21: URL: https://github.com/apache/lucene-site/pull/21 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-site] msokolov opened a new pull request #21: publish commits adding sokolov to PMC
msokolov opened a new pull request #21: URL: https://github.com/apache/lucene-site/pull/21 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-site] msokolov closed pull request #20: Publish edits adding sokolov to PMC
msokolov closed pull request #20: URL: https://github.com/apache/lucene-site/pull/20 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-site] msokolov commented on pull request #20: Publish edits adding sokolov to PMC
msokolov commented on pull request #20: URL: https://github.com/apache/lucene-site/pull/20#issuecomment-653521986 hmm this PR got 35 commits in it somehow -- abandoning This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-site] msokolov opened a new pull request #20: Publish edits adding sokolov to PMC
msokolov opened a new pull request #20: URL: https://github.com/apache/lucene-site/pull/20 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
