Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3243570062 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -589,16 +589,17 @@ The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000 All three nodes must be started before the quorum can elect a leader and become operational. NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions. +When nodes share the same working directory, also set `solr.zookeeper.server.datadir` to a distinct path per node to avoid ZooKeeper data directory conflicts. [source,bash] export LH="localhost" ZK_HOST="$LH:21000,$LH:31000,$LH:41000" ROLES="data:on,overseer:allowed,zookeeper_quorum:on" -bin/solr start -p 2 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -bin/solr start -p 3 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -bin/solr start -p 4 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" +bin/solr start -p 2 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -Dsolr.zookeeper.server.datadir=zoo_home_1 Review Comment: Yeah, I think the whole point of this embedded mode is that people don't need to think about it. I think if "I want to do it somewhere special" then that leads you more towards the "I will set up my own zk ensemble"... Just thinking from the perspective of a non Solr expert... This feature is really going to be a hit among the novices to Solr who don't want to think about Cloud architectures ;-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on PR #2391: URL: https://github.com/apache/solr/pull/2391#issuecomment-4453755547 > I just made the offsets for zookeeper server port and quorum and election ports configurable with sysprops. Still defaults to +1000. And defaulting quorum port to +1010 and election port to +1020. This means you can run 10 instances on same host on 8983, 8984, 8985, 8986 without any conflicts. And it is easily configurable to +1, +2, +3 I like it! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on PR #2391: URL: https://github.com/apache/solr/pull/2391#issuecomment-4452794698 I just made the offsets for zookeeper server port and quorum and election ports configurable with sysprops. Still defaults to +1000. And defaulting quorum port to +1010 and election port to +1020. This means you can run 10 instances on same host on 8983, 8984, 8985, 8986 without any conflicts. And it is easily configurable to +1, +2, +3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3242921988
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -192,13 +196,62 @@ public void initZooKeeper(final CoreContainer cc,
CloudConfig config) {
e);
}
} else if (zkServerEnabled) {
- // Old school ZooKeeperServerMain being used under the covers (bin/solr
-c without ZK_HOST).
- zkServer =
- SolrZkServer.createAndStart(config.getZkHost(), solrHome,
config.getSolrHostPort());
+ try {
+final Path zkDataHome =
+Path.of(
+EnvUtils.getProperty(
+"solr.zookeeper.server.datadir",
solrHome.resolve("zoo_data").toString()));
Review Comment:
Yea, can be extracted to a constant if we start using the same property
multiple times..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3242919339 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -589,16 +589,17 @@ The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000 All three nodes must be started before the quorum can elect a leader and become operational. NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions. +When nodes share the same working directory, also set `solr.zookeeper.server.datadir` to a distinct path per node to avoid ZooKeeper data directory conflicts. [source,bash] export LH="localhost" ZK_HOST="$LH:21000,$LH:31000,$LH:41000" ROLES="data:on,overseer:allowed,zookeeper_quorum:on" -bin/solr start -p 2 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -bin/solr start -p 3 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -bin/solr start -p 4 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" +bin/solr start -p 2 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -Dsolr.zookeeper.server.datadir=zoo_home_1 Review Comment: The multi solr on single host is mostly for dev/test, so nice to have but not front and center? At least, now the zoo-data folder is configurable if folks want to place it somewhere special. But maybe we should also allow auto unique data folder through property? Also the zoo-config folder (defaulting to SOLR_HOME) needs the same... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3241382915 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -589,16 +589,17 @@ The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000 All three nodes must be started before the quorum can elect a leader and become operational. NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions. +When nodes share the same working directory, also set `solr.zookeeper.server.datadir` to a distinct path per node to avoid ZooKeeper data directory conflicts. [source,bash] export LH="localhost" ZK_HOST="$LH:21000,$LH:31000,$LH:41000" ROLES="data:on,overseer:allowed,zookeeper_quorum:on" -bin/solr start -p 2 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -bin/solr start -p 3 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -bin/solr start -p 4 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" +bin/solr start -p 2 -z "$ZK_HOST" -Dsolr.node.roles="$ROLES" -Dsolr.zookeeper.server.datadir=zoo_home_1 Review Comment: is this really better? What if we did soemthing like `./blah/blah/zoo_home/localhost:21000` and `./blah/blah/zoo_home/localhost:31000` and ``./blah/blah/zoo_home/localhost:41000`, and generated it. I don't think the vast majority of folks even want to think about zook_home! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3241366574 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 Review Comment: yeah... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3241365135
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -192,13 +196,62 @@ public void initZooKeeper(final CoreContainer cc,
CloudConfig config) {
e);
}
} else if (zkServerEnabled) {
- // Old school ZooKeeperServerMain being used under the covers (bin/solr
-c without ZK_HOST).
- zkServer =
- SolrZkServer.createAndStart(config.getZkHost(), solrHome,
config.getSolrHostPort());
+ try {
+final Path zkDataHome =
+Path.of(
+EnvUtils.getProperty(
+"solr.zookeeper.server.datadir",
solrHome.resolve("zoo_data").toString()));
Review Comment:
There is NOT a strong pattern. I think it's personal preference. For
me, I always find the text version easier to read than the
`SOLR_ZOOKEEPER_SERVER_DETAILED` version ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3234373446
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -192,13 +196,62 @@ public void initZooKeeper(final CoreContainer cc,
CloudConfig config) {
e);
}
} else if (zkServerEnabled) {
- // Old school ZooKeeperServerMain being used under the covers (bin/solr
-c without ZK_HOST).
- zkServer =
- SolrZkServer.createAndStart(config.getZkHost(), solrHome,
config.getSolrHostPort());
+ try {
+final Path zkDataHome =
+Path.of(
+EnvUtils.getProperty(
+"solr.zookeeper.server.datadir",
solrHome.resolve("zoo_data").toString()));
Review Comment:
Do we normally keep constants like this inline or do we have somewhere we
try to keep all sysprop-names for e.g. zookeeper together in one place?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3234355099 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 Review Comment: And come to think of it, each and every of these ports should probably be configurable individually even if they default to some port offset... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3229675779
##
solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
##
@@ -253,10 +263,10 @@ public static Properties getProperties(Path configPath)
throws ConfigException {
// Given zkHost=localhost:,localhost: this will inject
// server.0=localhost:1112:1113
// server.1=localhost:2223:2224
- public static void injectServers(Properties props, boolean zkRun, String
zkHost) {
+ public static void injectServers(Properties props, String zkHost) {
// if clientPort not already set, use zkRun
-if (zkRun && props.getProperty("clientPort") == null) {
+if (props.getProperty("clientPort") == null) {
// int portIdx = zkRun.lastIndexOf(':');
int portIdx = "".lastIndexOf(':');
Review Comment:
Feel like figuring out what is the correct code here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3229656915 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 Review Comment: I’d prefer if Solr occupied a more narrow port space, say within 100 ports total, so you could start Solr on 8000, 8100, 8200 and each instance would have ports close. Instead of node 1 using 8000, 9000, 1, 11000, while node 2 using 8100, 9100, 10100, 11100…. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3228619564 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 Review Comment: Yeah, after commenting elsewhere, I like your +2000 and +3000 idea. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3228599273 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 +|=== + +=== Starting a 3-Node Embedded ZK Ensemble + +The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000`), and every node must use `zookeeper_quorum:on`. +All three nodes must be started before the quorum can elect a leader and become operational. + +NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions. + +[source,bash] + +export LH="localhost" +ZK="-z $LH:21000,$LH:31000,$LH:41000" +ROLES="-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on" + +bin/solr start -p 2 $ZK $ROLES +bin/solr start -p 3 $ZK $ROLES +bin/solr start -p 4 $ZK $ROLES Review Comment: so, we can't do 8983, 8984,8985? That is definitly a conceptual change for all of us. I'm used to zk being 1000 more that my port... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3228557730
##
solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
##
@@ -253,10 +263,10 @@ public static Properties getProperties(Path configPath)
throws ConfigException {
// Given zkHost=localhost:,localhost: this will inject
// server.0=localhost:1112:1113
// server.1=localhost:2223:2224
- public static void injectServers(Properties props, boolean zkRun, String
zkHost) {
+ public static void injectServers(Properties props, String zkHost) {
// if clientPort not already set, use zkRun
-if (zkRun && props.getProperty("clientPort") == null) {
+if (props.getProperty("clientPort") == null) {
// int portIdx = zkRun.lastIndexOf(':');
int portIdx = "".lastIndexOf(':');
Review Comment:
not totally... And I think it was me hacking around a bit trying to figure
out all the various sys props that existed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3217855127 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 Review Comment: I modified the default ports used for quorum peer and leader election ports. They were something like client-port minus 3000 and minus 4000. Now they are +1 and +2. I feel we spread our port numbers too wide. But then after the change we got another problem that we could not start three test solr nodes on 8983, 8984, 8985, since the first one would occupy port 9983, 9984, 9985 for zookeeper. I am willing to revert the +1, +2 to something else, but -3000 and -4000 seemed a bit arbitrary, why not +2000 and +3000 since client-port is +1000? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3217834548 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 +|=== + +=== Starting a 3-Node Embedded ZK Ensemble + +The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000`), and every node must use `zookeeper_quorum:on`. +All three nodes must be started before the quorum can elect a leader and become operational. + +NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions. + +[source,bash] + +export LH="localhost" +ZK="-z $LH:21000,$LH:31000,$LH:41000" +ROLES="-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on" + +bin/solr start -p 2 $ZK $ROLES +bin/solr start -p 3 $ZK $ROLES +bin/solr start -p 4 $ZK $ROLES Review Comment: I fixed it it commit ccf6ec3add1d0ee583d25f6cf518e4030d1f04be - appending `-` to the `zoo_home` folder name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r3217792431 ## solr/solr-ref-guide/modules/deployment-guide/pages/zookeeper-ensemble.adoc: ## @@ -546,6 +546,68 @@ set SOLR_OPTS=%SOLR_OPTS% -Djute.maxbuffer=0x20 == +[[embedded-zookeeper-ensemble]] +== Embedded ZooKeeper Ensemble (Experimental) + +WARNING: This feature is *experimental / alpha* and is not suitable for production use. +The configuration interface may change without notice in future Solr releases. + +Normally, Solr's embedded ZooKeeper runs in "standalone" mode and cannot tolerate node failures. +Starting in Solr 10.1, it is possible to run Solr nodes where each node's embedded ZooKeeper participates in a multi-node quorum (ensemble). +This allows a self-contained SolrCloud cluster without a separate ZooKeeper deployment — useful for development, testing, or small low-stakes clusters. + +=== How It Works + +Each Solr node that has the `zookeeper_quorum` xref:node-roles.adoc[node role] set to `on` will start an embedded `ZooKeeperServerEmbedded` instance on startup. +Solr identifies which host in the ZK connection string it corresponds to (by matching host+port) and assigns it a ZooKeeper `myid`. + +Port scheme used by each node: + +|=== +|Port |Formula |Example (Solr on 8983) + +|Solr HTTP +|`solr_port` +|8983 + +|ZK client +|`solr_port + 1000` +|9983 + +|ZK quorum peer +|`zk_client_port + 1` +|9984 + +|ZK leader election +|`zk_client_port + 2` +|9985 +|=== + +=== Starting a 3-Node Embedded ZK Ensemble + +The ZK connection string must list all nodes' ZK client ports (`solr_port + 1000`), and every node must use `zookeeper_quorum:on`. +All three nodes must be started before the quorum can elect a leader and become operational. + +NOTE: On a single machine, each node needs distinct ports. Because each ZK instance also binds to `zk_client_port+1` (quorum peer) and `zk_client_port+2` (leader election), Solr ports must be spaced *at least 1001 apart* to avoid collisions. + +[source,bash] + +export LH="localhost" +ZK="-z $LH:21000,$LH:31000,$LH:41000" +ROLES="-Dsolr.node.roles=data:on,overseer:allowed,zookeeper_quorum:on" + +bin/solr start -p 2 $ZK $ROLES +bin/solr start -p 3 $ZK $ROLES +bin/solr start -p 4 $ZK $ROLES Review Comment: Running three nodes like this causes all three embedded zookeepers to use exact same `server/solr/zoo_home/data` folder, which is not good of course. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217647289
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -70,42 +81,119 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
-boolean zkRun =
EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+// zkServerEnabled is set whenever in solrCloud mode ('-c') but no
explicit zkHost/ZK_HOST is
+// provided.
+final boolean zkServerEnabled =
+EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+boolean zkQuorumNode = false;
+if
(NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM)))
{
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+}
-if (zkRun && config == null)
+if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+}
-if (config == null) return; // not in zk mode
+if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+}
+
+final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+final var solrHome = cc.getSolrHome();
+if (zkServerEnabled) {
+ if (!runAsQuorum) {
Review Comment:
Fixed. AI assistant refactored relevant code fragments, see the change in
this commit
https://github.com/apache/solr/pull/2391/changes/47251b9c7cc7904975bb1be303552cbf18fd5fdc?diff=split&w=1
- it looks solid..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217600740
##
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##
@@ -342,6 +344,185 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each
Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for
embedded ZK quorum mode");
+}
+if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+}
+
+Objects.requireNonNull(securityJson);
+this.baseDir = Objects.requireNonNull(baseDir);
+this.jettyConfig = Objects.requireNonNull(jettyConfig);
+this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+this.trackJettyMetrics = trackJettyMetrics;
+this.externalZkServer = true; // No ZkTestServer in quorum mode
+this.zkServer = null; // No single ZK server
+
+log.info("Starting cluster of {} servers with embedded ZK quorum in {}",
numServers, baseDir);
+Files.createDirectories(baseDir);
+
+// Phase 1: Reserve random ports for all nodes
+int[] ports = new int[numServers];
+for (int i = 0; i < numServers; i++) {
+ try (java.net.ServerSocket socket = new java.net.ServerSocket(0)) {
+ports[i] = socket.getLocalPort();
+ }
+}
+
+// Build the zkHost string with all ZK ports (Solr port + 1000)
+StringBuilder zkHostBuilder = new StringBuilder();
+for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+}
+this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers,
java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+}
+
+// Set system properties for embedded ZK quorum mode
+System.setProperty("solr.zookeeper.server.enabled", "true");
Review Comment:
So I modified things so that it is now enough to specify the
`zookeeper_quorum` node role, and it will then enable embedded zk without the
need for also `solr.zookeeper.server.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217570947
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -241,9 +333,17 @@ public SolrMetricsContext getSolrMetricsContext() {
}
}
- private String stripChroot(String zkRun) {
-if (zkRun == null || zkRun.trim().isEmpty() || zkRun.lastIndexOf('/') < 0)
return zkRun;
-return zkRun.substring(0, zkRun.lastIndexOf('/'));
+ private void startZooKeeperServerEmbedded(int port, String zkHomeDir) throws
Exception {
+Properties p = new Properties();
+try (FileReader fr = new FileReader(zkHomeDir + "/zoo.cfg",
StandardCharsets.UTF_8)) {
+ p.load(fr);
+}
Review Comment:
Redundant read of zoo.cfg, the file is already in memory in call site.
Refactor this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217566162
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -313,8 +413,37 @@ public void close() {
zkController.close();
}
} finally {
-if (zkServer != null) {
- zkServer.stop();
+try {
+ if (zkServer != null) {
+zkServer.stop();
+ }
+} finally {
+ if (zkServerEmbedded != null) {
+try {
+ zkServerEmbedded.close();
+ log.info("Closed embedded ZooKeeper server in quorum mode");
Review Comment:
Close log should come after the wait loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217562546
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -70,42 +81,119 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
-boolean zkRun =
EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+// zkServerEnabled is set whenever in solrCloud mode ('-c') but no
explicit zkHost/ZK_HOST is
+// provided.
+final boolean zkServerEnabled =
+EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+boolean zkQuorumNode = false;
+if
(NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM)))
{
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+}
-if (zkRun && config == null)
+if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+}
-if (config == null) return; // not in zk mode
+if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+}
+
+final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+final var solrHome = cc.getSolrHome();
+if (zkServerEnabled) {
+ if (!runAsQuorum) {
Review Comment:
It should be enough to specify the node role to start zk, no need for also
`-dsolr.zookeeper.server.enabled`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on PR #2391: URL: https://github.com/apache/solr/pull/2391#issuecomment-4418795685 I addressed Copilot's review points, then ran another review with Claude and addressed some of those concerns. Added ref-guide docs, marked as "experimental": https://github.com/user-attachments/assets/5361117b-adc6-4d34-ad63-6b63888ccc59"; /> https://github.com/user-attachments/assets/7e493131-55df-4a28-9adb-983004c40d9a"; /> https://github.com/user-attachments/assets/6ccf9d62-1424-4b55-84b3-421a230fa2d4"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217201302
##
solr/packaging/build.gradle:
##
@@ -305,6 +305,7 @@ task integrationTests(type: BatsTask) {
environment SOLR2_PORT: solrPort + 1
environment SOLR3_PORT: solrPort + 2
environment ZK_PORT: solrPort + 1000
+ environment SOLR_EXPORTER_PORT: solrPort + 100
Review Comment:
Unrelated change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217196903
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -70,42 +80,109 @@ public class ZkContainer {
public ZkContainer() {}
public void initZooKeeper(final CoreContainer cc, CloudConfig config) {
-boolean zkRun =
EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+// zkServerEnabled is set whenever in solrCloud mode ('-c') but no
explicit zkHost/ZK_HOST is
+// provided.
+final boolean zkServerEnabled =
+EnvUtils.getPropertyAsBool("solr.zookeeper.server.enabled", false);
+boolean zkQuorumNode = false;
+if
(NodeRoles.MODE_ON.equals(cc.nodeRoles.getRoleMode(NodeRoles.Role.ZOOKEEPER_QUORUM)))
{
+ zkQuorumNode = true;
+ log.info("Starting node in ZooKeeper Quorum role.");
+}
-if (zkRun && config == null)
+if (zkServerEnabled && config == null) {
throw new SolrException(
SolrException.ErrorCode.SERVER_ERROR,
"Cannot start Solr in cloud mode - no cloud config provided");
+}
+
+if (config == null) {
+ log.info("Solr is running in standalone mode");
+ return;
+}
-if (config == null) return; // not in zk mode
+final boolean runAsQuorum = config.getZkHost() != null && zkQuorumNode;
String zookeeperHost = config.getZkHost();
+final var solrHome = cc.getSolrHome();
+if (zkServerEnabled) {
+ if (!runAsQuorum) {
+// Old school ZooKeeperServerMain being used under the covers.
+zkServer =
+SolrZkServer.createAndStart(config.getZkHost(), solrHome,
config.getSolrHostPort());
+
+// set client from server config if not already set
+if (zookeeperHost == null) {
+ zookeeperHost = zkServer.getClientString();
+}
+ } else {
+// ZooKeeperServerEmbedded being used under the covers.
+// Figure out where to put zoo-data
+final var zkHomeDir = solrHome.resolve("zoo_home");
+final var zkDataDir = zkHomeDir.resolve("data");
+
+// Populate a zoo.cfg
+final String zooCfgTemplate =
+""
++ "tickTime=2000\n"
++ "initLimit=10\n"
++ "syncLimit=5\n"
++ "dataDir=@@DATA_DIR@@\n"
++ "4lw.commands.whitelist=mntr,conf,ruok\n"
++ "admin.enableServer=false\n"
++ "clientPort=@@ZK_CLIENT_PORT@@\n";
+
+final int zkPort = config.getSolrHostPort() + 1000;
+String zooCfgContents =
+zooCfgTemplate
+.replace("@@DATA_DIR@@", zkDataDir.toString())
+.replace("@@ZK_CLIENT_PORT@@", String.valueOf(zkPort));
+final String[] zkHosts = config.getZkHost().split(",");
+int myId = -1;
+// TODO: myId detection uses exact string matching between
config.getHost() and the host
+// portion of zkHost entries. This fails when zkHost uses "localhost"
but Solr is
+// configured with "127.0.0.1" (or vice versa). Consider resolving
hostnames/IPs before
+// matching, or matching by port alone (which is unique per node in a
single-machine setup).
+final String targetConnStringSection = config.getHost() + ":" + zkPort;
+if (log.isInfoEnabled()) {
+ log.info(
+ "Trying to match {} against zkHostString {} to determine myid",
+ targetConnStringSection,
+ config.getZkHost());
+}
+for (int i = 0; i < zkHosts.length; i++) {
+ final String host = zkHosts[i];
Review Comment:
Risk of ArrayIndexOutOfBounds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217187756
##
solr/core/src/java/org/apache/solr/core/ZkContainer.java:
##
@@ -60,7 +65,12 @@ public class ZkContainer {
private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
protected ZkController zkController;
+
+ // zkServer (and SolrZkServer) wrap a ZooKeeperServerMain if standalone
mode, but in quorum we
+ // just use ZooKeeperServerEmbedded
+ // directly! Why? Can we use ZooKeeperServerEmbedded in one node directly
instead?
private SolrZkServer zkServer;
+ private ZooKeeperServerEmbedded zkServerEmbedded;
Review Comment:
Must be volatile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217152288
##
solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
##
@@ -253,10 +263,10 @@ public static Properties getProperties(Path configPath)
throws ConfigException {
// Given zkHost=localhost:,localhost: this will inject
// server.0=localhost:1112:1113
// server.1=localhost:2223:2224
- public static void injectServers(Properties props, boolean zkRun, String
zkHost) {
+ public static void injectServers(Properties props, String zkHost) {
// if clientPort not already set, use zkRun
-if (zkRun && props.getProperty("clientPort") == null) {
+if (props.getProperty("clientPort") == null) {
// int portIdx = zkRun.lastIndexOf(':');
int portIdx = "".lastIndexOf(':');
Review Comment:
@epugh This line was modified by you and causes the below if() to be dead
code. Remember what's going on here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on PR #2391: URL: https://github.com/apache/solr/pull/2391#issuecomment-4418544846 @gerlowskija FYI - I updated the PR description to match current code state -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3217033659
##
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##
@@ -389,6 +663,87 @@ public void waitForNode(JettySolrRunner jetty, int
timeoutSeconds)
timeoutSeconds, TimeUnit.SECONDS, (o, n) -> n != null &&
n.contains(nodeName));
}
+ /**
+ * Wait for the expected number of live nodes in the cluster.
+ *
+ * @param expectedCount expected number of live nodes
+ * @param timeoutSeconds timeout in seconds
+ * @throws InterruptedException if interrupted while waiting
+ * @throws TimeoutException if the expected count is not reached within the
timeout
+ */
+ public void waitForLiveNodes(int expectedCount, int timeoutSeconds)
+ throws InterruptedException, TimeoutException {
+TimeOut timeout = new TimeOut(timeoutSeconds, TimeUnit.SECONDS,
TimeSource.NANO_TIME);
+while (!timeout.hasTimedOut()) {
+ long runningNodes =
jettys.stream().filter(JettySolrRunner::isRunning).count();
+ if (runningNodes == expectedCount) {
+log.info("Verified {} live nodes", runningNodes);
+return;
+ }
+ Thread.sleep(200);
+}
+// Final check after timeout
+long actualCount =
jettys.stream().filter(JettySolrRunner::isRunning).count();
+throw new TimeoutException(
+"Live node count mismatch: expected " + expectedCount + " but got " +
actualCount);
Review Comment:
Added a TODO to code..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
Copilot commented on code in PR #2391:
URL: https://github.com/apache/solr/pull/2391#discussion_r3215665335
##
solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java:
##
@@ -340,6 +346,274 @@ public MiniSolrCloudCluster(
}
}
+ /**
+ * Create a MiniSolrCloudCluster with embedded ZooKeeper quorum mode. Each
Solr node runs its own
+ * embedded ZooKeeper server, and together they form a quorum.
+ *
+ * @param numServers number of Solr servers (must be at least 3 for quorum)
+ * @param baseDir base directory that the mini cluster should be run from
+ * @param solrXml solr.xml file content
+ * @param jettyConfig Jetty configuration
+ * @param securityJson Optional security.json configuration
+ * @param trackJettyMetrics whether to track Jetty metrics
+ * @throws Exception if there was an error starting the cluster
+ */
+ MiniSolrCloudCluster(
+ int numServers,
+ Path baseDir,
+ String solrXml,
+ JettyConfig jettyConfig,
+ Optional securityJson,
+ boolean trackJettyMetrics,
+ boolean useEmbeddedZkQuorum)
+ throws Exception {
+
+if (!useEmbeddedZkQuorum) {
+ throw new IllegalArgumentException("This constructor is only for
embedded ZK quorum mode");
+}
+if (numServers < 3) {
+ throw new IllegalArgumentException(
+ "ZooKeeper quorum requires at least 3 nodes, got: " + numServers);
+}
+
+Objects.requireNonNull(securityJson);
+this.baseDir = Objects.requireNonNull(baseDir);
+this.jettyConfig = Objects.requireNonNull(jettyConfig);
+this.solrXml = solrXml == null ? DEFAULT_CLOUD_SOLR_XML : solrXml;
+this.trackJettyMetrics = trackJettyMetrics;
+this.externalZkServer = true; // No ZkTestServer in quorum mode
+this.zkServer = null; // No single ZK server
+
+log.info("Starting cluster of {} servers with embedded ZK quorum in {}",
numServers, baseDir);
+Files.createDirectories(baseDir);
+
+// Phase 1: Reserve random ports for all nodes
+int[] ports = reservePortPairs(numServers);
+
+// Build the zkHost string with all ZK ports (Solr port + 1000)
+StringBuilder zkHostBuilder = new StringBuilder();
+for (int i = 0; i < numServers; i++) {
+ if (i > 0) {
+zkHostBuilder.append(",");
+ }
+ int zkPort = ports[i] + 1000;
+ zkHostBuilder.append("127.0.0.1:").append(zkPort);
+}
+this.zkHost = zkHostBuilder.toString(); // Save for later use
+
+if (log.isInfoEnabled()) {
+ log.info("Reserved ports for {} nodes: {}", numServers,
java.util.Arrays.toString(ports));
+ log.info("ZK connection string: {}", this.zkHost);
+}
+
+// Set system properties for embedded ZK quorum mode
+System.setProperty("solr.zookeeper.server.enabled", "true");
+System.setProperty("solr.security.manager.enabled", "false");
+System.setProperty("solr.node.roles",
"data:on,overseer:allowed,zookeeper_quorum:on");
+System.setProperty("solr.test.sys.prop1", "propone");
+System.setProperty("solr.test.sys.prop2", "proptwo");
+System.setProperty("solr.zookeeper.client.timeout", "30"); // 5 minutes
+
+// Phase 2: Start all nodes in parallel
+List> startups = new ArrayList<>(numServers);
+for (int i = 0; i < numServers; i++) {
+ final int solrPort = ports[i];
+ final String nodeName = newNodeName();
+ startups.add(
+ () -> {
+Path runnerPath = createInstancePath(nodeName);
+Files.write(runnerPath.resolve("solr.xml"),
solrXml.getBytes(StandardCharsets.UTF_8));
+
+Properties nodeProps = new Properties();
+nodeProps.setProperty("zkHost", this.zkHost);
+nodeProps.setProperty("hostPort", String.valueOf(solrPort));
+
+JettyConfig newConfig =
JettyConfig.builder(jettyConfig).setPort(solrPort).build();
+
+JettySolrRunner jetty =
+!trackJettyMetrics
+? new JettySolrRunner(runnerPath.toString(), nodeProps,
newConfig)
+: new JettySolrRunnerWithMetrics(runnerPath.toString(),
nodeProps, newConfig);
+
+int zkPort = solrPort + 1000;
+log.info("Starting {} on port {} with ZK on port {}", nodeName,
solrPort, zkPort);
+jetty.start();
+log.info("Node {} started successfully", nodeName);
+
+jettys.add(jetty);
+synchronized (startupWait) {
+ startupWait.notifyAll();
+}
+return jetty;
+ });
+}
+
+final ExecutorService executorLauncher =
+ExecutorUtil.newMDCAwareCachedThreadPool(new
SolrNamedThreadFactory("jetty-launcher"));
+Collection> futures =
executorLauncher.invokeAll(startups);
+ExecutorUtil.shutdownAndAwaitTermination(executorLauncher);
+Exception startupError =
+checkForExceptions(
+"Error starting up MiniSolrCloudCluster with embedded ZK quorum",
future
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on PR #2391: URL: https://github.com/apache/solr/pull/2391#issuecomment-4416567928 Merged in latest main, hoping to pay some attention to this again. Also kicked off a Copilot review, to have some early feedback. Are you on board with trying to polish this PR branch in a "Phase 1" stable state which is mergable for 10.1? It would be a fantastic thing to get some alpha testers trying out, and then move on to Phase 2 in 10.2? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
epugh commented on code in PR #2391: URL: https://github.com/apache/solr/pull/2391#discussion_r2741464532 ## solr/core/src/test/org/apache/solr/cloud/TestEmbeddedZkQuorum.java: ## Review Comment: Would writing these as bats style change up the stability dynamics? And or help figure out why not working?? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-18094 Support running embedded-zk in "ensemble" mode with new node role [solr]
janhoy commented on PR #2391: URL: https://github.com/apache/solr/pull/2391#issuecomment-3816458591 I created SOLR-18094 as jira for this issue (sub-task of the SIP JIRA). Let's focus on getting this PR in shape for releasing the building blocks of the Zk code and the node-role. Target 10.1 as an experimental feature, can document it as such and solicit feedback from users. I made Precommit run, will spend some time some day to try to do a more thorough review. Let's make review comments for all known rough edges and things lacking. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
