Hello! I'm new to the list and I have a bit of an issue that I could use some help with.
I'm in the process of upgrading our Solr installation from legacy to cloud. I'm new to the idea of Solr Cloud, so I've been wading through the documentation and trying to get a basic cluster up and running. I've got my Zookeeper ensemble set up, talking to each other, and accessible to my network via DNS hostnames. I'm using the official Solr 8.2 docker image from docker hub. Please see more environment information following the gigantic stack trace below. I've tried to run the docker image both locally and in the same Amazon VPC as the ZK ensemble, but every time it tries to start up I get this message in both cases: 2019-10-17 22:30:03.443 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper 2019-10-17 22:30:23.539 WARN (main-SendThread(zk1:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 20095ms for sessionid 0x0 2019-10-17 22:30:43.612 WARN (main-SendThread(zk3:2181)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 20005ms for sessionid 0x0 2019-10-17 22:30:43.724 ERROR (main-EventThread) [ ] o.a.z.ClientCnxn Error while calling watcher => java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$186/0x0000000100328440@5b1d0665 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@64e89eea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$186/0x0000000100328440@5b1d0665 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@64e89eea[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) ~[?:?] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194) ~[?:?] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient$ProcessWatchWithExecutor.process(SolrZkClient.java:843) ~[?:?] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) ~[?:?] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) ~[?:?] 2019-10-17 22:30:43.742 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs 2019-10-17 22:30:43.818 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: Error occurred while loading solr.xml from zookeeper at org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig(SolrDispatchFilter.java:289) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:259) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:181) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:136) at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:369) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:854) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:510) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:153) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:172) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:436) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:65) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:598) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167) at org.eclipse.jetty.server.Server.start(Server.java:418) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:382) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1797) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1746) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) at org.eclipse.jetty.start.Main.start(Main.java:490) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper zk1:2181,zk2:2181,zk3:2181 within 30000 ms at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:201) at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:125) at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:120) at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:107) at org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig(SolrDispatchFilter.java:282) ... 49 more Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper zk1:2181,zk2:2181,zk3:2181 within 30000 ms at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:250) at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:192) ... 53 more * zk1, zk2, and zk3 are all resolvable from within my docker container (running `echo ruok | nc zk1 2181` returns the expected "imok" response from ZK within the docker container where Solr is located) * The netcat command mentioned above shows up in the ZK logs, but the Solr attempts to connect do not (it's like the request isn't even getting to ZK) * Zookeeper is not set up as secure at this time (no ACLs required) * I'm using the following command line to start a basic solr cloud instance as per the documentation: `bin/solr start -c -z zk1:2181,zk2:2181,zk3:2181` with all other parameters being the defaults as specified by the docker image * Interestingly, I don't see a connection attempt to zk2 showing up in the above log trace. Is that a clue? Does anyone have any ideas why solr can't connect to ZK? I haven't been able to find any logs or information as to WHY it can't connect to ZK, and to my knowledge there's no reason it shouldn't connect to the ZK ensemble if the netcat command is able to resolve the hostnames. Where can I find out this information or where can I look? Any and all suggestions are welcomed and very much appreciated! -- Drew(i...@gmail.com) Kidder -- I Drive Way Too Fast To Worry About Cholesterol.
