Are you running zookeeper as container too. If yes , port 2181 needs to be exposed.
-Rajeswari On Thu, Oct 17, 2019 at 4:25 PM Drew Kidder <dre...@gmail.com> wrote: > Hello! I'm new to the list and I have a bit of an issue that I could use > some help with. > > I'm in the process of upgrading our Solr installation from legacy to cloud. > I'm new to the idea of Solr Cloud, so I've been wading through the > documentation and trying to get a basic cluster up and running. I've got my > Zookeeper ensemble set up, talking to each other, and accessible to my > network via DNS hostnames. I'm using the official Solr 8.2 docker image > from docker hub. Please see more environment information following the > gigantic stack trace below. I've tried to run the docker image both locally > and in the same Amazon VPC as the ZK ensemble, but every time it tries to > start up I get this message in both cases: > > 2019-10-17 22:30:03.443 INFO (main) [ ] o.a.s.c.c.ConnectionManager > Waiting for client to connect to ZooKeeper > 2019-10-17 22:30:23.539 WARN (main-SendThread(zk1:2181)) [ ] > o.a.z.ClientCnxn Client session timed out, have not heard from server in > 20095ms for sessionid 0x0 > 2019-10-17 22:30:43.612 WARN (main-SendThread(zk3:2181)) [ ] > o.a.z.ClientCnxn Client session timed out, have not heard from server in > 20005ms for sessionid 0x0 > 2019-10-17 22:30:43.724 ERROR (main-EventThread) [ ] o.a.z.ClientCnxn > Error while calling watcher => > java.util.concurrent.RejectedExecutionException: Task > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$186/0x0000000100328440@5b1d0665 > rejected from > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@64e89eea > [Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] > at > > java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) > java.util.concurrent.RejectedExecutionException: Task > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$186/0x0000000100328440@5b1d0665 > rejected from > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@64e89eea > [Terminated, > pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0] > at > > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) > ~[?:?] > at > > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) > ~[?:?] > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:194) > ~[?:?] > at > > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) > ~[?:?] > at > > org.apache.solr.common.cloud.SolrZkClient$ProcessWatchWithExecutor.process(SolrZkClient.java:843) > ~[?:?] > at > > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) > ~[?:?] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) > ~[?:?] > 2019-10-17 22:30:43.742 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could > not start Solr. Check solr/home property and the logs > 2019-10-17 22:30:43.818 ERROR (main) [ ] o.a.s.c.SolrCore > null:org.apache.solr.common.SolrException: Error occurred while loading > solr.xml from zookeeper > at > > org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig(SolrDispatchFilter.java:289) > at > > org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:259) > at > > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:181) > at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:136) > at > > org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) > at > > java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) > at > > java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) > at > > java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734) > at > > java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658) > at > > org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) > at > > org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:369) > at > org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) > at > > org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) > at > > org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:854) > at > > org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) > at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) > at > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > > org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) > at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) > at > > org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:510) > at > > org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:153) > at > > org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:172) > at > > org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:436) > at > > org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:65) > at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) > at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) > at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) > at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) > at > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > > org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:145) > at > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > > org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:598) > at > > org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) > at > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > > org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167) > at org.eclipse.jetty.server.Server.start(Server.java:418) > at > > org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119) > at > > org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) > at org.eclipse.jetty.server.Server.doStart(Server.java:382) > at > > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > > org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1797) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1746) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) > at org.eclipse.jetty.start.Main.start(Main.java:490) > at org.eclipse.jetty.start.Main.main(Main.java:77) > Caused by: org.apache.solr.common.SolrException: > java.util.concurrent.TimeoutException: Could not connect to ZooKeeper > zk1:2181,zk2:2181,zk3:2181 within 30000 ms > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:201) > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:125) > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:120) > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:107) > at > > org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig(SolrDispatchFilter.java:282) > ... 49 more > Caused by: java.util.concurrent.TimeoutException: Could not connect to > ZooKeeper zk1:2181,zk2:2181,zk3:2181 within 30000 ms > at > > org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:250) > at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:192) > ... 53 more > > * zk1, zk2, and zk3 are all resolvable from within my docker container > (running `echo ruok | nc zk1 2181` returns the expected "imok" response > from ZK within the docker container where Solr is located) > * The netcat command mentioned above shows up in the ZK logs, but the Solr > attempts to connect do not (it's like the request isn't even getting to ZK) > * Zookeeper is not set up as secure at this time (no ACLs required) > * I'm using the following command line to start a basic solr cloud instance > as per the documentation: `bin/solr start -c -z zk1:2181,zk2:2181,zk3:2181` > with all other parameters being the defaults as specified by the docker > image > * Interestingly, I don't see a connection attempt to zk2 showing up in the > above log trace. Is that a clue? > > Does anyone have any ideas why solr can't connect to ZK? I haven't been > able to find any logs or information as to WHY it can't connect to ZK, and > to my knowledge there's no reason it shouldn't connect to the ZK ensemble > if the netcat command is able to resolve the hostnames. Where can I find > out this information or where can I look? > > Any and all suggestions are welcomed and very much appreciated! > > -- > Drew(i...@gmail.com) > Kidder > > -- I Drive Way Too Fast To Worry About Cholesterol. >
