Hi Chris, Not had this myself but is it possible there is still a 6.1 instance running on your solr nodes (which happened on my first attempt to update 6.1 to 6.2 yesterday)?
So check for running java processes. Hope this helps, Jon -----Original Message----- From: Chris Rogers [mailto:chris.rog...@bodleian.ox.ac.uk] Sent: Wednesday, August 31, 2016 9:34 AM To: solr-user@lucene.apache.org Subject: RE: SolrCloud: A previous ephemeral live node still exists Just pinging this again as I sent it late last night. Would be great if someone could help with this. It's got me totally stumped... Chris Rogers Digital Projects Manager Bodleian Digital Library Systems and Services chris.rog...@bodleian.ox.ac.uk ________________________________________ From: Chris Rogers [chris.rog...@bodleian.ox.ac.uk] Sent: 30 August 2016 21:34 To: solr-user@lucene.apache.org Subject: SolrCloud: A previous ephemeral live node still exists Hi all, I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0 and Zookeeper 3.4.8 I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to start more than one node in cloud mode with Solr 6.2.0. I have four VMs connected on a private network of Vagrant boxes, one running Zookeeper, and three Solr nodes. The first Solr node connects to Zookeeper as expected. I can access the admin and see the Cloud info there. But when I try and connect with a second node, something goes wrong. I get the following error in the Solr log: 2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized @490ms 2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server jetty-9.3.8.v20160314 2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at interval 0 2016-08-30 20:12:12.906 INFO (main) [ ] o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet 2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} has uncovered http methods for path: / 2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr' 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx) 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr 2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider 2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper 2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25b8acce name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider 2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) 2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml 2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr 2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New CoreContainer 182259421 2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr] 2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath: /home/vagrant/solr-6.2.0/server/solr/lib 2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,connectionsEvictorSleepDelay : 5000,maxConnectionIdleTime : 40000, 2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true 2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory 2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] 2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=172.28.128.3:2181 2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca 2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper 2016-08-30 20:12:13.549 INFO (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ece6797 name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties 2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs 2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.SolrException: A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already. at org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741) at org.apache.solr.cloud.ZkController.init(ZkController.java:675) at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419) at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) at org.eclipse.jetty.server.Server.start(Server.java:405) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.eclipse.jetty.start.Main.invokeMain(Main.java:214) at org.eclipse.jetty.start.Main.start(Main.java:457) at org.eclipse.jetty.start.Main.main(Main.java:75) 2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done 2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp} 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983} 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms It does start in the end, but it's not working properly, I just get errors in the Solr admin. The only reference I can find to the the error: "A previous ephemeral live node still exists. Solr cannot continue. Please ensure that no other Solr process using the same port is running already." ,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0 https://issues.apache.org/jira/browse/SOLR-8777 Has anyone else experienced this issue? Any hints on resolving? Chris Rogers Digital Projects Manager Bodleian Digital Library Systems and Services chris.rog...@bodleian.ox.ac.uk<mailto://chris.rog...@bodleian.ox.ac.uk>
