Need some inputs or help in resolving replication across solr nodes. We have
installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes
configured. Enabled Solr replication in my Solrj client but the replication
fails and is unable to create a collection.
The same code works in our different environment where in I have 1 zookeeper
and 3 Solr nodes configured. Here is the exception I see on one of the nodes of
Solr when I try to create a collection in the environment where it fails. I
have compared the Solrconfig.xml on both the environments and didn’t see any
difference.
2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
status=0 QTime=6
2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer
Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer
(id=170740497916499012-sr01:8983_solr-n_0000000029) closing
2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager
Closing metric reporters for: solr.node
^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
at java.lang.Thread.run(Thread.java:745)
2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
status=0 QTime=6
2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer
Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer
(id=170740497916499012-sr01:8983_solr-n_0000000029) closing
2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager
Closing metric reporters for: solr.node
^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading
solr.xml from SolrHome (not found in ZooKeeper)
2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading
container configuration from /var/solr/data/Node122/solr.xml
2017-09-19 23:12:22.644 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating
UpdateShardHandler HTTP client with params:
socketTimeout=600000&connTimeout=60000&retry=true
2017-09-19 23:12:22.650 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
client=zk01:2181,zk02:2181,zk03:2181
2017-09-19 23:12:22.731 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live
nodes from ZooKeeper... (0) -> (1)
2017-09-19 23:12:22.762 INFO (main) [ ] o.a.s.c.Overseer Overseer (id=null)
closing
2017-09-19 23:12:22.780 INFO (main) [ ] o.a.s.c.ZkController Register node
as live in ZooKeeper:/live_nodes/sr01:8983_solr
2017-09-19 23:12:22.788 INFO
(zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
2017-09-19 23:12:23.241 INFO (main) [ ] o.a.s.c.CorePropertiesLocator Found
0 core definitions underneath /var/solr/data/Node122
2017-09-19 23:12:23.374 INFO (main) [ ] o.e.j.s.Server Started @2345ms
2017-09-19 23:12:47.125 INFO (qtp401424608-13) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=meetx&newCollection=true&name=CIUZLW_shard1_replica1&action=CREATE&numShards=1&collection=CIUZLW&shard=shard1&wt=javabin&version=2
2017-09-19 23:12:47.256 INFO
(zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json]
for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
2017-09-19 23:12:48.148 INFO
(zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json]
for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
2017-09-19 23:12:48.175 ERROR (qtp401424608-13) [c:CIUZLW s:shard1
x:CIUZLW_shard1_replica1] o.a.s.c.Config Exception during parsing file:
solrconfig.xml:org.xml.sax.SAXParseException; systemId:
solrres:/solrconfig.xml; lineNumber: 135; columnNumber: 13; Open quote is
expected for attribute "name" associated with an element type "str".
at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
at
com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1437)
at
com.sun.org.apache.xerces.internal.impl.XMLScanner.scanAttributeValue(XMLScanner.java:830)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanAttribute(XMLNSDocumentScannerImpl.java:440)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:256)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at org.apache.solr.core.Config.<init>(Config.java:143)
at org.apache.solr.core.Config.<init>(Config.java:90)
at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
at
org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
at
org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:96)
at
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
at
org.apache.solr.core.CoreContainer.create(CoreContainer.java:915)
at
org.apache.solr.core.CoreContainer.create(CoreContainer.java:854)
at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:370)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:745)
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive
and confidential use of the intended recipient and may constitute non-public
information. If you received this e-mail in error, disclosing, copying,
distributing or taking any action in reliance of this e-mail is strictly
prohibited and may be unlawful. Instead, please notify us immediately by return
e-mail and promptly delete this message and its attachments from your computer
system. We do not waive any work product or other applicable legal privilege(s)
by the transmission of this message.
***********************************************************
