[jira] (SOLR-9861) Support Cross Data Center Replication under HDFS
[
https://issues.apache.org/jira/browse/SOLR-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847554#comment-15847554
]
Kevin Risden commented on SOLR-9861:
I looked into this some and just sharing what I found in case it helps:
* CDCR has CdcrUpdateLog and CdcrTransactionLog which do a bulk of the CDCR work
* HDFS has HdfsUpdateLog and HdfsTransactionLog which do a bulk of the HDFS work
* CdcrUpdateLog and HdfsUpdateLog both extend the UpdateLog class
* CdcrTransactionLog and CdcrUpdateLog both extend the TransactionLog class
* CdcrUpdateLog and CdcrTransactionLog both deal with Linux files
The items above make it hard for the existing CDCR implementation classes to
reuse the HDFS implementation classes. It would probably be useful to refactor
some first to allow for more reuse between classes. The UpdateLog class is
pretty big and its not clear what all the pieces are for. There are inner
classes and enums within the UpdateLog class used elsewhere.
> Support Cross Data Center Replication under HDFS
>
>
> Key: SOLR-9861
> URL: https://issues.apache.org/jira/browse/SOLR-9861
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: CDCR
>Affects Versions: 6.3
>Reporter: Gero Willmes
>
> Currently Solr Cloud does not support CDCR under HDFS (according to
> http://search-lucene.com/m/Solr/eHNlLSriaVCpuU1?subj=Re+Using+Solr+CDCR+with+HdfsDirectoryFactory).
>
> If the CDCR configuration is performed according to
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462
> and at the same time the HdfsDirectoryFactory is configured on both Source
> cluster and Target cluster according to
> https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS
> it leads to the following error message during reload of the collection
> “collection_source” of the source cluster:
> {code}
> 2016-11-23 12:05:35.604 ERROR (qtp1134712904-8045) [c:collection_source
> s:shard1 r:core_node1 x:collection_source_shard1_replica1]
> o.a.s.s.HttpSolrCall null:org.apache.solr.common.SolrException: Error
> handling 'reload' action
> at
> org.apache.solr.handler.admin.CoreAdminOperation$3.call(CoreAdminOperation.java:150)
> at
> org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:367)
> at
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:158)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
> at
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:518)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
>
[jira] (SOLR-9861) Support Cross Data Center Replication under HDFS
[
https://issues.apache.org/jira/browse/SOLR-9861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated SOLR-9861:
---
Description:
Currently Solr Cloud does not support CDCR under HDFS (according to
http://search-lucene.com/m/Solr/eHNlLSriaVCpuU1?subj=Re+Using+Solr+CDCR+with+HdfsDirectoryFactory).
If the CDCR configuration is performed according to
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462
and at the same time the HdfsDirectoryFactory is configured on both Source
cluster and Target cluster according to
https://cwiki.apache.org/confluence/display/solr/Running+Solr+on+HDFS
it leads to the following error message during reload of the collection
“collection_source” of the source cluster:
{code}
2016-11-23 12:05:35.604 ERROR (qtp1134712904-8045) [c:collection_source
s:shard1 r:core_node1 x:collection_source_shard1_replica1] o.a.s.s.HttpSolrCall
null:org.apache.solr.common.SolrException: Error handling 'reload' action
at
org.apache.solr.handler.admin.CoreAdminOperation$3.call(CoreAdminOperation.java:150)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:367)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:158)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to reload core
[collection_source_shard1_replica1]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:920)
at
org.apache.solr.handler.admin.CoreAdminOperation$3.call(CoreAdminOperation.java:148)
... 31 more
Caused by: org.apache.solr.common.SolrException: Solr instance is not
configured with the cdcr update log.
{code}
However, the solr instance was configured correctly. CDCR should work in the
same way with HdfsDirectoryFactory as it works with StandardDirectoryFactory or
NRTCachingDirectoryFactory.
was:
Currently Solr Cloud does not support CDCR under HDFS (according to
http://search-lucene.com/m/Solr/eHNlLSriaVCpuU1?subj=Re+Using+Solr+CDCR+with+HdfsDirectoryFactory).
If the CDCR configuration is performed according to
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62687462
and at the same time the HdfsDirectoryFactory is configured on both Source
cluster and Target cluster
