Re: CDCR (Solr6.x) does not start
Hi Uwe I am facing the same error as you , I am getting error 6538 ERROR (qtp110456297-20) [c:multi_dc_poc s:shard1 r:core_node3 x:multi_dc_poc_shard1_replica2] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to non-leader replica at org.apache.solr.handler.CdcrRequestHandler.handleLastProcessedVersionAction(CdcrRequestHandler.java:498) - I have 3 shards and 3 replicas in 3 solr machines Can you please help Thanks Neeraj -- View this message in context: http://lucene.472066.n3.nabble.com/CDCR-Solr6-x-does-not-start-tp4284742p4305205.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: CDCR (Solr6.x) does not start
Hi Renaud,
thank you for your response.
You asked for some further information:
1. Log messages at the source cluster:
As mentioned in my addendum "CDCR (Solr6.x) does not start (logfile)". I
changed the log level for all Handlers to TRACE and I got three Messages
for each shard caused by "Action LASTPROCESSEDVERSION sent to non-leader
replica .."
For me this looks like the blocker.
2.
Replication should start even if no commit has been sent to the source cluster.
Thanks for the clarification. It helps me to understand.
3.
The empty queue seems to indicate there is an issue, and that cdcr was unable
to instantiate the replicator for the target cluster.
Just to be sure, your source cluster has 4 shards, but not replica ? If it has
replicas, can you ensure that you execute these command on the shard leader.
At the beginning I tried to replicate 4 shards with an replication
factor of 3. Later on i simplified the environment by omitting the
replicas. (replication factor = 1)
Do you think having no replicas could the reason for the log messages above?
Regards
Uwe
Am 05.07.2016 um 14:55 schrieb Renaud Delbru:
Hi Uwe,
At first look, your configuration seems correct,
see my comments below.
On 28/06/16 15:36, Uwe Reh wrote:
9. Start CDCR
http://SOURCE:s_port/solr/scoll/cdcr?action=start&wt=json
{"responseHeader":{"status":0,"QTime":13},"status":["process","started","buffer","enabled"]}
! (not even a single query to the target's zookeeper ??)
Indeed, you should have observed a communication between the source
cluster and the target zookeeper. Do you see any errors in the log of
the source cluster ? Or a log message such as:
"Unable to instantiate the log reader for target collection ..."
10. Enter some test data into the SOURCE
11. Explicit commit in SOURCE
http://SOURCE:s_port/solr/scoll/update?commit=true&opensearcher=true
!! (at least now there should be some traffic, or?)
Replication should start even if no commit has been sent to the source
cluster.
12. Check errors and queues
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=queues&wt=json
{"responseHeader":{"status":0,"QTime":0},"queues":[],"tlogTotalSize":135,"tlogTotalCount":1,"updateLogSynchronizer":"stopped"}
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=errors&wt=json
{"responseHeader":{"status":0,"QTime":0},"errors":[]}
! Why is the element queues is empty
The empty queue seems to indicate there is an issue, and that cdcr was
unable to instantiate the replicator for the target cluster.
Just to be sure, your source cluster has 4 shards, but not replica ? If
it has replicas, can you ensure that you execute these command on the
shard leader.
Kind Regards
Re: CDCR (Solr6.x) does not start
Hi Uwe,
At first look, your configuration seems correct,
see my comments below.
On 28/06/16 15:36, Uwe Reh wrote:
9. Start CDCR
http://SOURCE:s_port/solr/scoll/cdcr?action=start&wt=json
{"responseHeader":{"status":0,"QTime":13},"status":["process","started","buffer","enabled"]}
! (not even a single query to the target's zookeeper ??)
Indeed, you should have observed a communication between the source
cluster and the target zookeeper. Do you see any errors in the log of
the source cluster ? Or a log message such as:
"Unable to instantiate the log reader for target collection ..."
10. Enter some test data into the SOURCE
11. Explicit commit in SOURCE
http://SOURCE:s_port/solr/scoll/update?commit=true&opensearcher=true
!! (at least now there should be some traffic, or?)
Replication should start even if no commit has been sent to the source
cluster.
12. Check errors and queues
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=queues&wt=json
{"responseHeader":{"status":0,"QTime":0},"queues":[],"tlogTotalSize":135,"tlogTotalCount":1,"updateLogSynchronizer":"stopped"}
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=errors&wt=json
{"responseHeader":{"status":0,"QTime":0},"errors":[]}
! Why is the element queues is empty
The empty queue seems to indicate there is an issue, and that cdcr was
unable to instantiate the replicator for the target cluster.
Just to be sure, your source cluster has 4 shards, but not replica ? If
it has replicas, can you ensure that you execute these command on the
shard leader.
Kind Regards
--
Renaud Delbru
Re: CDCR (Solr6.x) does not start (logfile)
Hi, trying to get more information, I restarted the SOURCE node and watched the log. For each shard i got following triple: WARN org.apache.solr.handler.CdcrRequestHandler - Action LASTPROCESSEDVERSION sent to non-leader replica @ scoll:shard1 ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: Action LASTPROCESSEDVERSION sent to non-leader replica WARN org.apache.solr.handler.CdcrUpdateLogSynchronizer - Caught unexpected exception org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://SOURCE:s_port/solr/scoll_shard1_replica1: Action LASTPROCESSEDVERSION sent to non-leader replica Could this the reason, why there is no further action? The SOURCE cloud has just the replicationfactor '1'. 'scoll_shard1_replica1' should have to be allays the leader, or? Regards Uwe
CDCR (Solr6.x) does not start
Hi,
I'm trying to get CDCR to run, but I can't even trigger any
communication between SOURCE and TARGET.
It seems to be a small but grave misunderstanding. I've tested a lot of
variants but now I'm blind on this point.
If anyone could give me a hint, I would appreciate.
Uwe
Testsetting:
Two nearly identical hosts (open solaris) with:
- a minimal zookeeper ensemble (one local installation (not embedded),
listening on port 2181)
- a minimal cloud (one node, one empty collection, 4 shards)
Initial both installations differ only in solrconfig.xml (snipplets below)
The tcp traffic was observed with 'snoop' (tcpdump). There are no packet
filters or other firewalls between both machines.
Testprocess:
1. Start node for TARGET
2. Create TARGET collection 'tcoll'
http://TARGET:t_port/solr/admin/collections?action=CREATE&name=tcoll&numShards=4&replicationFactor=1&maxShardsPerNode=4&collection.configName=cdcr
3. Get status
http://TARGET:t_port/solr/tcoll/cdcr?action=status&wt=json
{"responseHeader":{"status":0,"QTime":0},"status":["process","stopped","buffer","enabled"]}
4. Disable buffer
http://TARGET:t_port/solr/tcoll/cdcr?action=disablebuffer&wt=json
{"responseHeader":{"status":0,"QTime":12},"status":["process","stopped","buffer","disabled"]}
6. Start node for SOURCE
(like expected, no tcp between both hosts)
7. Create SOURCE collection 'scoll'
http://SOURCE:s_port/solr/admin/collections?action=CREATE&name=scoll&numShards=4&replicationFactor=1&maxShardsPerNode=4&collection.configName=cdcr
(no tcp between both hosts)
8. Get status
http://SOURCE:s_port/solr/scoll/cdcr?action=status&wt=json
{"responseHeader":{"status":0,"QTime":13},"status":["process","stopped","buffer","enabled"]}
(like expected, no tcp between both hosts)
9. Start CDCR
http://SOURCE:s_port/solr/scoll/cdcr?action=start&wt=json
{"responseHeader":{"status":0,"QTime":13},"status":["process","started","buffer","enabled"]}
! (not even a single query to the target's zookeeper ??)
10. Enter some test data into the SOURCE
11. Explicit commit in SOURCE
http://SOURCE:s_port/solr/scoll/update?commit=true&opensearcher=true
!! (at least now there should be some traffic, or?)
12. Check errors and queues
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=queues&wt=json
{"responseHeader":{"status":0,"QTime":0},"queues":[],"tlogTotalSize":135,"tlogTotalCount":1,"updateLogSynchronizer":"stopped"}
http://SOURCE:s_port/solr/scoll_shard1_replica1/cdcr?action=errors&wt=json
{"responseHeader":{"status":0,"QTime":0},"errors":[]}
! Why is the element queues is empty
where is my stupid bug
# solrconfig Source
${solr.ulog.dir:}
TARGET:2181
scoll
tcoll
1
# solrconfig Target
${solr.ulog.dir:}
${solr.autoCommit.maxdocs:1000}
${solr.autoCommit.maxTime:300}
true
${solr.autoSoftCommit.maxTime:60}
disabled
cdcr-processor-chain
##
# EOF
#
