Re: CDCR Bootstrap
Thanks, Tom. Is that correct that i have to execute this for each shard? On Thu, Apr 26, 2018 at 10:19 AM, Tom Peters <tpet...@synacor.com> wrote: > I'm not sure under what conditions it will be automatically triggered, but > if you manually wanted to trigger a CDCR Bootstrap you need to issue the > following query to the leader in your target data center. > > /solr//cdcr?action=BOOTSTRAP= URL> > > The masterUrl will look something like (change the necessary values): > http%3A%2F%2Fsolr-leader.solrurl%3A8983%2Fsolr%2Fcollection > > > On Apr 26, 2018, at 10:15 AM, Susheel Kumar <susheel2...@gmail.com> > wrote: > > > > Anybody has idea how to trigger Solr CDCR BOOTSTRAP or under what > condition > > it gets triggered ? > > > > Thanks, > > Susheel > > > > On Tue, Apr 24, 2018 at 12:34 PM, Susheel Kumar <susheel2...@gmail.com> > > wrote: > > > >> Hello, > >> > >> I am wondering under what different conditions does that CDCR bootstrap > >> process gets triggered. I did notice it getting triggered after I > stopped > >> CDCR and then started again later and now I am trying to reproduce the > same > >> behavior. > >> > >> In case target cluster is left behind and buffer was disabled on > source, i > >> would like the CDCR bootstrap to trigger and sync target. > >> > >> Does deleting records from target and then starting CDCR would trigger > >> bootstrap ? > >> > >> Thanks, > >> Susheel > >> > >> > >> > > > > > > This message and any attachment may contain information that is > confidential and/or proprietary. Any use, disclosure, copying, storing, or > distribution of this e-mail or any attached file by anyone other than the > intended recipient is strictly prohibited. If you have received this > message in error, please notify the sender by reply email and delete the > message and any attachments. Thank you. >
Re: CDCR Bootstrap
I'm not sure under what conditions it will be automatically triggered, but if you manually wanted to trigger a CDCR Bootstrap you need to issue the following query to the leader in your target data center. /solr//cdcr?action=BOOTSTRAP= The masterUrl will look something like (change the necessary values): http%3A%2F%2Fsolr-leader.solrurl%3A8983%2Fsolr%2Fcollection > On Apr 26, 2018, at 10:15 AM, Susheel Kumar <susheel2...@gmail.com> wrote: > > Anybody has idea how to trigger Solr CDCR BOOTSTRAP or under what condition > it gets triggered ? > > Thanks, > Susheel > > On Tue, Apr 24, 2018 at 12:34 PM, Susheel Kumar <susheel2...@gmail.com> > wrote: > >> Hello, >> >> I am wondering under what different conditions does that CDCR bootstrap >> process gets triggered. I did notice it getting triggered after I stopped >> CDCR and then started again later and now I am trying to reproduce the same >> behavior. >> >> In case target cluster is left behind and buffer was disabled on source, i >> would like the CDCR bootstrap to trigger and sync target. >> >> Does deleting records from target and then starting CDCR would trigger >> bootstrap ? >> >> Thanks, >> Susheel >> >> >> This message and any attachment may contain information that is confidential and/or proprietary. Any use, disclosure, copying, storing, or distribution of this e-mail or any attached file by anyone other than the intended recipient is strictly prohibited. If you have received this message in error, please notify the sender by reply email and delete the message and any attachments. Thank you.
Re: CDCR Bootstrap
Anybody has idea how to trigger Solr CDCR BOOTSTRAP or under what condition it gets triggered ? Thanks, Susheel On Tue, Apr 24, 2018 at 12:34 PM, Susheel Kumar <susheel2...@gmail.com> wrote: > Hello, > > I am wondering under what different conditions does that CDCR bootstrap > process gets triggered. I did notice it getting triggered after I stopped > CDCR and then started again later and now I am trying to reproduce the same > behavior. > > In case target cluster is left behind and buffer was disabled on source, i > would like the CDCR bootstrap to trigger and sync target. > > Does deleting records from target and then starting CDCR would trigger > bootstrap ? > > Thanks, > Susheel > > >
CDCR Bootstrap
Hello, I am wondering under what different conditions does that CDCR bootstrap process gets triggered. I did notice it getting triggered after I stopped CDCR and then started again later and now I am trying to reproduce the same behavior. In case target cluster is left behind and buffer was disabled on source, i would like the CDCR bootstrap to trigger and sync target. Does deleting records from target and then starting CDCR would trigger bootstrap ? Thanks, Susheel
Re: Does CDCR Bootstrap sync leaves replica's out of sync
Thanks Amrit, Peter. I'll go with option#2 but what else i am seeing is that after bootstrap, target has not been synched further (even though we have continous indexing happening in source) which I believe due to the leaders on source cluster shows updateLogSynchronizer stopped while replica's on source cluster shows updateLogSynchronizer started. How can we start updateLogSynchronizer on leader and stop updateLogSynchronizer on replica on source without switching the leader/follower. Any idea? Thnx On Mon, Apr 16, 2018 at 2:20 PM, Tom Peters <tpet...@synacor.com> wrote: > There are two ways I've gotten around this issue: > > 1. Add replicas in the target data center after CDCR bootstrapping has > completed. > > -or- > > 2. After the bootstrapping has completed, restart the replica nodes > one-at-time in the target data center (restart, wait for replica to catch > up, then restart the next). > > > I recommend doing method #1 over #2 if you can. If you accidentally > restart the leader node using method #2, it will promote an out-of-sync > replica to the leader and all followers will receive that out-of-date index. > > I also recommend pausing indexing if you can while you let the target > replicas catch up. I have run into issues where the replicas will not catch > up if the leader has a fair amount of updates to replay from the source. > > > On Apr 16, 2018, at 2:15 PM, Amrit Sarkar <sarkaramr...@gmail.com> > wrote: > > > > Hi Susheel, > > > > Pretty sure you are talking about this: > > https://issues.apache.org/jira/browse/SOLR-11724 > > > > Amrit Sarkar > > Search Engineer > > Lucidworks, Inc. > > 415-589-9269 > > www.lucidworks.com > > Twitter http://twitter.com/lucidworks > > LinkedIn: https://www.linkedin.com/in/sarkaramrit2 > > Medium: https://medium.com/@sarkaramrit2 > > > > On Mon, Apr 16, 2018 at 11:35 PM, Susheel Kumar <susheel2...@gmail.com> > > wrote: > > > >> Does anybody know about known issue where CDCR bootstrap sync leaves the > >> replica's on target cluster non touched/out of sync. > >> > >> After I stopped and restart CDCR, it builds my target leaders index but > >> replica's on target cluster still showing old index / not modified. > >> > >> > >> Thnx > >> > > > > This message and any attachment may contain information that is > confidential and/or proprietary. Any use, disclosure, copying, storing, or > distribution of this e-mail or any attached file by anyone other than the > intended recipient is strictly prohibited. If you have received this > message in error, please notify the sender by reply email and delete the > message and any attachments. Thank you. >
Re: Does CDCR Bootstrap sync leaves replica's out of sync
There are two ways I've gotten around this issue: 1. Add replicas in the target data center after CDCR bootstrapping has completed. -or- 2. After the bootstrapping has completed, restart the replica nodes one-at-time in the target data center (restart, wait for replica to catch up, then restart the next). I recommend doing method #1 over #2 if you can. If you accidentally restart the leader node using method #2, it will promote an out-of-sync replica to the leader and all followers will receive that out-of-date index. I also recommend pausing indexing if you can while you let the target replicas catch up. I have run into issues where the replicas will not catch up if the leader has a fair amount of updates to replay from the source. > On Apr 16, 2018, at 2:15 PM, Amrit Sarkar <sarkaramr...@gmail.com> wrote: > > Hi Susheel, > > Pretty sure you are talking about this: > https://issues.apache.org/jira/browse/SOLR-11724 > > Amrit Sarkar > Search Engineer > Lucidworks, Inc. > 415-589-9269 > www.lucidworks.com > Twitter http://twitter.com/lucidworks > LinkedIn: https://www.linkedin.com/in/sarkaramrit2 > Medium: https://medium.com/@sarkaramrit2 > > On Mon, Apr 16, 2018 at 11:35 PM, Susheel Kumar <susheel2...@gmail.com> > wrote: > >> Does anybody know about known issue where CDCR bootstrap sync leaves the >> replica's on target cluster non touched/out of sync. >> >> After I stopped and restart CDCR, it builds my target leaders index but >> replica's on target cluster still showing old index / not modified. >> >> >> Thnx >> This message and any attachment may contain information that is confidential and/or proprietary. Any use, disclosure, copying, storing, or distribution of this e-mail or any attached file by anyone other than the intended recipient is strictly prohibited. If you have received this message in error, please notify the sender by reply email and delete the message and any attachments. Thank you.
Re: Does CDCR Bootstrap sync leaves replica's out of sync
Hi Susheel, Pretty sure you are talking about this: https://issues.apache.org/jira/browse/SOLR-11724 Amrit Sarkar Search Engineer Lucidworks, Inc. 415-589-9269 www.lucidworks.com Twitter http://twitter.com/lucidworks LinkedIn: https://www.linkedin.com/in/sarkaramrit2 Medium: https://medium.com/@sarkaramrit2 On Mon, Apr 16, 2018 at 11:35 PM, Susheel Kumar <susheel2...@gmail.com> wrote: > Does anybody know about known issue where CDCR bootstrap sync leaves the > replica's on target cluster non touched/out of sync. > > After I stopped and restart CDCR, it builds my target leaders index but > replica's on target cluster still showing old index / not modified. > > > Thnx >
Does CDCR Bootstrap sync leaves replica's out of sync
Does anybody know about known issue where CDCR bootstrap sync leaves the replica's on target cluster non touched/out of sync. After I stopped and restart CDCR, it builds my target leaders index but replica's on target cluster still showing old index / not modified. Thnx
Re: cdcr bootstrap errors
restarting the zookeeper on the source cloud seems to have helped On Tue, Jul 4, 2017 at 3:42 PM, Webster Homer <webster.ho...@sial.com> wrote: > Another strange error message I'm seeing > 2017-07-04 18:59:40.585 WARN (cdcr-replicator-110-thread- > 4-processing-n:dfw-pauth-msc02:8983_solr) [ ] o.a.s.h.CdcrReplicator > Failed to forward update request to target: sial-catalog-product > org.apache.solr.common.SolrException: Could not load collection from ZK: > sial-catalog-product > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive( > ZkStateReader.java:1093) > at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get( > ZkStateReader.java:638) > at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull( > ClusterState.java:212) > at org.apache.solr.common.cloud.ClusterState.hasCollection( > ClusterState.java:114) > at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames( > CloudSolrClient.java:1302) > at org.apache.solr.client.solrj.impl.CloudSolrClient. > requestWithRetryOnStaleState(CloudSolrClient.java:1024) > at org.apache.solr.client.solrj.impl.CloudSolrClient.request( > CloudSolrClient.java:997) > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149) > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166) > at org.apache.solr.handler.CdcrReplicator.sendRequest( > CdcrReplicator.java:135) > at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:115) > at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0( > CdcrReplicatorScheduler.java:81) > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor. > lambda$execute$0(ExecutorUtil.java:229) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /collections/sial-catalog- > product/state.json > > So is Zookeeper hosed? How do I tell? > > On Tue, Jul 4, 2017 at 3:27 PM, Webster Homer <webster.ho...@sial.com> > wrote: > >> We've been using cdcr for a while now. It seems to be pretty fragile. >> >> Currently we're seeing tons of errors like this: >> 2017-07-04 14:41:27.015 ERROR (cdcr-bootstrap-status-51-thre >> ad-1-processing-n:dfw-pauth-msc02:8983_solr) [ ] >> o.a.s.h.CdcrReplicatorManager Exception during bootstrap status request >> >> In this case we have one server throwing the above errors a lot! >> >> The error isn't very informative what can cause this? >> >> I also see these messages: >> 2017-07-04 18:59:39.730 WARN (cdcr-replicator-122-thread-3 >> -processing-n:dfw-pauth-msc02:8983_solr x:sial-catalog-gene_shard1_replica1 >> s:shard1 c:sial-catalog-gene r:core_node1) [c:sial-catalog-gene s:shard1 >> r:core_node1 x:sial-catalog-gene_shard1_replica1] o.a.s.h.CdcrReplicator >> Log reader for target sial-catalog-gene is not initialised, it will be >> ignored. >> >> 2017-07-04 18:59:39.730 INFO (cdcr-replicator-122-thread-1- >> processing-n:dfw-pauth-msc02:8983_solr x:sial-catalog-gene_shard1_replica1 >> s:shard1 c:sial-catalog-gene r:core_node1) [c:sial-catalog-gene s:shard1 >> r:core_node1 x:sial-catalog-gene_shard1_replica1] o.a.s.h.CdcrReplicator >> Forwarded 0 updates to target sial-catalog-gene 2017-07-04 18:59:39.975 >> WARN (cdcr-replicator-100-thread-3-processing-n:dfw-pauth-msc02:8983_solr) >> [ ] o.a.s.h.CdcrReplicator Failed to forward update request to target: >> bb-catalog-material java.lang.RuntimeException: Unknown type 17 >> >> We are using Solr 6.2 >> We have a 2 node cloud with multiple collections all with 2 shards >> replicating to two solr clouds running in Google cloud. >> We noticed that some of the prod collections only had data in one of the >> shards. >> >> So how do we diagnose this issue? >> >> > -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
Re: cdcr bootstrap errors
Another strange error message I'm seeing 2017-07-04 18:59:40.585 WARN (cdcr-replicator-110-thread-4-processing-n:dfw-pauth-msc02:8983_solr) [ ] o.a.s.h.CdcrReplicator Failed to forward update request to target: sial-catalog-product org.apache.solr.common.SolrException: Could not load collection from ZK: sial-catalog-product at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1093) at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:638) at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:212) at org.apache.solr.common.cloud.ClusterState.hasCollection(ClusterState.java:114) at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1302) at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1024) at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166) at org.apache.solr.handler.CdcrReplicator.sendRequest(CdcrReplicator.java:135) at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:115) at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/sial-catalog-product/state.json So is Zookeeper hosed? How do I tell? On Tue, Jul 4, 2017 at 3:27 PM, Webster Homer <webster.ho...@sial.com> wrote: > We've been using cdcr for a while now. It seems to be pretty fragile. > > Currently we're seeing tons of errors like this: > 2017-07-04 14:41:27.015 ERROR (cdcr-bootstrap-status-51- > thread-1-processing-n:dfw-pauth-msc02:8983_solr) [ ] > o.a.s.h.CdcrReplicatorManager Exception during bootstrap status request > > In this case we have one server throwing the above errors a lot! > > The error isn't very informative what can cause this? > > I also see these messages: > 2017-07-04 18:59:39.730 WARN (cdcr-replicator-122-thread- > 3-processing-n:dfw-pauth-msc02:8983_solr x:sial-catalog-gene_shard1_replica1 > s:shard1 c:sial-catalog-gene r:core_node1) [c:sial-catalog-gene s:shard1 > r:core_node1 x:sial-catalog-gene_shard1_replica1] o.a.s.h.CdcrReplicator > Log reader for target sial-catalog-gene is not initialised, it will be > ignored. > > 2017-07-04 18:59:39.730 INFO (cdcr-replicator-122-thread-1- > processing-n:dfw-pauth-msc02:8983_solr x:sial-catalog-gene_shard1_replica1 > s:shard1 c:sial-catalog-gene r:core_node1) [c:sial-catalog-gene s:shard1 > r:core_node1 x:sial-catalog-gene_shard1_replica1] o.a.s.h.CdcrReplicator > Forwarded 0 updates to target sial-catalog-gene 2017-07-04 18:59:39.975 > WARN (cdcr-replicator-100-thread-3-processing-n:dfw-pauth-msc02:8983_solr) > [ ] o.a.s.h.CdcrReplicator Failed to forward update request to target: > bb-catalog-material java.lang.RuntimeException: Unknown type 17 > > We are using Solr 6.2 > We have a 2 node cloud with multiple collections all with 2 shards > replicating to two solr clouds running in Google cloud. > We noticed that some of the prod collections only had data in one of the > shards. > > So how do we diagnose this issue? > > -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
cdcr bootstrap errors
We've been using cdcr for a while now. It seems to be pretty fragile. Currently we're seeing tons of errors like this: 2017-07-04 14:41:27.015 ERROR (cdcr-bootstrap-status-51-thread-1-processing-n:dfw-pauth-msc02:8983_solr) [ ] o.a.s.h.CdcrReplicatorManager Exception during bootstrap status request In this case we have one server throwing the above errors a lot! The error isn't very informative what can cause this? I also see these messages: 2017-07-04 18:59:39.730 WARN (cdcr-replicator-122-thread-3-processing-n:dfw-pauth-msc02:8983_solr x:sial-catalog-gene_shard1_replica1 s:shard1 c:sial-catalog-gene r:core_node1) [c:sial-catalog-gene s:shard1 r:core_node1 x:sial-catalog-gene_shard1_replica1] o.a.s.h.CdcrReplicator Log reader for target sial-catalog-gene is not initialised, it will be ignored. 2017-07-04 18:59:39.730 INFO (cdcr-replicator-122-thread-1-processing-n:dfw-pauth-msc02:8983_solr x:sial-catalog-gene_shard1_replica1 s:shard1 c:sial-catalog-gene r:core_node1) [c:sial-catalog-gene s:shard1 r:core_node1 x:sial-catalog-gene_shard1_replica1] o.a.s.h.CdcrReplicator Forwarded 0 updates to target sial-catalog-gene 2017-07-04 18:59:39.975 WARN (cdcr-replicator-100-thread-3-processing-n:dfw-pauth-msc02:8983_solr) [ ] o.a.s.h.CdcrReplicator Failed to forward update request to target: bb-catalog-material java.lang.RuntimeException: Unknown type 17 We are using Solr 6.2 We have a 2 node cloud with multiple collections all with 2 shards replicating to two solr clouds running in Google cloud. We noticed that some of the prod collections only had data in one of the shards. So how do we diagnose this issue? -- This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient, you must not copy this message or attachment or disclose the contents to any other person. If you have received this transmission in error, please notify the sender immediately and delete the message and any attachment from your system. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not accept liability for any omissions or errors in this message which may arise as a result of E-Mail-transmission or for damages resulting from any unauthorized changes of the content of this message and any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its subsidiaries do not guarantee that this message is free of viruses and does not accept liability for any damages caused by any virus transmitted therewith. Click http://www.emdgroup.com/disclaimer to access the German, French, Spanish and Portuguese versions of this disclaimer.
