Re: Push/pull model between leader and replica in one shard
Hej Mark; What did you use to prepare your presentation, its really nice. 2013/4/17 Furkan KAMACI furkankam...@gmail.com Really nice presentation. 2013/4/17 Mark Miller markrmil...@gmail.com On Apr 16, 2013, at 1:36 AM, SuoNayi suonayi2...@163.com wrote: Hi, can someone explain more details about what model is used to sync docs between the lead and replica in the shard? The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas, when the leader receives a update request, does it will scatter the request to each available and active replica at first and then processes the request locally at last?In this case if the replicas are able to catch up with the leader can I think this is a push model that the leader pushes updates to it's replicas? Currently, the leader adds the doc locally and then sends it to all replicas concurrently. What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs it will replay updates in the tlog at last? If an update forwarded from a leader to a replica fails it's likely because that replica died. Just in case, the leader will ask that replica to enter recovery. When a node comes up and is not a leader, it also enters recovery. Recovery tries to peersync from the leader, and if that fails (works if off by about 100 updates), it replicates the entire index. If you are interested in more details on the SolrCloud architecture, I've given a few talks on it - two of them here: http://vimeo.com/43913870 http://www.youtube.com/watch?v=eVK0wLkLw9w - Mark
Re: Push/pull model between leader and replica in one shard
Thanks, the earlier presentation is done with KeyNote and the later (more animation) is done with Tumult Hype. - Mark On Apr 17, 2013, at 3:43 AM, Furkan KAMACI furkankam...@gmail.com wrote: Hej Mark; What did you use to prepare your presentation, its really nice. 2013/4/17 Furkan KAMACI furkankam...@gmail.com Really nice presentation. 2013/4/17 Mark Miller markrmil...@gmail.com On Apr 16, 2013, at 1:36 AM, SuoNayi suonayi2...@163.com wrote: Hi, can someone explain more details about what model is used to sync docs between the lead and replica in the shard? The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas, when the leader receives a update request, does it will scatter the request to each available and active replica at first and then processes the request locally at last?In this case if the replicas are able to catch up with the leader can I think this is a push model that the leader pushes updates to it's replicas? Currently, the leader adds the doc locally and then sends it to all replicas concurrently. What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs it will replay updates in the tlog at last? If an update forwarded from a leader to a replica fails it's likely because that replica died. Just in case, the leader will ask that replica to enter recovery. When a node comes up and is not a leader, it also enters recovery. Recovery tries to peersync from the leader, and if that fails (works if off by about 100 updates), it replicates the entire index. If you are interested in more details on the SolrCloud architecture, I've given a few talks on it - two of them here: http://vimeo.com/43913870 http://www.youtube.com/watch?v=eVK0wLkLw9w - Mark
Re: Push/pull model between leader and replica in one shard
Hi, Replication when everything is working well is push: * request comes to any node, ideally leader * doc is indexed on leader * doc is copied to replicas If replica falls too far behind (not exactly sure what the too far threshold is), it uses pull to replicate the whole index from leader. Mark can answer the part about where tlog gets replayed to catch up on docs that were missed while big index replication pull was happening. This is a good thread to read on this topic: http://search-lucene.com/m/y1yj218J2v82 Otis -- Solr ElasticSearch Support http://sematext.com/ On Tue, Apr 16, 2013 at 1:36 AM, SuoNayi suonayi2...@163.com wrote: Hi, can someone explain more details about what model is used to sync docs between the lead and replica in the shard? The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas, when the leader receives a update request, does it will scatter the request to each available and active replica at first and then processes the request locally at last?In this case if the replicas are able to catch up with the leader can I think this is a push model that the leader pushes updates to it's replicas? What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs it will replay updates in the tlog at last? regards
Re: Push/pull model between leader and replica in one shard
On Apr 16, 2013, at 1:36 AM, SuoNayi suonayi2...@163.com wrote: Hi, can someone explain more details about what model is used to sync docs between the lead and replica in the shard? The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas, when the leader receives a update request, does it will scatter the request to each available and active replica at first and then processes the request locally at last?In this case if the replicas are able to catch up with the leader can I think this is a push model that the leader pushes updates to it's replicas? Currently, the leader adds the doc locally and then sends it to all replicas concurrently. What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs it will replay updates in the tlog at last? If an update forwarded from a leader to a replica fails it's likely because that replica died. Just in case, the leader will ask that replica to enter recovery. When a node comes up and is not a leader, it also enters recovery. Recovery tries to peersync from the leader, and if that fails (works if off by about 100 updates), it replicates the entire index. If you are interested in more details on the SolrCloud architecture, I've given a few talks on it - two of them here: http://vimeo.com/43913870 http://www.youtube.com/watch?v=eVK0wLkLw9w - Mark
Re: Push/pull model between leader and replica in one shard
Really nice presentation. 2013/4/17 Mark Miller markrmil...@gmail.com On Apr 16, 2013, at 1:36 AM, SuoNayi suonayi2...@163.com wrote: Hi, can someone explain more details about what model is used to sync docs between the lead and replica in the shard? The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas, when the leader receives a update request, does it will scatter the request to each available and active replica at first and then processes the request locally at last?In this case if the replicas are able to catch up with the leader can I think this is a push model that the leader pushes updates to it's replicas? Currently, the leader adds the doc locally and then sends it to all replicas concurrently. What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs it will replay updates in the tlog at last? If an update forwarded from a leader to a replica fails it's likely because that replica died. Just in case, the leader will ask that replica to enter recovery. When a node comes up and is not a leader, it also enters recovery. Recovery tries to peersync from the leader, and if that fails (works if off by about 100 updates), it replicates the entire index. If you are interested in more details on the SolrCloud architecture, I've given a few talks on it - two of them here: http://vimeo.com/43913870 http://www.youtube.com/watch?v=eVK0wLkLw9w - Mark
Push/pull model between leader and replica in one shard
Hi, can someone explain more details about what model is used to sync docs between the lead and replica in the shard? The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas, when the leader receives a update request, does it will scatter the request to each available and active replica at first and then processes the request locally at last?In this case if the replicas are able to catch up with the leader can I think this is a push model that the leader pushes updates to it's replicas? What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs it will replay updates in the tlog at last? regards
