Re: SolrCloud replicas out of sync
Tomás, Good find, but I don’t think the rate of updates was high enough during the network outage to create the overrun situation described in the ticket. I did notice that one of the proposed fixes, https://issues.apache.org/jira/browse/SOLR-8586, is an entire-index consistency check between leader and replica. I really hope they are able to get this to work. Ideally, the replicas would never become (permanently) inconsistent, but given that they do, it is crucial that SolrCloud can internally detect and fix, no matter what caused it or how long ago it happened. Regards, David On 1/28/16, 1:08 PM, "Tomás Fernández Löbbe" wrote: >Maybe you are hitting the reordering issue described in SOLR-8129? > >Tomás > >On Wed, Jan 27, 2016 at 11:32 AM, David Smith >wrote: > >> Sure. Here is our SolrCloud cluster: >> >>+ Three (3) instances of Zookeeper on three separate (physical) >> servers. The ZK servers are beefy and fairly recently built, with 2x10 >> GigE (bonded) Ethernet connectivity to the rest of the data center. We >> recognize importance of the stability and responsiveness of ZK to the >> stability of SolrCloud as a whole. >> >>+ 364 collections, all with single shards and a replication factor of >> 3. Currently housing only 100,000,000 documents in aggregate. Expected to >> grow to 25 billion+. The size of a single document would be considered >> “large”, by the standards of what I’ve seen posted elsewhere on this >> mailing list. >> >> We are always open to ZK recommendations from you or anyone else, >> particularly for running a SolrCloud cluster of this size. >> >> Kind Regards, >> >> David >> >> >> >> On 1/27/16, 12:46 PM, "Jeff Wartes" wrote: >> >> > >> >If you can identify the problem documents, you can just re-index those >> after forcing a sync. Might save a full rebuild and downtime. >> > >> >You might describe your cluster setup, including ZK. it sounds like >> you’ve done your research, but improper ZK node distribution could >> certainly invalidate some of Solr’s assumptions. >> > >> > >> > >> > >> >On 1/27/16, 7:59 AM, "David Smith" wrote: >> > >> >>Jeff, again, very much appreciate your feedback. >> >> >> >>It is interesting — the article you linked to by Shalin is exactly why >> we picked SolrCloud over ES, because (eventual) consistency is critical for >> our application and we will sacrifice availability for it. To be clear, >> after the outage, NONE of our three replicas are correct or complete. >> >> >> >>So we definitely don’t have CP yet — our very first network outage >> resulted in multiple overlapped lost updates. As a result, I can’t pick >> one replica and make it the new “master”. I must rebuild this collection >> from scratch, which I can do, but that requires downtime which is a problem >> in our app (24/7 High Availability with few maintenance windows). >> >> >> >> >> >>So, I definitely need to “fix” this somehow. I wish I could outline a >> reproducible test case, but as the root cause is likely very tight timing >> issues and complicated interactions with Zookeeper, that is not really an >> option. I’m happy to share the full logs of all 3 replicas though if that >> helps. >> >> >> >>I am curious though if the thoughts have changed since >> https://issues.apache.org/jira/browse/SOLR-5468 of seriously considering >> a “majority quorum” model, with rollback? Done properly, this should be >> free of all lost update problems, at the cost of availability. Some >> SolrCloud users (like us!!!) would gladly accept that tradeoff. >> >> >> >>Regards >> >> >> >>David >> >> >> >> >> >>
Re: SolrCloud replicas out of sync
Maybe you are hitting the reordering issue described in SOLR-8129? Tomás On Wed, Jan 27, 2016 at 11:32 AM, David Smith wrote: > Sure. Here is our SolrCloud cluster: > >+ Three (3) instances of Zookeeper on three separate (physical) > servers. The ZK servers are beefy and fairly recently built, with 2x10 > GigE (bonded) Ethernet connectivity to the rest of the data center. We > recognize importance of the stability and responsiveness of ZK to the > stability of SolrCloud as a whole. > >+ 364 collections, all with single shards and a replication factor of > 3. Currently housing only 100,000,000 documents in aggregate. Expected to > grow to 25 billion+. The size of a single document would be considered > “large”, by the standards of what I’ve seen posted elsewhere on this > mailing list. > > We are always open to ZK recommendations from you or anyone else, > particularly for running a SolrCloud cluster of this size. > > Kind Regards, > > David > > > > On 1/27/16, 12:46 PM, "Jeff Wartes" wrote: > > > > >If you can identify the problem documents, you can just re-index those > after forcing a sync. Might save a full rebuild and downtime. > > > >You might describe your cluster setup, including ZK. it sounds like > you’ve done your research, but improper ZK node distribution could > certainly invalidate some of Solr’s assumptions. > > > > > > > > > >On 1/27/16, 7:59 AM, "David Smith" wrote: > > > >>Jeff, again, very much appreciate your feedback. > >> > >>It is interesting — the article you linked to by Shalin is exactly why > we picked SolrCloud over ES, because (eventual) consistency is critical for > our application and we will sacrifice availability for it. To be clear, > after the outage, NONE of our three replicas are correct or complete. > >> > >>So we definitely don’t have CP yet — our very first network outage > resulted in multiple overlapped lost updates. As a result, I can’t pick > one replica and make it the new “master”. I must rebuild this collection > from scratch, which I can do, but that requires downtime which is a problem > in our app (24/7 High Availability with few maintenance windows). > >> > >> > >>So, I definitely need to “fix” this somehow. I wish I could outline a > reproducible test case, but as the root cause is likely very tight timing > issues and complicated interactions with Zookeeper, that is not really an > option. I’m happy to share the full logs of all 3 replicas though if that > helps. > >> > >>I am curious though if the thoughts have changed since > https://issues.apache.org/jira/browse/SOLR-5468 of seriously considering > a “majority quorum” model, with rollback? Done properly, this should be > free of all lost update problems, at the cost of availability. Some > SolrCloud users (like us!!!) would gladly accept that tradeoff. > >> > >>Regards > >> > >>David > >> > >> > >
Re: SolrCloud replicas out of sync
Sure. Here is our SolrCloud cluster: + Three (3) instances of Zookeeper on three separate (physical) servers. The ZK servers are beefy and fairly recently built, with 2x10 GigE (bonded) Ethernet connectivity to the rest of the data center. We recognize importance of the stability and responsiveness of ZK to the stability of SolrCloud as a whole. + 364 collections, all with single shards and a replication factor of 3. Currently housing only 100,000,000 documents in aggregate. Expected to grow to 25 billion+. The size of a single document would be considered “large”, by the standards of what I’ve seen posted elsewhere on this mailing list. We are always open to ZK recommendations from you or anyone else, particularly for running a SolrCloud cluster of this size. Kind Regards, David On 1/27/16, 12:46 PM, "Jeff Wartes" wrote: > >If you can identify the problem documents, you can just re-index those after >forcing a sync. Might save a full rebuild and downtime. > >You might describe your cluster setup, including ZK. it sounds like you’ve >done your research, but improper ZK node distribution could certainly >invalidate some of Solr’s assumptions. > > > > >On 1/27/16, 7:59 AM, "David Smith" wrote: > >>Jeff, again, very much appreciate your feedback. >> >>It is interesting — the article you linked to by Shalin is exactly why we >>picked SolrCloud over ES, because (eventual) consistency is critical for our >>application and we will sacrifice availability for it. To be clear, after >>the outage, NONE of our three replicas are correct or complete. >> >>So we definitely don’t have CP yet — our very first network outage resulted >>in multiple overlapped lost updates. As a result, I can’t pick one replica >>and make it the new “master”. I must rebuild this collection from scratch, >>which I can do, but that requires downtime which is a problem in our app >>(24/7 High Availability with few maintenance windows). >> >> >>So, I definitely need to “fix” this somehow. I wish I could outline a >>reproducible test case, but as the root cause is likely very tight timing >>issues and complicated interactions with Zookeeper, that is not really an >>option. I’m happy to share the full logs of all 3 replicas though if that >>helps. >> >>I am curious though if the thoughts have changed since >>https://issues.apache.org/jira/browse/SOLR-5468 of seriously considering a >>“majority quorum” model, with rollback? Done properly, this should be free >>of all lost update problems, at the cost of availability. Some SolrCloud >>users (like us!!!) would gladly accept that tradeoff. >> >>Regards >> >>David >> >>
Re: SolrCloud replicas out of sync
If you can identify the problem documents, you can just re-index those after forcing a sync. Might save a full rebuild and downtime. You might describe your cluster setup, including ZK. it sounds like you’ve done your research, but improper ZK node distribution could certainly invalidate some of Solr’s assumptions. On 1/27/16, 7:59 AM, "David Smith" wrote: >Jeff, again, very much appreciate your feedback. > >It is interesting — the article you linked to by Shalin is exactly why we >picked SolrCloud over ES, because (eventual) consistency is critical for our >application and we will sacrifice availability for it. To be clear, after the >outage, NONE of our three replicas are correct or complete. > >So we definitely don’t have CP yet — our very first network outage resulted in >multiple overlapped lost updates. As a result, I can’t pick one replica and >make it the new “master”. I must rebuild this collection from scratch, which >I can do, but that requires downtime which is a problem in our app (24/7 High >Availability with few maintenance windows). > > >So, I definitely need to “fix” this somehow. I wish I could outline a >reproducible test case, but as the root cause is likely very tight timing >issues and complicated interactions with Zookeeper, that is not really an >option. I’m happy to share the full logs of all 3 replicas though if that >helps. > >I am curious though if the thoughts have changed since >https://issues.apache.org/jira/browse/SOLR-5468 of seriously considering a >“majority quorum” model, with rollback? Done properly, this should be free of >all lost update problems, at the cost of availability. Some SolrCloud users >(like us!!!) would gladly accept that tradeoff. > >Regards > >David > >
Re: SolrCloud replicas out of sync
On 1/27/16, 8:28 AM, "Shawn Heisey" wrote: > >I don't think any documentation states this, but it seems like a good >idea to me use an alias from day one, so that you always have the option >of swapping the "real" collection that you are using without needing to >change anything else. I'll need to ask some people if they think this >is a good documentation addition, and think of a good place to mention >it in the reference guide. +1 - I recommend this at every opportunity. I’ve even considered creating 10 aliases for a single collection and having the client select one of the aliases randomly per query. This would allow transparently shifting traffic between collections in 10% increments.
Re: SolrCloud replicas out of sync
On 1/27/2016 8:59 AM, David Smith wrote: > So we definitely don’t have CP yet — our very first network outage resulted > in multiple overlapped lost updates. As a result, I can’t pick one replica > and make it the new “master”. I must rebuild this collection from scratch, > which I can do, but that requires downtime which is a problem in our app > (24/7 High Availability with few maintenance windows). I don't have anything to add regarding the initial problem or how you can prevent it from happening again. You're already on the latest minor Solr version (5.4), though I think you could probably upgrade to 5.4.1. The list of bugfixes for 5.4.1 does not seem to include anything that could explain the problem you have described. There is SOLR-8496, but that only affects multi-select faceting, not the numDoc counts on the core summary screen. For the specific issue mentioned above (rebuilding without downtime), what I would recommend that you do is build an entirely new collection, then delete the old one and create an alias to the new collection using the old collection's name. Deleting the old collection and making the alias should happen very quickly. I don't think any documentation states this, but it seems like a good idea to me use an alias from day one, so that you always have the option of swapping the "real" collection that you are using without needing to change anything else. I'll need to ask some people if they think this is a good documentation addition, and think of a good place to mention it in the reference guide. Thanks, Shawn
Re: SolrCloud replicas out of sync
Jeff, again, very much appreciate your feedback. It is interesting — the article you linked to by Shalin is exactly why we picked SolrCloud over ES, because (eventual) consistency is critical for our application and we will sacrifice availability for it. To be clear, after the outage, NONE of our three replicas are correct or complete. So we definitely don’t have CP yet — our very first network outage resulted in multiple overlapped lost updates. As a result, I can’t pick one replica and make it the new “master”. I must rebuild this collection from scratch, which I can do, but that requires downtime which is a problem in our app (24/7 High Availability with few maintenance windows). So, I definitely need to “fix” this somehow. I wish I could outline a reproducible test case, but as the root cause is likely very tight timing issues and complicated interactions with Zookeeper, that is not really an option. I’m happy to share the full logs of all 3 replicas though if that helps. I am curious though if the thoughts have changed since https://issues.apache.org/jira/browse/SOLR-5468 of seriously considering a “majority quorum” model, with rollback? Done properly, this should be free of all lost update problems, at the cost of availability. Some SolrCloud users (like us!!!) would gladly accept that tradeoff. Regards David On 1/26/16, 4:32 PM, "Jeff Wartes" wrote: > >Ah, perhaps you fell into something like this then? >https://issues.apache.org/jira/browse/SOLR-7844 > >That says it’s fixed in 5.4, but that would be an example of a split-brain >type incident, where different documents were accepted by different replicas >who each thought they were the leader. If this is the case, and you actually >have different data on each replica, I’m not aware of any way to fix the >problem short of reindexing those documents. Before that, you’ll probably need >to choose a replica and just force the others to get in sync with it. I’d >choose the current leader, since that’s slightly easier. > >Typically, a leader writes an update to it’s transaction log, then sends the >request to all replicas, and when those all finish it acknowledges the update. >If a replica gets restarted, and is less than N documents behind, the leader >will only replay that transaction log. (Where N is the numRecordsToKeep >configured in the updateLog section of solrconfig.xml) > >What you want is to provoke the heavy-duty process normally invoked if a >replica has missed more than N docs, which essentially does a checksum and >file copy on all the raw index files. FetchIndex would probably work, but it’s >a replication handler API originally designed for master/slave replication, so >take care: https://wiki.apache.org/solr/SolrReplication#HTTP_API >Probably a lot easier would be to just delete the replica and re-create it. >That will also trigger a full file copy of the index from the leader onto the >new replica. > >I think design decisions around Solr generally use CP as a goal. (I sometimes >wish I could get more AP behavior!) See posts like this: >http://lucidworks.com/blog/2014/12/10/call-maybe-solrcloud-jepsen-flaky-networks/ > >So the fact that you encountered this sounds like a bug to me. >That said, another general recommendation (of mine) is that you not use Solr >as your primary data source, so you can rebuild your index from scratch if you >really need to. > > > > > > >On 1/26/16, 1:10 PM, "David Smith" wrote: > >>Thanks Jeff! A few comments >> Although you could probably bounce a node and get your document counts back in sync (by provoking a check) >> >> >>If the check is a simple doc count, that will not work. We have found that >>replica1 and replica3, although they contain the same doc count, don’t have >>the SAME docs. They each missed at least one update, but of different docs. >>This also means none of our three replicas are complete. >> it’s interesting that you’re in this situation. It implies to me that at some point the leader couldn’t write a doc to one of the replicas, >> >>That is our belief as well. We experienced a datacenter-wide network >>disruption of a few seconds, and user complaints started the first workday >>after that event. >> >>The most interesting log entry during the outage is this: >> >>"1/19/2016, 5:08:07 PM ERROR null DistributedUpdateProcessorRequest says it >>is coming from leader, but we are the leader: >>update.distrib=FROMLEADER&distrib.from=http://dot.dot.dot.dot:8983/solr/blah_blah_shard1_replica3/&wt=javabin&version=2"; >> You might watch the achieved replication factor of your updates and see if it ever changes >> >>This is a good tip. I’m not sure I like the implication that any failure to >>write all 3 of our replicas must be retried at the app layer. Is this really >>how SolrCloud applications must be built to survive network partitions >>without data loss? >>
Re: SolrCloud replicas out of sync
This on the surface appears to be similar to an earlier thread by me: "Query results change" On Tue, Jan 26, 2016 at 4:32 PM, Jeff Wartes wrote: > > Ah, perhaps you fell into something like this then? > https://issues.apache.org/jira/browse/SOLR-7844 > > That says it’s fixed in 5.4, but that would be an example of a split-brain > type incident, where different documents were accepted by different > replicas who each thought they were the leader. If this is the case, and > you actually have different data on each replica, I’m not aware of any way > to fix the problem short of reindexing those documents. Before that, you’ll > probably need to choose a replica and just force the others to get in sync > with it. I’d choose the current leader, since that’s slightly easier. > > Typically, a leader writes an update to it’s transaction log, then sends > the request to all replicas, and when those all finish it acknowledges the > update. If a replica gets restarted, and is less than N documents behind, > the leader will only replay that transaction log. (Where N is the > numRecordsToKeep configured in the updateLog section of solrconfig.xml) > > What you want is to provoke the heavy-duty process normally invoked if a > replica has missed more than N docs, which essentially does a checksum and > file copy on all the raw index files. FetchIndex would probably work, but > it’s a replication handler API originally designed for master/slave > replication, so take care: > https://wiki.apache.org/solr/SolrReplication#HTTP_API > Probably a lot easier would be to just delete the replica and re-create > it. That will also trigger a full file copy of the index from the leader > onto the new replica. > > I think design decisions around Solr generally use CP as a goal. (I > sometimes wish I could get more AP behavior!) See posts like this: > http://lucidworks.com/blog/2014/12/10/call-maybe-solrcloud-jepsen-flaky-networks/ > So the fact that you encountered this sounds like a bug to me. > That said, another general recommendation (of mine) is that you not use > Solr as your primary data source, so you can rebuild your index from > scratch if you really need to. > > > > > > > On 1/26/16, 1:10 PM, "David Smith" wrote: > > >Thanks Jeff! A few comments > > > >>> > >>> Although you could probably bounce a node and get your document counts > back in sync (by provoking a check) > >>> > > > > > >If the check is a simple doc count, that will not work. We have found > that replica1 and replica3, although they contain the same doc count, don’t > have the SAME docs. They each missed at least one update, but of different > docs. This also means none of our three replicas are complete. > > > >>> > >>>it’s interesting that you’re in this situation. It implies to me that > at some point the leader couldn’t write a doc to one of the replicas, > >>> > > > >That is our belief as well. We experienced a datacenter-wide network > disruption of a few seconds, and user complaints started the first workday > after that event. > > > >The most interesting log entry during the outage is this: > > > >"1/19/2016, 5:08:07 PM ERROR null DistributedUpdateProcessorRequest says > it is coming from leader, but we are the leader: > update.distrib=FROMLEADER&distrib.from= > http://dot.dot.dot.dot:8983/solr/blah_blah_shard1_replica3/&wt=javabin&version=2 > " > > > >>> > >>> You might watch the achieved replication factor of your updates and > see if it ever changes > >>> > > > >This is a good tip. I’m not sure I like the implication that any failure > to write all 3 of our replicas must be retried at the app layer. Is this > really how SolrCloud applications must be built to survive network > partitions without data loss? > > > >Regards, > > > >David > > > > > >On 1/26/16, 12:20 PM, "Jeff Wartes" wrote: > > > >> > >>My understanding is that the "version" represents the timestamp the > searcher was opened, so it doesn’t really offer any assurances about your > data. > >> > >>Although you could probably bounce a node and get your document counts > back in sync (by provoking a check), it’s interesting that you’re in this > situation. It implies to me that at some point the leader couldn’t write a > doc to one of the replicas, but that the replica didn’t consider itself > down enough to check itself. > >> > >>You might watch the achieved replication factor of your updates and see > if it ever changes: > >> > https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance > (See Achieved Replication Factor/min_rf) > >> > >>If it does, that might give you clues about how this is happening. Also, > it might allow you to work around the issue by trying the write again. > >> > >> > >> > >> > >> > >> > >>On 1/22/16, 10:52 AM, "David Smith" > wrote: > >> > >>>I have a SolrCloud v5.4 collection with 3 replicas that appear to have > fallen permanently out of sync. Users started to complain that the same > search, executed twice, sometimes returned
Re: SolrCloud replicas out of sync
Ah, perhaps you fell into something like this then? https://issues.apache.org/jira/browse/SOLR-7844 That says it’s fixed in 5.4, but that would be an example of a split-brain type incident, where different documents were accepted by different replicas who each thought they were the leader. If this is the case, and you actually have different data on each replica, I’m not aware of any way to fix the problem short of reindexing those documents. Before that, you’ll probably need to choose a replica and just force the others to get in sync with it. I’d choose the current leader, since that’s slightly easier. Typically, a leader writes an update to it’s transaction log, then sends the request to all replicas, and when those all finish it acknowledges the update. If a replica gets restarted, and is less than N documents behind, the leader will only replay that transaction log. (Where N is the numRecordsToKeep configured in the updateLog section of solrconfig.xml) What you want is to provoke the heavy-duty process normally invoked if a replica has missed more than N docs, which essentially does a checksum and file copy on all the raw index files. FetchIndex would probably work, but it’s a replication handler API originally designed for master/slave replication, so take care: https://wiki.apache.org/solr/SolrReplication#HTTP_API Probably a lot easier would be to just delete the replica and re-create it. That will also trigger a full file copy of the index from the leader onto the new replica. I think design decisions around Solr generally use CP as a goal. (I sometimes wish I could get more AP behavior!) See posts like this: http://lucidworks.com/blog/2014/12/10/call-maybe-solrcloud-jepsen-flaky-networks/ So the fact that you encountered this sounds like a bug to me. That said, another general recommendation (of mine) is that you not use Solr as your primary data source, so you can rebuild your index from scratch if you really need to. On 1/26/16, 1:10 PM, "David Smith" wrote: >Thanks Jeff! A few comments > >>> >>> Although you could probably bounce a node and get your document counts back >>> in sync (by provoking a check) >>> > > >If the check is a simple doc count, that will not work. We have found that >replica1 and replica3, although they contain the same doc count, don’t have >the SAME docs. They each missed at least one update, but of different docs. >This also means none of our three replicas are complete. > >>> >>>it’s interesting that you’re in this situation. It implies to me that at >>>some point the leader couldn’t write a doc to one of the replicas, >>> > >That is our belief as well. We experienced a datacenter-wide network >disruption of a few seconds, and user complaints started the first workday >after that event. > >The most interesting log entry during the outage is this: > >"1/19/2016, 5:08:07 PM ERROR null DistributedUpdateProcessorRequest says it is >coming from leader, but we are the leader: >update.distrib=FROMLEADER&distrib.from=http://dot.dot.dot.dot:8983/solr/blah_blah_shard1_replica3/&wt=javabin&version=2"; > >>> >>> You might watch the achieved replication factor of your updates and see if >>> it ever changes >>> > >This is a good tip. I’m not sure I like the implication that any failure to >write all 3 of our replicas must be retried at the app layer. Is this really >how SolrCloud applications must be built to survive network partitions without >data loss? > >Regards, > >David > > >On 1/26/16, 12:20 PM, "Jeff Wartes" wrote: > >> >>My understanding is that the "version" represents the timestamp the searcher >>was opened, so it doesn’t really offer any assurances about your data. >> >>Although you could probably bounce a node and get your document counts back >>in sync (by provoking a check), it’s interesting that you’re in this >>situation. It implies to me that at some point the leader couldn’t write a >>doc to one of the replicas, but that the replica didn’t consider itself down >>enough to check itself. >> >>You might watch the achieved replication factor of your updates and see if it >>ever changes: >>https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance >> (See Achieved Replication Factor/min_rf) >> >>If it does, that might give you clues about how this is happening. Also, it >>might allow you to work around the issue by trying the write again. >> >> >> >> >> >> >>On 1/22/16, 10:52 AM, "David Smith" wrote: >> >>>I have a SolrCloud v5.4 collection with 3 replicas that appear to have >>>fallen permanently out of sync. Users started to complain that the same >>>search, executed twice, sometimes returned different result counts. Sure >>>enough, our replicas are not identical: >>> > shard1_replica1: 89867 documents / version 1453479763194 > shard1_replica2: 89866 documents / version 1453479763194 > shard1_replica3: 89867 documents / version 1453479763191 >>> >>>I do no
Re: SolrCloud replicas out of sync
Thanks Jeff! A few comments >> >> Although you could probably bounce a node and get your document counts back >> in sync (by provoking a check) >> If the check is a simple doc count, that will not work. We have found that replica1 and replica3, although they contain the same doc count, don’t have the SAME docs. They each missed at least one update, but of different docs. This also means none of our three replicas are complete. >> >>it’s interesting that you’re in this situation. It implies to me that at some >>point the leader couldn’t write a doc to one of the replicas, >> That is our belief as well. We experienced a datacenter-wide network disruption of a few seconds, and user complaints started the first workday after that event. The most interesting log entry during the outage is this: "1/19/2016, 5:08:07 PM ERROR null DistributedUpdateProcessorRequest says it is coming from leader, but we are the leader: update.distrib=FROMLEADER&distrib.from=http://dot.dot.dot.dot:8983/solr/blah_blah_shard1_replica3/&wt=javabin&version=2"; >> >> You might watch the achieved replication factor of your updates and see if >> it ever changes >> This is a good tip. I’m not sure I like the implication that any failure to write all 3 of our replicas must be retried at the app layer. Is this really how SolrCloud applications must be built to survive network partitions without data loss? Regards, David On 1/26/16, 12:20 PM, "Jeff Wartes" wrote: > >My understanding is that the "version" represents the timestamp the searcher >was opened, so it doesn’t really offer any assurances about your data. > >Although you could probably bounce a node and get your document counts back in >sync (by provoking a check), it’s interesting that you’re in this situation. >It implies to me that at some point the leader couldn’t write a doc to one of >the replicas, but that the replica didn’t consider itself down enough to check >itself. > >You might watch the achieved replication factor of your updates and see if it >ever changes: >https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance > (See Achieved Replication Factor/min_rf) > >If it does, that might give you clues about how this is happening. Also, it >might allow you to work around the issue by trying the write again. > > > > > > >On 1/22/16, 10:52 AM, "David Smith" wrote: > >>I have a SolrCloud v5.4 collection with 3 replicas that appear to have fallen >>permanently out of sync. Users started to complain that the same search, >>executed twice, sometimes returned different result counts. Sure enough, our >>replicas are not identical: >> shard1_replica1: 89867 documents / version 1453479763194 shard1_replica2: 89866 documents / version 1453479763194 shard1_replica3: 89867 documents / version 1453479763191 >> >>I do not think this discrepancy is going to resolve itself. The Solr Admin >>screen reports all 3 replicas as “Current”. The last modification to this >>collection was 2 hours before I captured this information, and our auto >>commit time is 60 seconds. >> >>I have a lot of concerns here, but my first question is if anyone else has >>had problems with out of sync replicas, and if so, what they have done to >>correct this? >> >>Kind Regards, >> >>David >>
Re: SolrCloud replicas out of sync
My understanding is that the "version" represents the timestamp the searcher was opened, so it doesn’t really offer any assurances about your data. Although you could probably bounce a node and get your document counts back in sync (by provoking a check), it’s interesting that you’re in this situation. It implies to me that at some point the leader couldn’t write a doc to one of the replicas, but that the replica didn’t consider itself down enough to check itself. You might watch the achieved replication factor of your updates and see if it ever changes: https://cwiki.apache.org/confluence/display/solr/Read+and+Write+Side+Fault+Tolerance (See Achieved Replication Factor/min_rf) If it does, that might give you clues about how this is happening. Also, it might allow you to work around the issue by trying the write again. On 1/22/16, 10:52 AM, "David Smith" wrote: >I have a SolrCloud v5.4 collection with 3 replicas that appear to have fallen >permanently out of sync. Users started to complain that the same search, >executed twice, sometimes returned different result counts. Sure enough, our >replicas are not identical: > >>> shard1_replica1: 89867 documents / version 1453479763194 >>> shard1_replica2: 89866 documents / version 1453479763194 >>> shard1_replica3: 89867 documents / version 1453479763191 > >I do not think this discrepancy is going to resolve itself. The Solr Admin >screen reports all 3 replicas as “Current”. The last modification to this >collection was 2 hours before I captured this information, and our auto commit >time is 60 seconds. > >I have a lot of concerns here, but my first question is if anyone else has had >problems with out of sync replicas, and if so, what they have done to correct >this? > >Kind Regards, > >David >
SolrCloud replicas out of sync
I have a SolrCloud v5.4 collection with 3 replicas that appear to have fallen permanently out of sync. Users started to complain that the same search, executed twice, sometimes returned different result counts. Sure enough, our replicas are not identical: >> shard1_replica1: 89867 documents / version 1453479763194 >> shard1_replica2: 89866 documents / version 1453479763194 >> shard1_replica3: 89867 documents / version 1453479763191 I do not think this discrepancy is going to resolve itself. The Solr Admin screen reports all 3 replicas as “Current”. The last modification to this collection was 2 hours before I captured this information, and our auto commit time is 60 seconds. I have a lot of concerns here, but my first question is if anyone else has had problems with out of sync replicas, and if so, what they have done to correct this? Kind Regards, David
