Re: SolrCloud replica and leader out of Sync somehow
Do you have the logs for this? Either around startup or when you are forcing replication. Logs around both would be helpful. Also the doc counts for each shard? On Mar 20, 2012, at 10:16 AM, Jamie Johnson wrote: > I'm trying to figure out how it's possible for 2 solr instances (1 > which is leader 1 is replica) to be out of sync. I've done commits to > the solr instances, forced replication but still the solr instances > have different info. The relevant snippet from my clusterstate.json > is listed below. > > >\"shard3\":{ > \"host2:7577_solr_shard3-core2\":{ >\"shard\":\"shard3\", >\"leader\":\"true\", >\"state\":\"active\", >\"core\":\"shard3-core2\", >\"collection\":\"collection1\", >\"node_name\":\"host2:7577_solr\", >\"base_url\":\"http://host2:7577/solr\"}, > \"host1:7575_solr_shard3-core1\":{ >\"shard\":\"shard3\", >\"state\":\"active\", >\"core\":\"shard3-core1\", >\"collection\":\"collection1\", >\"node_name\":\"host1:7575_solr\", >\"base_url\":\"http://host1:7575/solr\"}}, > > > Where can I look to see why this is happening? - Mark Miller lucidimagination.com
Re: SolrCloud replica and leader out of Sync somehow
DocCounts are the same. I am going to disable my custom component to see if that is mucking with something but it seems to be working properly. After looking at the results a little closer (expanding the number of results coming back) it seems that the same information is in both but the order in which the items are being returned is not the same. I'm sorting by score when they seem to be in different orders, if I sort by key then the results look the same. On Tue, Mar 20, 2012 at 10:52 AM, Mark Miller wrote: > Do you have the logs for this? Either around startup or when you are forcing > replication. Logs around both would be helpful. > > Also the doc counts for each shard? > > On Mar 20, 2012, at 10:16 AM, Jamie Johnson wrote: > >> I'm trying to figure out how it's possible for 2 solr instances (1 >> which is leader 1 is replica) to be out of sync. I've done commits to >> the solr instances, forced replication but still the solr instances >> have different info. The relevant snippet from my clusterstate.json >> is listed below. >> >> >> \"shard3\":{ >> \"host2:7577_solr_shard3-core2\":{ >> \"shard\":\"shard3\", >> \"leader\":\"true\", >> \"state\":\"active\", >> \"core\":\"shard3-core2\", >> \"collection\":\"collection1\", >> \"node_name\":\"host2:7577_solr\", >> \"base_url\":\"http://host2:7577/solr\"}, >> \"host1:7575_solr_shard3-core1\":{ >> \"shard\":\"shard3\", >> \"state\":\"active\", >> \"core\":\"shard3-core1\", >> \"collection\":\"collection1\", >> \"node_name\":\"host1:7575_solr\", >> \"base_url\":\"http://host1:7575/solr\"}}, >> >> >> Where can I look to see why this is happening? > > - Mark Miller > lucidimagination.com > > > > > > > > > > >
Re: SolrCloud replica and leader out of Sync somehow
ok, with my custom component out of the picture I still have the same issue. Specifically, when sorting by score on a leader and replica I am getting different doc orderings. Is this something anyone has seen? On Tue, Mar 20, 2012 at 11:09 AM, Jamie Johnson wrote: > DocCounts are the same. I am going to disable my custom component to > see if that is mucking with something but it seems to be working > properly. > > After looking at the results a little closer (expanding the number of > results coming back) it seems that the same information is in both but > the order in which the items are being returned is not the same. I'm > sorting by score when they seem to be in different orders, if I sort > by key then the results look the same. > > On Tue, Mar 20, 2012 at 10:52 AM, Mark Miller wrote: >> Do you have the logs for this? Either around startup or when you are forcing >> replication. Logs around both would be helpful. >> >> Also the doc counts for each shard? >> >> On Mar 20, 2012, at 10:16 AM, Jamie Johnson wrote: >> >>> I'm trying to figure out how it's possible for 2 solr instances (1 >>> which is leader 1 is replica) to be out of sync. I've done commits to >>> the solr instances, forced replication but still the solr instances >>> have different info. The relevant snippet from my clusterstate.json >>> is listed below. >>> >>> >>> \"shard3\":{ >>> \"host2:7577_solr_shard3-core2\":{ >>> \"shard\":\"shard3\", >>> \"leader\":\"true\", >>> \"state\":\"active\", >>> \"core\":\"shard3-core2\", >>> \"collection\":\"collection1\", >>> \"node_name\":\"host2:7577_solr\", >>> \"base_url\":\"http://host2:7577/solr\"}, >>> \"host1:7575_solr_shard3-core1\":{ >>> \"shard\":\"shard3\", >>> \"state\":\"active\", >>> \"core\":\"shard3-core1\", >>> \"collection\":\"collection1\", >>> \"node_name\":\"host1:7575_solr\", >>> \"base_url\":\"http://host1:7575/solr\"}}, >>> >>> >>> Where can I look to see why this is happening? >> >> - Mark Miller >> lucidimagination.com >> >> >> >> >> >> >> >> >> >> >>
Re: SolrCloud replica and leader out of Sync somehow
On Tue, Mar 20, 2012 at 11:17 AM, Jamie Johnson wrote: > ok, with my custom component out of the picture I still have the same > issue. Specifically, when sorting by score on a leader and replica I > am getting different doc orderings. Is this something anyone has > seen? This is certainly possible and expected - sorting tiebreakers is by internal lucene docid, which can change (even on a single node!) If you need lists that don't shift around due to unrelated changes, make sure you don't have any ties! -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
HmmmOk, I don't see how it's possible for me to ensure that there are no ties. If a query were for *:* everything has a constant score, if the user requested 1 page then requested the next the results on the second page could be duplicates from what was on the first page. I don't remember ever seeing this issue on older versions of SolrCloud, although from what you're saying I should have. What could explain why I never saw this before? Another possible fix to ensure proper ordering couldn't we always specify a sort order which contained the key? So for instance the user asks for score asc, we'd make this score asc,key asc so that results would be order by score and then by key so the results across pages would be consistent? On Tue, Mar 20, 2012 at 11:30 AM, Yonik Seeley wrote: > On Tue, Mar 20, 2012 at 11:17 AM, Jamie Johnson wrote: >> ok, with my custom component out of the picture I still have the same >> issue. Specifically, when sorting by score on a leader and replica I >> am getting different doc orderings. Is this something anyone has >> seen? > > This is certainly possible and expected - sorting tiebreakers is by > internal lucene docid, which can change (even on a single node!) > If you need lists that don't shift around due to unrelated changes, > make sure you don't have any ties! > > -Yonik > lucenerevolution.com - Lucene/Solr Open Source Search Conference. > Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
On Tue, Mar 20, 2012 at 11:39 AM, Jamie Johnson wrote: > HmmmOk, I don't see how it's possible for me to ensure that there > are no ties. If a query were for *:* everything has a constant score, > if the user requested 1 page then requested the next the results on > the second page could be duplicates from what was on the first page. > I don't remember ever seeing this issue on older versions of > SolrCloud, although from what you're saying I should have. What could > explain why I never saw this before? If you use replication only to duplicate an index (and avoid any merges), then you will have identical docids. > Another possible fix to ensure proper ordering couldn't we always > specify a sort order which contained the key? So for instance the > user asks for score asc, we'd make this score asc,key asc so that > results would be order by score and then by key so the results across > pages would be consistent? Yep. And like I said, this is also an issue even on a single node. docid A can be before docid B, then a segment merge can cause these to be shuffled. -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
I believe we're using replication to only duplicate the index (standard SolrCloud nothing special on our end) so I don't see why the docids wouldn't be the sameam I missing something that is happening there that I am unaware of? On Tue, Mar 20, 2012 at 11:50 AM, Yonik Seeley wrote: > On Tue, Mar 20, 2012 at 11:39 AM, Jamie Johnson wrote: >> HmmmOk, I don't see how it's possible for me to ensure that there >> are no ties. If a query were for *:* everything has a constant score, >> if the user requested 1 page then requested the next the results on >> the second page could be duplicates from what was on the first page. >> I don't remember ever seeing this issue on older versions of >> SolrCloud, although from what you're saying I should have. What could >> explain why I never saw this before? > > If you use replication only to duplicate an index (and avoid any > merges), then you will have identical docids. > >> Another possible fix to ensure proper ordering couldn't we always >> specify a sort order which contained the key? So for instance the >> user asks for score asc, we'd make this score asc,key asc so that >> results would be order by score and then by key so the results across >> pages would be consistent? > > Yep. > > And like I said, this is also an issue even on a single node. > docid A can be before docid B, then a segment merge can cause these to > be shuffled. > > -Yonik > lucenerevolution.com - Lucene/Solr Open Source Search Conference. > Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
Thanks Yonik, I really appreciate the explanation. It sounds like the best solution for me to solve this is to add the additional sort parameter. That being said is there a significant memory increase to do this when sorting by score? I don't see how with SolrCloud I can avoid doing this, and how others wouldn't need to do the same thing. On Tue, Mar 20, 2012 at 1:38 PM, Yonik Seeley wrote: > On Tue, Mar 20, 2012 at 1:07 PM, Jamie Johnson wrote: >> I believe we're using replication to only duplicate the index >> (standard SolrCloud nothing special on our end) so I don't see why the >> docids wouldn't be the sameam I missing something that is >> happening there that I am unaware of? > > Each document is pushed to the replicas (i.e. standard whole-index > "replication" is only used in recovery scenarios). If you're using > multiple threads to index, then docA can be indexed before docB on one > replica and vice-versa on a different replica (or on the leader). > Although even if this were not the case, I don't believe Lucene is > deterministic in this respect anyway (i.e. indexing identically on two > different boxes is not guaranteed to result in the exact same internal > document order). > > -Yonik > lucenerevolution.com - Lucene/Solr Open Source Search Conference. > Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
Given that in a distributed environment the docids are not guaranteed to be the same across shards should the sorting use the uniqueId field as the tie breaker by default? On Tue, Mar 20, 2012 at 2:10 PM, Yonik Seeley wrote: > On Tue, Mar 20, 2012 at 2:02 PM, Jamie Johnson wrote: >> I'll try to dig for the JIRA. Also I'm assuming this could happen on >> any sort, not just score correct? Meaning if we sorted by a date >> field and there were duplicates in that date field order wouldn't be >> guaranteed for the same reasons right? > > Correct - internal docid is the tiebreaker for all sorts. > > -Yonik > lucenerevolution.com - Lucene/Solr Open Source Search Conference. > Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
Not sure if this got lost in the shuffle, were there any thoughts on this? On Wed, Mar 21, 2012 at 11:02 AM, Jamie Johnson wrote: > Given that in a distributed environment the docids are not guaranteed > to be the same across shards should the sorting use the uniqueId field > as the tie breaker by default? > > On Tue, Mar 20, 2012 at 2:10 PM, Yonik Seeley > wrote: >> On Tue, Mar 20, 2012 at 2:02 PM, Jamie Johnson wrote: >>> I'll try to dig for the JIRA. Also I'm assuming this could happen on >>> any sort, not just score correct? Meaning if we sorted by a date >>> field and there were duplicates in that date field order wouldn't be >>> guaranteed for the same reasons right? >> >> Correct - internal docid is the tiebreaker for all sorts. >> >> -Yonik >> lucenerevolution.com - Lucene/Solr Open Source Search Conference. >> Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
On Thu, Apr 5, 2012 at 12:19 AM, Jamie Johnson wrote: > Not sure if this got lost in the shuffle, were there any thoughts on this? Sorting by "id" could be pretty expensive (memory-wise), so I don't think it should be default or anything. We also need a way for a client to hit the same set of servers again anyway (to handle other possible variations like commit time). To handle the tiebreak stuff, you could also sort by _version_ - that should be unique in an index and is already used under the covers and hence shouldn't add any extra memory overhead. versions increase over time, so "_version desc" should give you newer documents first. -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10 > On Wed, Mar 21, 2012 at 11:02 AM, Jamie Johnson wrote: >> Given that in a distributed environment the docids are not guaranteed >> to be the same across shards should the sorting use the uniqueId field >> as the tie breaker by default? >> >> On Tue, Mar 20, 2012 at 2:10 PM, Yonik Seeley >> wrote: >>> On Tue, Mar 20, 2012 at 2:02 PM, Jamie Johnson wrote: I'll try to dig for the JIRA. Also I'm assuming this could happen on any sort, not just score correct? Meaning if we sorted by a date field and there were duplicates in that date field order wouldn't be guaranteed for the same reasons right? >>> >>> Correct - internal docid is the tiebreaker for all sorts. >>> >>> -Yonik >>> lucenerevolution.com - Lucene/Solr Open Source Search Conference. >>> Boston May 7-10
Re: SolrCloud replica and leader out of Sync somehow
awesome Yonik. I'll indeed try this. Thanks! On Thu, Apr 5, 2012 at 10:20 AM, Yonik Seeley wrote: > On Thu, Apr 5, 2012 at 12:19 AM, Jamie Johnson wrote: >> Not sure if this got lost in the shuffle, were there any thoughts on this? > > Sorting by "id" could be pretty expensive (memory-wise), so I don't > think it should be default or anything. > We also need a way for a client to hit the same set of servers again > anyway (to handle other possible variations like commit time). > > To handle the tiebreak stuff, you could also sort by _version_ - that > should be unique in an index and is already used under the covers and > hence shouldn't add any extra memory overhead. versions increase over > time, so "_version desc" should give you newer documents first. > > -Yonik > lucenerevolution.com - Lucene/Solr Open Source Search Conference. > Boston May 7-10 > > > > >> On Wed, Mar 21, 2012 at 11:02 AM, Jamie Johnson wrote: >>> Given that in a distributed environment the docids are not guaranteed >>> to be the same across shards should the sorting use the uniqueId field >>> as the tie breaker by default? >>> >>> On Tue, Mar 20, 2012 at 2:10 PM, Yonik Seeley >>> wrote: On Tue, Mar 20, 2012 at 2:02 PM, Jamie Johnson wrote: > I'll try to dig for the JIRA. Also I'm assuming this could happen on > any sort, not just score correct? Meaning if we sorted by a date > field and there were duplicates in that date field order wouldn't be > guaranteed for the same reasons right? Correct - internal docid is the tiebreaker for all sorts. -Yonik lucenerevolution.com - Lucene/Solr Open Source Search Conference. Boston May 7-10