RE: Document Missing from Share in Solr cloud
Field type is string and this has happened for multiple docs over the past week. Regards, Ayush Date: Tue, 16 Apr 2013 14:06:40 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org btw ... what is the field type of your unique ID field? On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter thelabd...@gmail.comwrote: Ok, that makes more sense and is definitely cause for concern. Do you have a sense for whether this is ongoing or happened a few times unexpectedly in the past? If ongoing, then will probably be easier to track down the root cause. On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi cooltec...@outlook.comwrote: That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
Re: Document Missing from Share in Solr cloud
Well, your numdocs *is* the same. Your maxdocs isn't, which sounds right to me. maxdocs is the number of documents, including deleted ones. Given deleted docs are purged by background merges, it makes sense that each index is deciding differently when to do those merges. But the number of undeleted docs is the same which is a good thing. Do queries against each replica for a shard, with distrib=false, and see whether the results are the same. Upayavira On Wed, Apr 17, 2013, at 09:14 AM, Cool Techi wrote: Shouldnt the number of docs across shards be same, I can see a difference Shard 1 Last Modified:about 2 hours agoNum Docs:26236135Max Doc:26592164Deleted Docs:356029Version:6672183Segment Count:34Shard1 Replica Last Modified: about 2 hours agoNum Docs:26236135Max Doc:26594887Deleted Docs:358752Version:6678209Segment Count: 27 From: cooltec...@outlook.com To: solr-user@lucene.apache.org Subject: RE: Document Missing from Share in Solr cloud Date: Wed, 17 Apr 2013 13:28:16 +0530 Field type is string and this has happened for multiple docs over the past week. Regards, Ayush Date: Tue, 16 Apr 2013 14:06:40 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org btw ... what is the field type of your unique ID field? On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter thelabd...@gmail.comwrote: Ok, that makes more sense and is definitely cause for concern. Do you have a sense for whether this is ongoing or happened a few times unexpectedly in the past? If ongoing, then will probably be easier to track down the root cause. On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi cooltec...@outlook.comwrote: That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
RE: Document Missing from Share in Solr cloud
Sorry, made a copy paste mistake. The numbers are different. My cloud has two shards with each shard having 1 replica. One of the shards and replica have the same number of docs, while in the other shard there is a mismatch. Regards, Ayush From: u...@odoko.co.uk To: solr-user@lucene.apache.org Subject: Re: Document Missing from Share in Solr cloud Date: Wed, 17 Apr 2013 09:48:03 +0100 Well, your numdocs *is* the same. Your maxdocs isn't, which sounds right to me. maxdocs is the number of documents, including deleted ones. Given deleted docs are purged by background merges, it makes sense that each index is deciding differently when to do those merges. But the number of undeleted docs is the same which is a good thing. Do queries against each replica for a shard, with distrib=false, and see whether the results are the same. Upayavira On Wed, Apr 17, 2013, at 09:14 AM, Cool Techi wrote: Shouldnt the number of docs across shards be same, I can see a difference Shard 1 Last Modified:about 2 hours agoNum Docs:26236135Max Doc:26592164Deleted Docs:356029Version:6672183Segment Count:34Shard1 Replica Last Modified: about 2 hours agoNum Docs:26236135Max Doc:26594887Deleted Docs:358752Version:6678209Segment Count: 27 From: cooltec...@outlook.com To: solr-user@lucene.apache.org Subject: RE: Document Missing from Share in Solr cloud Date: Wed, 17 Apr 2013 13:28:16 +0530 Field type is string and this has happened for multiple docs over the past week. Regards, Ayush Date: Tue, 16 Apr 2013 14:06:40 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org btw ... what is the field type of your unique ID field? On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter thelabd...@gmail.comwrote: Ok, that makes more sense and is definitely cause for concern. Do you have a sense for whether this is ongoing or happened a few times unexpectedly in the past? If ongoing, then will probably be easier to track down the root cause. On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi cooltec...@outlook.comwrote: That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
Re: Document Missing from Share in Solr cloud
I have just experienced the same thing on 4.2.1. 4 Shards - each with 2 replicas. Did some bulk loading and all but one Shard match up. Small discrepancy between the replicas, but no obvious errors either. Will be doing further loading shortly and will report findings. Regards. Netty. On 17 April 2013 10:26, Cool Techi cooltec...@outlook.com wrote: Sorry, made a copy paste mistake. The numbers are different. My cloud has two shards with each shard having 1 replica. One of the shards and replica have the same number of docs, while in the other shard there is a mismatch. Regards, Ayush From: u...@odoko.co.uk To: solr-user@lucene.apache.org Subject: Re: Document Missing from Share in Solr cloud Date: Wed, 17 Apr 2013 09:48:03 +0100 Well, your numdocs *is* the same. Your maxdocs isn't, which sounds right to me. maxdocs is the number of documents, including deleted ones. Given deleted docs are purged by background merges, it makes sense that each index is deciding differently when to do those merges. But the number of undeleted docs is the same which is a good thing. Do queries against each replica for a shard, with distrib=false, and see whether the results are the same. Upayavira On Wed, Apr 17, 2013, at 09:14 AM, Cool Techi wrote: Shouldnt the number of docs across shards be same, I can see a difference Shard 1 Last Modified:about 2 hours agoNum Docs:26236135Max Doc:26592164Deleted Docs:356029Version:6672183Segment Count:34Shard1 Replica Last Modified: about 2 hours agoNum Docs:26236135Max Doc:26594887Deleted Docs:358752Version:6678209Segment Count: 27 From: cooltec...@outlook.com To: solr-user@lucene.apache.org Subject: RE: Document Missing from Share in Solr cloud Date: Wed, 17 Apr 2013 13:28:16 +0530 Field type is string and this has happened for multiple docs over the past week. Regards, Ayush Date: Tue, 16 Apr 2013 14:06:40 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org btw ... what is the field type of your unique ID field? On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter thelabd...@gmail.comwrote: Ok, that makes more sense and is definitely cause for concern. Do you have a sense for whether this is ongoing or happened a few times unexpectedly in the past? If ongoing, then will probably be easier to track down the root cause. On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi cooltec...@outlook.comwrote: That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush -- Annette Newton Database Administrator ServiceTick Ltd T:+44(0)1603 618326 Seebohm House, 2-4 Queen Street, Norwich, England NR2 4SQ www.servicetick.com *www.sessioncam.com* -- *This message is confidential and is intended to be read solely by the addressee. The contents should not be disclosed to any other person or copies taken unless authorised to do so. If you are not the intended recipient, please notify the sender and permanently delete this message. As Internet communications are not secure ServiceTick accepts neither legal responsibility for the contents of this message nor responsibility for any change made to this message after it was forwarded by the original author.*
Re: Document Missing from Share in Solr cloud
If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
RE: Document Missing from Share in Solr cloud
That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
Re: Document Missing from Share in Solr cloud
Ok, that makes more sense and is definitely cause for concern. Do you have a sense for whether this is ongoing or happened a few times unexpectedly in the past? If ongoing, then will probably be easier to track down the root cause. On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi cooltec...@outlook.com wrote: That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
Re: Document Missing from Share in Solr cloud
btw ... what is the field type of your unique ID field? On Tue, Apr 16, 2013 at 12:34 PM, Timothy Potter thelabd...@gmail.comwrote: Ok, that makes more sense and is definitely cause for concern. Do you have a sense for whether this is ongoing or happened a few times unexpectedly in the past? If ongoing, then will probably be easier to track down the root cause. On Tue, Apr 16, 2013 at 12:08 PM, Cool Techi cooltec...@outlook.comwrote: That's what I am trying to say, the document is not replicated across all the replicas for a specific shard, hence the query show different results on every refresh. Date: Tue, 16 Apr 2013 11:34:18 -0600 Subject: Re: Document Missing from Share in Solr cloud From: thelabd...@gmail.com To: solr-user@lucene.apache.org If you are using the default doc router for indexing in SolrCloud, then a document only exists in a single shard but can be replicated in that shard to any number of replicas. Can you clarify your question as it sounds like you're saying that the document is not replicated across all the replicas for a specific shard? If so, that's definitely a problem ... On Tue, Apr 16, 2013 at 11:22 AM, Cool Techi cooltec...@outlook.com wrote: Hi, We noticed a strange behavior in our solr cloud setup, we are using solr4.2 with 1:3 replication setting. We noticed that some of the documents were showing up in search sometimes and not at other, the reason being the document was not present in all the shards. We have restarted zookeeper and also entire cloud, but these documents are not being replicated in all the shards for some reason and hence inconsistent search results. Regards, Ayush
