I have since removed the files but when I had looked there was an index
directory, the only files I remember being there were the segments, one of
the _* files were present. I'll watch it to see if it happens again but it
happened on 2 of the shards while heavy indexing.
On Wed, Apr 3, 2013 at 10:13 PM, Mark Miller <markrmil...@gmail.com> wrote:
> Is that file still there when you look? Not being able to find an index
> file is not a common error I've seen recently.
>
> Do those replicas have an index directory or when you look on disk, is it
> an index.timestamp directory?
>
> - Mark
>
> On Apr 3, 2013, at 10:01 PM, Jamie Johnson <jej2...@gmail.com> wrote:
>
> > so something is still not right. Things were going ok, but I'm seeing
> this
> > in the logs of several of the replicas
> >
> > SEVERE: Unable to create core: dsc-shard3-core1
> > org.apache.solr.common.SolrException: Error opening new searcher
> > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:822)
> > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
> > at
> > org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:967)
> > at
> > org.apache.solr.core.CoreContainer.create(CoreContainer.java:1049)
> > at
> org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634)
> > at
> org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: org.apache.solr.common.SolrException: Error opening new
> searcher
> > at
> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1435)
> > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1547)
> > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:797)
> > ... 13 more
> > Caused by: org.apache.solr.common.SolrException: Error opening Reader
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:172)
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:183)
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:179)
> > at
> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1411)
> > ... 15 more
> > Caused by: java.io.FileNotFoundException:
> > /cce2/solr/data/dsc-shard3-core1/index/_13x.si (No such file or
> directory)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
> > at
> > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:193)
> > at
> >
> org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:232)
> > at
> >
> org.apache.lucene.codecs.lucene40.Lucene40SegmentInfoReader.read(Lucene40SegmentInfoReader.java:50)
> > at
> org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:301)
> > at
> >
> org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:56)
> > at
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
> > at
> >
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
> > at
> > org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
> > at
> >
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
> > ... 18 more
> >
> >
> >
> > On Wed, Apr 3, 2013 at 8:54 PM, Jamie Johnson <jej2...@gmail.com> wrote:
> >
> >> Thanks I will try that.
> >>
> >>
> >> On Wed, Apr 3, 2013 at 8:28 PM, Mark Miller <markrmil...@gmail.com>
> wrote:
> >>
> >>>
> >>>
> >>> On Apr 3, 2013, at 8:17 PM, Jamie Johnson <jej2...@gmail.com> wrote:
> >>>
> >>>> I am not using the concurrent low pause garbage collector, I could
> look
> >>> at
> >>>> switching, I'm assuming you're talking about adding
> >>> -XX:+UseConcMarkSweepGC
> >>>> correct?
> >>>
> >>> Right - if you don't do that, the default is almost always the
> throughput
> >>> collector (I've only seen OSX buck this trend when apple handled java).
> >>> That means stop the world garbage collections, so with larger heaps,
> that
> >>> can be a fair amount of time that no threads can run. It's not that
> great
> >>> for something as interactive as search generally is anyway, but it's
> always
> >>> not that great when added to heavy load and a 15 sec session timeout
> >>> between solr and zk.
> >>>
> >>>
> >>> The below is odd - a replica node is waiting for the leader to see it
> as
> >>> recovering and live - live means it has created an ephemeral node for
> that
> >>> Solr corecontainer in zk - it's very strange if that didn't happen,
> unless
> >>> this happened during shutdown or something.
> >>>
> >>>>
> >>>> I also just had a shard go down and am seeing this in the log
> >>>>
> >>>> SEVERE: org.apache.solr.common.SolrException: I was asked to wait on
> >>> state
> >>>> down for 10.38.33.17:7576_solr but I still do not see the requested
> >>> state.
> >>>> I see state: recovering live:false
> >>>> at
> >>>>
> >>>
> org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:890)
> >>>> at
> >>>>
> >>>
> org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:186)
> >>>> at
> >>>>
> >>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >>>> at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:591)
> >>>> at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:192)
> >>>> at
> >>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
> >>>> at
> >>>>
> >>>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> >>>>
> >>>> Nothing other than this in the log jumps out as interesting though.
> >>>>
> >>>>
> >>>> On Wed, Apr 3, 2013 at 7:47 PM, Mark Miller <markrmil...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> This shouldn't be a problem though, if things are working as they are
> >>>>> supposed to. Another node should simply take over as the overseer and
> >>>>> continue processing the work queue. It's just best if you configure
> so
> >>> that
> >>>>> session timeouts don't happen unless a node is really down. On the
> >>> other
> >>>>> hand, it's nicer to detect that faster. Your tradeoff to make.
> >>>>>
> >>>>> - Mark
> >>>>>
> >>>>> On Apr 3, 2013, at 7:46 PM, Mark Miller <markrmil...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Yeah. Are you using the concurrent low pause garbage collector?
> >>>>>>
> >>>>>> This means the overseer wasn't able to communicate with zk for 15
> >>>>> seconds - due to load or gc or whatever. If you can't resolve the
> root
> >>>>> cause of that, or the load just won't allow for it, next best thing
> >>> you can
> >>>>> do is raise it to 30 seconds.
> >>>>>>
> >>>>>> - Mark
> >>>>>>
> >>>>>> On Apr 3, 2013, at 7:41 PM, Jamie Johnson <jej2...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> I am occasionally seeing this in the log, is this just a timeout
> >>> issue?
> >>>>>>> Should I be increasing the zk client timeout?
> >>>>>>>
> >>>>>>> WARNING: Overseer cannot talk to ZK
> >>>>>>> Apr 3, 2013 11:14:25 PM
> >>>>>>> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher process
> >>>>>>> INFO: Watcher fired on path: null state: Expired type None
> >>>>>>> Apr 3, 2013 11:14:25 PM
> >>>>> org.apache.solr.cloud.Overseer$ClusterStateUpdater
> >>>>>>> run
> >>>>>>> WARNING: Solr cannot talk to ZK, exiting Overseer main queue loop
> >>>>>>> org.apache.zookeeper.KeeperException$SessionExpiredException:
> >>>>>>> KeeperErrorCode = Session expired for /overseer/queue
> >>>>>>> at
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> >>>>>>> at
> >>>>>>>
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>>>> at
> >>> org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:236)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:233)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:233)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:89)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.cloud.DistributedQueue.element(DistributedQueue.java:131)
> >>>>>>> at
> >>>>>>>
> >>> org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:326)
> >>>>>>> at
> >>>>>>>
> >>>>>
> >>>
> org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:128)
> >>>>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Apr 3, 2013 at 7:25 PM, Jamie Johnson <jej2...@gmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> just an update, I'm at 1M records now with no issues. This looks
> >>>>>>>> promising as to the cause of my issues, thanks for the help. Is
> the
> >>>>>>>> routing method with numShards documented anywhere? I know
> >>> numShards is
> >>>>>>>> documented but I didn't know that the routing changed if you don't
> >>>>> specify
> >>>>>>>> it.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Apr 3, 2013 at 4:44 PM, Jamie Johnson <jej2...@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> with these changes things are looking good, I'm up to 600,000
> >>>>> documents
> >>>>>>>>> without any issues as of right now. I'll keep going and add more
> >>> to
> >>>>> see if
> >>>>>>>>> I find anything.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, Apr 3, 2013 at 4:01 PM, Jamie Johnson <jej2...@gmail.com
> >
> >>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> ok, so that's not a deal breaker for me. I just changed it to
> >>> match
> >>>>> the
> >>>>>>>>>> shards that are auto created and it looks like things are happy.
> >>>>> I'll go
> >>>>>>>>>> ahead and try my test to see if I can get things out of sync.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Apr 3, 2013 at 3:56 PM, Mark Miller <
> >>> markrmil...@gmail.com
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I had thought you could - but looking at the code recently, I
> >>> don't
> >>>>>>>>>>> think you can anymore. I think that's a technical limitation
> more
> >>>>> than
> >>>>>>>>>>> anything though. When these changes were made, I think support
> >>> for
> >>>>> that was
> >>>>>>>>>>> simply not added at the time.
> >>>>>>>>>>>
> >>>>>>>>>>> I'm not sure exactly how straightforward it would be, but it
> >>> seems
> >>>>>>>>>>> doable - as it is, the overseer will preallocate shards when
> >>> first
> >>>>> creating
> >>>>>>>>>>> the collection - that's when they get named shard(n). There
> would
> >>>>> have to
> >>>>>>>>>>> be logic to replace shard(n) with the custom shard name when
> the
> >>>>> core
> >>>>>>>>>>> actually registers.
> >>>>>>>>>>>
> >>>>>>>>>>> - Mark
> >>>>>>>>>>>
> >>>>>>>>>>> On Apr 3, 2013, at 3:42 PM, Jamie Johnson <jej2...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> answered my own question, it now says compositeId. What is
> >>>>>>>>>>> problematic
> >>>>>>>>>>>> though is that in addition to my shards (which are say
> >>>>> jamie-shard1)
> >>>>>>>>>>> I see
> >>>>>>>>>>>> the solr created shards (shard1). I assume that these were
> >>> created
> >>>>>>>>>>> because
> >>>>>>>>>>>> of the numShards param. Is there no way to specify the names
> of
> >>>>> these
> >>>>>>>>>>>> shards?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Apr 3, 2013 at 3:25 PM, Jamie Johnson <
> >>> jej2...@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> ah interesting....so I need to specify num shards, blow out
> zk
> >>> and
> >>>>>>>>>>> then
> >>>>>>>>>>>>> try this again to see if things work properly now. What is
> >>> really
> >>>>>>>>>>> strange
> >>>>>>>>>>>>> is that for the most part things have worked right and on
> >>> 4.2.1 I
> >>>>>>>>>>> have
> >>>>>>>>>>>>> 600,000 items indexed with no duplicates. In any event I
> will
> >>>>>>>>>>> specify num
> >>>>>>>>>>>>> shards clear out zk and begin again. If this works properly
> >>> what
> >>>>>>>>>>> should
> >>>>>>>>>>>>> the router type be?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Apr 3, 2013 at 3:14 PM, Mark Miller <
> >>>>> markrmil...@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> If you don't specify numShards after 4.1, you get an
> implicit
> >>> doc
> >>>>>>>>>>> router
> >>>>>>>>>>>>>> and it's up to you to distribute updates. In the past,
> >>>>> partitioning
> >>>>>>>>>>> was
> >>>>>>>>>>>>>> done on the fly - but for shard splitting and perhaps other
> >>>>>>>>>>> features, we
> >>>>>>>>>>>>>> now divvy up the hash range up front based on numShards and
> >>> store
> >>>>>>>>>>> it in
> >>>>>>>>>>>>>> ZooKeeper. No numShards is now how you take complete control
> >>> of
> >>>>>>>>>>> updates
> >>>>>>>>>>>>>> yourself.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Apr 3, 2013, at 2:57 PM, Jamie Johnson <
> jej2...@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The router says "implicit". I did start from a blank zk
> >>> state
> >>>>> but
> >>>>>>>>>>>>>> perhaps
> >>>>>>>>>>>>>>> I missed one of the ZkCLI commands? One of my shards from
> >>> the
> >>>>>>>>>>>>>>> clusterstate.json is shown below. What is the process that
> >>>>> should
> >>>>>>>>>>> be
> >>>>>>>>>>>>>> done
> >>>>>>>>>>>>>>> to bootstrap a cluster other than the ZkCLI commands I
> listed
> >>>>>>>>>>> above? My
> >>>>>>>>>>>>>>> process right now is run those ZkCLI commands and then
> start
> >>>>> solr
> >>>>>>>>>>> on
> >>>>>>>>>>>>>> all of
> >>>>>>>>>>>>>>> the instances with a command like this
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> java -server -Dshard=shard5 -DcoreName=shard5-core1
> >>>>>>>>>>>>>>> -Dsolr.data.dir=/solr/data/shard5-core1
> >>>>>>>>>>>>>> -Dcollection.configName=solr-conf
> >>>>>>>>>>>>>>> -Dcollection=collection1
> >>>>>>>>>>> -DzkHost=so-zoo1:2181,so-zoo2:2181,so-zoo3:2181
> >>>>>>>>>>>>>>> -Djetty.port=7575 -DhostPort=7575 -jar start.jar
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I feel like maybe I'm missing a step.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> "shard5":{
> >>>>>>>>>>>>>>> "state":"active",
> >>>>>>>>>>>>>>> "replicas":{
> >>>>>>>>>>>>>>> "10.38.33.16:7575_solr_shard5-core1":{
> >>>>>>>>>>>>>>> "shard":"shard5",
> >>>>>>>>>>>>>>> "state":"active",
> >>>>>>>>>>>>>>> "core":"shard5-core1",
> >>>>>>>>>>>>>>> "collection":"collection1",
> >>>>>>>>>>>>>>> "node_name":"10.38.33.16:7575_solr",
> >>>>>>>>>>>>>>> "base_url":"http://10.38.33.16:7575/solr";,
> >>>>>>>>>>>>>>> "leader":"true"},
> >>>>>>>>>>>>>>> "10.38.33.17:7577_solr_shard5-core2":{
> >>>>>>>>>>>>>>> "shard":"shard5",
> >>>>>>>>>>>>>>> "state":"recovering",
> >>>>>>>>>>>>>>> "core":"shard5-core2",
> >>>>>>>>>>>>>>> "collection":"collection1",
> >>>>>>>>>>>>>>> "node_name":"10.38.33.17:7577_solr",
> >>>>>>>>>>>>>>> "base_url":"http://10.38.33.17:7577/solr"}}}
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 2:40 PM, Mark Miller <
> >>>>> markrmil...@gmail.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> It should be part of your clusterstate.json. Some users
> have
> >>>>>>>>>>> reported
> >>>>>>>>>>>>>>>> trouble upgrading a previous zk install when this change
> >>> came.
> >>>>> I
> >>>>>>>>>>>>>>>> recommended manually updating the clusterstate.json to
> have
> >>> the
> >>>>>>>>>>> right
> >>>>>>>>>>>>>> info,
> >>>>>>>>>>>>>>>> and that seemed to work. Otherwise, I guess you have to
> >>> start
> >>>>>>>>>>> from a
> >>>>>>>>>>>>>> clean
> >>>>>>>>>>>>>>>> zk state.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> If you don't have that range information, I think there
> >>> will be
> >>>>>>>>>>>>>> trouble.
> >>>>>>>>>>>>>>>> Do you have an router type defined in the
> clusterstate.json?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Apr 3, 2013, at 2:24 PM, Jamie Johnson <
> >>> jej2...@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Where is this information stored in ZK? I don't see it
> in
> >>> the
> >>>>>>>>>>> cluster
> >>>>>>>>>>>>>>>>> state (or perhaps I don't understand it ;) ).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Perhaps something with my process is broken. What I do
> >>> when I
> >>>>>>>>>>> start
> >>>>>>>>>>>>>> from
> >>>>>>>>>>>>>>>>> scratch is the following
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> ZkCLI -cmd upconfig ...
> >>>>>>>>>>>>>>>>> ZkCLI -cmd linkconfig ....
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> but I don't ever explicitly create the collection. What
> >>>>> should
> >>>>>>>>>>> the
> >>>>>>>>>>>>>> steps
> >>>>>>>>>>>>>>>>> from scratch be? I am moving from an unreleased snapshot
> >>> of
> >>>>> 4.0
> >>>>>>>>>>> so I
> >>>>>>>>>>>>>>>> never
> >>>>>>>>>>>>>>>>> did that previously either so perhaps I did create the
> >>>>>>>>>>> collection in
> >>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> my steps to get this working but have forgotten it along
> >>> the
> >>>>> way.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 2:16 PM, Mark Miller <
> >>>>>>>>>>> markrmil...@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks for digging Jamie. In 4.2, hash ranges are
> >>> assigned up
> >>>>>>>>>>> front
> >>>>>>>>>>>>>>>> when a
> >>>>>>>>>>>>>>>>>> collection is created - each shard gets a range, which
> is
> >>>>>>>>>>> stored in
> >>>>>>>>>>>>>>>>>> zookeeper. You should not be able to end up with the
> same
> >>> id
> >>>>> on
> >>>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>> shards - something very odd going on.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hopefully I'll have some time to try and help you
> >>> reproduce.
> >>>>>>>>>>> Ideally
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> can capture it in a test case.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Apr 3, 2013, at 1:13 PM, Jamie Johnson <
> >>> jej2...@gmail.com
> >>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> no, my thought was wrong, it appears that even with the
> >>>>>>>>>>> parameter
> >>>>>>>>>>>>>> set I
> >>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>> seeing this behavior. I've been able to duplicate it
> on
> >>>>> 4.2.0
> >>>>>>>>>>> by
> >>>>>>>>>>>>>>>>>> indexing
> >>>>>>>>>>>>>>>>>>> 100,000 documents on 10 threads (10,000 each) when I
> get
> >>> to
> >>>>>>>>>>> 400,000
> >>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>>> so.
> >>>>>>>>>>>>>>>>>>> I will try this on 4.2.1. to see if I see the same
> >>> behavior
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 12:37 PM, Jamie Johnson <
> >>>>>>>>>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Since I don't have that many items in my index I
> >>> exported
> >>>>> all
> >>>>>>>>>>> of
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> keys
> >>>>>>>>>>>>>>>>>>>> for each shard and wrote a simple java program that
> >>> checks
> >>>>> for
> >>>>>>>>>>>>>>>>>> duplicates.
> >>>>>>>>>>>>>>>>>>>> I found some duplicate keys on different shards, a
> grep
> >>> of
> >>>>> the
> >>>>>>>>>>>>>> files
> >>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> the keys found does indicate that they made it to the
> >>> wrong
> >>>>>>>>>>> places.
> >>>>>>>>>>>>>>>> If
> >>>>>>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>>>>>>> notice documents with the same ID are on shard 3 and
> >>> shard
> >>>>> 5.
> >>>>>>>>>>> Is
> >>>>>>>>>>>>>> it
> >>>>>>>>>>>>>>>>>>>> possible that the hash is being calculated taking into
> >>>>>>>>>>> account only
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>> "live" nodes? I know that we don't specify the
> >>> numShards
> >>>>>>>>>>> param @
> >>>>>>>>>>>>>>>>>> startup
> >>>>>>>>>>>>>>>>>>>> so could this be what is happening?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> grep -c "7cd1a717-3d94-4f5d-bcb1-9d8a95ca78de" *
> >>>>>>>>>>>>>>>>>>>> shard1-core1:0
> >>>>>>>>>>>>>>>>>>>> shard1-core2:0
> >>>>>>>>>>>>>>>>>>>> shard2-core1:0
> >>>>>>>>>>>>>>>>>>>> shard2-core2:0
> >>>>>>>>>>>>>>>>>>>> shard3-core1:1
> >>>>>>>>>>>>>>>>>>>> shard3-core2:1
> >>>>>>>>>>>>>>>>>>>> shard4-core1:0
> >>>>>>>>>>>>>>>>>>>> shard4-core2:0
> >>>>>>>>>>>>>>>>>>>> shard5-core1:1
> >>>>>>>>>>>>>>>>>>>> shard5-core2:1
> >>>>>>>>>>>>>>>>>>>> shard6-core1:0
> >>>>>>>>>>>>>>>>>>>> shard6-core2:0
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 10:42 AM, Jamie Johnson <
> >>>>>>>>>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Something interesting that I'm noticing as well, I
> just
> >>>>>>>>>>> indexed
> >>>>>>>>>>>>>>>> 300,000
> >>>>>>>>>>>>>>>>>>>>> items, and some how 300,020 ended up in the index. I
> >>>>> thought
> >>>>>>>>>>>>>>>> perhaps I
> >>>>>>>>>>>>>>>>>>>>> messed something up so I started the indexing again
> and
> >>>>>>>>>>> indexed
> >>>>>>>>>>>>>>>> another
> >>>>>>>>>>>>>>>>>>>>> 400,000 and I see 400,064 docs. Is there a good way
> to
> >>>>> find
> >>>>>>>>>>>>>>>> possibile
> >>>>>>>>>>>>>>>>>>>>> duplicates? I had tried to facet on key (our id
> field)
> >>>>> but
> >>>>>>>>>>> that
> >>>>>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>>>>>>>> give me anything with more than a count of 1.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 9:22 AM, Jamie Johnson <
> >>>>>>>>>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Ok, so clearing the transaction log allowed things
> to
> >>> go
> >>>>>>>>>>> again.
> >>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>>>> going to clear the index and try to replicate the
> >>>>> problem on
> >>>>>>>>>>>>>> 4.2.0
> >>>>>>>>>>>>>>>>>> and then
> >>>>>>>>>>>>>>>>>>>>>> I'll try on 4.2.1
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Apr 3, 2013 at 8:21 AM, Mark Miller <
> >>>>>>>>>>>>>> markrmil...@gmail.com
> >>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> No, not that I know if, which is why I say we need
> to
> >>>>> get
> >>>>>>>>>>> to the
> >>>>>>>>>>>>>>>>>> bottom
> >>>>>>>>>>>>>>>>>>>>>>> of it.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Apr 2, 2013, at 10:18 PM, Jamie Johnson <
> >>>>>>>>>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Mark
> >>>>>>>>>>>>>>>>>>>>>>>> It's there a particular jira issue that you think
> >>> may
> >>>>>>>>>>> address
> >>>>>>>>>>>>>>>> this?
> >>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>> read
> >>>>>>>>>>>>>>>>>>>>>>>> through it quickly but didn't see one that jumped
> >>> out
> >>>>>>>>>>>>>>>>>>>>>>>> On Apr 2, 2013 10:07 PM, "Jamie Johnson" <
> >>>>>>>>>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> I brought the bad one down and back up and it did
> >>>>>>>>>>> nothing. I
> >>>>>>>>>>>>>> can
> >>>>>>>>>>>>>>>>>>>>>>> clear
> >>>>>>>>>>>>>>>>>>>>>>>>> the index and try4.2.1. I will save off the logs
> >>> and
> >>>>> see
> >>>>>>>>>>> if
> >>>>>>>>>>>>>> there
> >>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>> anything else odd
> >>>>>>>>>>>>>>>>>>>>>>>>> On Apr 2, 2013 9:13 PM, "Mark Miller" <
> >>>>>>>>>>> markrmil...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> It would appear it's a bug given what you have
> >>> said.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Any other exceptions would be useful. Might be
> >>> best
> >>>>> to
> >>>>>>>>>>> start
> >>>>>>>>>>>>>>>>>>>>>>> tracking in
> >>>>>>>>>>>>>>>>>>>>>>>>>> a JIRA issue as well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> To fix, I'd bring the behind node down and back
> >>>>> again.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Unfortunately, I'm pressed for time, but we
> really
> >>>>> need
> >>>>>>>>>>> to
> >>>>>>>>>>>>>> get
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> bottom of this and fix it, or determine if it's
> >>>>> fixed in
> >>>>>>>>>>>>>> 4.2.1
> >>>>>>>>>>>>>>>>>>>>>>> (spreading
> >>>>>>>>>>>>>>>>>>>>>>>>>> to mirrors now).
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> On Apr 2, 2013, at 7:21 PM, Jamie Johnson <
> >>>>>>>>>>> jej2...@gmail.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Sorry I didn't ask the obvious question. Is
> >>> there
> >>>>>>>>>>> anything
> >>>>>>>>>>>>>>>> else
> >>>>>>>>>>>>>>>>>>>>>>> that I
> >>>>>>>>>>>>>>>>>>>>>>>>>>> should be looking for here and is this a bug?
> >>> I'd
> >>>>> be
> >>>>>>>>>>> happy
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>> troll
> >>>>>>>>>>>>>>>>>>>>>>>>>>> through the logs further if more information is
> >>>>>>>>>>> needed, just
> >>>>>>>>>>>>>>>> let
> >>>>>>>>>>>>>>>>>> me
> >>>>>>>>>>>>>>>>>>>>>>>>>> know.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Also what is the most appropriate mechanism to
> >>> fix
> >>>>>>>>>>> this.
> >>>>>>>>>>>>>> Is it
> >>>>>>>>>>>>>>>>>>>>>>>>>> required to
> >>>>>>>>>>>>>>>>>>>>>>>>>>> kill the index that is out of sync and let solr
> >>>>> resync
> >>>>>>>>>>>>>> things?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 2, 2013 at 5:45 PM, Jamie Johnson <
> >>>>>>>>>>>>>>>> jej2...@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> sorry for spamming here....
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> shard5-core2 is the instance we're having
> issues
> >>>>>>>>>>> with...
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 7:27:14 PM
> >>>>>>>>>>> org.apache.solr.common.SolrException
> >>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> SEVERE: shard update error StdNode:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> http://10.38.33.17:7577/solr/dsc-shard5-core2/:org.apache.solr.common.SolrException
> >>>>>>>>>>>>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Server at
> >>>>>>>>>>>>>>>> http://10.38.33.17:7577/solr/dsc-shard5-core2returned
> >>>>>>>>>>>>>>>>>>>>>>> non
> >>>>>>>>>>>>>>>>>>>>>>>>>> ok
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> status:503, message:Service Unavailable
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 2, 2013 at 5:43 PM, Jamie Johnson
> <
> >>>>>>>>>>>>>>>>>> jej2...@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> here is another one that looks interesting
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 7:27:14 PM
> >>>>>>>>>>>>>> org.apache.solr.common.SolrException
> >>>>>>>>>>>>>>>> log
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> SEVERE: org.apache.solr.common.SolrException:
> >>>>>>>>>>> ClusterState
> >>>>>>>>>>>>>>>> says
> >>>>>>>>>>>>>>>>>>>>>>> we are
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the leader, but locally we don't think so
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistributedUpdateProcessor.java:293)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:228)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:339)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>> org.apache.solr.core.SolrCore.execute(SolrCore.java:1797)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:637)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 2, 2013 at 5:41 PM, Jamie
> Johnson <
> >>>>>>>>>>>>>>>>>> jej2...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking at the master it looks like at some
> >>> point
> >>>>>>>>>>> there
> >>>>>>>>>>>>>> were
> >>>>>>>>>>>>>>>>>>>>>>> shards
> >>>>>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> went down. I am seeing things like what is
> >>>>> below.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> NFO: A cluster state change: WatchedEvent
> >>>>>>>>>>>>>>>> state:SyncConnected
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> type:NodeChildrenChanged path:/live_nodes,
> has
> >>>>>>>>>>> occurred -
> >>>>>>>>>>>>>>>>>>>>>>>>>> updating... (live
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nodes size: 12)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>>>>>>>>>>>>>>>>> org.apache.solr.common.cloud.ZkStateReader$3
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> process
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: Updating live nodes... (9)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runLeaderProcess
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: Running the leader process.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldIBeLeader
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: Checking if I should try and be the
> >>> leader.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> shouldIBeLeader
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: My last published State was Active,
> it's
> >>>>> okay
> >>>>>>>>>>> to be
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> leader.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:12:52 PM
> >>>>>>>>>>>>>>>>>>>>>>>>>> org.apache.solr.cloud.ShardLeaderElectionContext
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runLeaderProcess
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: I may be the new leader - try and sync
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 2, 2013 at 5:09 PM, Mark Miller
> <
> >>>>>>>>>>>>>>>>>>>>>>> markrmil...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I don't think the versions you are thinking
> >>> of
> >>>>>>>>>>> apply
> >>>>>>>>>>>>>> here.
> >>>>>>>>>>>>>>>>>>>>>>> Peersync
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does not look at that - it looks at version
> >>>>>>>>>>> numbers for
> >>>>>>>>>>>>>>>>>>>>>>> updates in
> >>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transaction log - it compares the last 100
> of
> >>>>> them
> >>>>>>>>>>> on
> >>>>>>>>>>>>>>>> leader
> >>>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> replica.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What it's saying is that the replica seems
> to
> >>>>> have
> >>>>>>>>>>>>>> versions
> >>>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>>>>>>>> the leader
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does not. Have you scanned the logs for any
> >>>>>>>>>>> interesting
> >>>>>>>>>>>>>>>>>>>>>>> exceptions?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Did the leader change during the heavy
> >>> indexing?
> >>>>>>>>>>> Did
> >>>>>>>>>>>>>> any zk
> >>>>>>>>>>>>>>>>>>>>>>> session
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> timeouts occur?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - Mark
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Apr 2, 2013, at 4:52 PM, Jamie Johnson <
> >>>>>>>>>>>>>>>> jej2...@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I am currently looking at moving our Solr
> >>>>> cluster
> >>>>>>>>>>> to
> >>>>>>>>>>>>>> 4.2
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>>>> noticed a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> strange issue while testing today.
> >>>>> Specifically
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>>> replica
> >>>>>>>>>>>>>>>>>>>>>>> has a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> higher
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version than the master which is causing
> the
> >>>>>>>>>>> index to
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>>>>>>>>>>> replicate.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Because of this the replica has fewer
> >>> documents
> >>>>>>>>>>> than
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>> master.
> >>>>>>>>>>>>>>>>>>>>>>>>>> What
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> could cause this and how can I resolve it
> >>>>> short of
> >>>>>>>>>>>>>> taking
> >>>>>>>>>>>>>>>>>>>>>>> down the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> index
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and scping the right version in?
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> MASTER:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Last Modified:about an hour ago
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Num Docs:164880
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Max Doc:164880
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Deleted Docs:0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Version:2387
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Segment Count:23
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> REPLICA:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Last Modified: about an hour ago
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Num Docs:164773
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Max Doc:164773
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Deleted Docs:0
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Version:3001
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Segment Count:30
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in the replicas log it says this:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: Creating new http client,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM
> >>>>>>>>>>> org.apache.solr.update.PeerSync
> >>>>>>>>>>>>>>>> sync
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> url=
> >>> http://10.38.33.17:7577/solrSTARTreplicas=[
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>> http://10.38.33.16:7575/solr/dsc-shard5-core1/
> >>>>> ]
> >>>>>>>>>>>>>>>>>> nUpdates=100
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM
> >>>>>>>>>>> org.apache.solr.update.PeerSync
> >>>>>>>>>>>>>>>>>>>>>>>>>> handleVersions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2 url=
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://10.38.33.17:7577/solr
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Received 100 versions from
> >>>>>>>>>>>>>>>>>>>>>>> 10.38.33.16:7575/solr/dsc-shard5-core1/
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM
> >>>>>>>>>>> org.apache.solr.update.PeerSync
> >>>>>>>>>>>>>>>>>>>>>>>>>> handleVersions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2 url=
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://10.38.33.17:7577/solr Our
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> versions are newer.
> >>>>>>>>>>> ourLowThreshold=1431233788792274944
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> otherHigh=1431233789440294912
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apr 2, 2013 8:15:06 PM
> >>>>>>>>>>> org.apache.solr.update.PeerSync
> >>>>>>>>>>>>>>>> sync
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> INFO: PeerSync: core=dsc-shard5-core2
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> url=http://10.38.33.17:7577/solrDONE.
> sync
> >>>>>>>>>>> succeeded
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> which again seems to point that it thinks
> it
> >>>>> has a
> >>>>>>>>>>>>>> newer
> >>>>>>>>>>>>>>>>>>>>>>> version of
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> index so it aborts. This happened while
> >>>>> having 10
> >>>>>>>>>>>>>> threads
> >>>>>>>>>>>>>>>>>>>>>>> indexing
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 10,000
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> items writing to a 6 shard (1 replica
> each)
> >>>>>>>>>>> cluster.
> >>>>>>>>>>>>>> Any
> >>>>>>>>>>>>>>>>>>>>>>> thoughts
> >>>>>>>>>>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> or what I should look for would be
> >>> appreciated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
