On 15 October 2015 at 23:54, Matteo Grolla <matteo.gro...@gmail.com> wrote:
> Don't think so,
> the default behaviour at 4), to my knowledge,is to wait 3 minutes
> (leaderVoteWait) for all replicas to come up to avoid electing a leader
> with stale data.
So the observed behaviour is unexpected to me
>
If I read your sequence of events properly i see that at point 4 there is
no replica recovering.
You have only 8984 in the cluster and in the history of the cluster 8983
has been a leader ( during 8984 dead period).
Let's directly go to the code and avoid all the doubts !
The waiting of 3 minutes is in here :
if (!weAreReplacement) {
> allReplicasInLine = waitForReplicasToComeUp(leaderVoteWait);
> }
In our case weAreReplacement should be true , because 8983 has been a
leader.
So we don't wait, we try to see if should be the leader, but we shouldn't
for this reason :
// maybe active but if the previous leader marked us as down and
> // we haven't recovered, then can't be leader
> final Replica.State lirState =
> zkController.getLeaderInitiatedRecoveryState(collection, shardId,
> core.getCoreDescriptor().getCloudDescriptor().getCoreNodeName());
> if (lirState == Replica.State.DOWN || lirState ==
> Replica.State.RECOVERING) {
> log.warn("Although my last published state is Active, the previous leader
> marked me "+core.getName()
> + " as " + lirState.toString()
> + " and I haven't recovered yet, so I shouldn't be the leader.");
> return false;
> }
> log.info("My last published State was Active, it's okay to be the
> leader.");
> return true;
> }
> log.info("My last published State was "
> + core.getCoreDescriptor().getCloudDescriptor().getLastPublished()
> + ", I won't be the leader.");
> // TODO: and if no one is a good candidate?
The second possible waiting is in registering the core :
// in this case, we want to wait for the leader as long as the leader might
> // wait for a vote, at least - but also long enough that a large cluster has
> // time to get its act together
> String leaderUrl = getLeader(cloudDesc, leaderVoteWait + 600000);
>
> I scrolled a little bit the code, and I think that because is the only
live node 8984, this second wait will not happen, as no leader is there and
an election should have happened before.
I have no time now to get into debug, would be interesting if you can, but
i would bet that with a single alive node , that came alive when no-one was
recovering, it will become the leader.
Actually, waiting for 3 minutes for the old replica to come back ( which
eventually could never happen), it's a little bit counterintuitive, because
more new replicas could come, and it's not so reasonable to keep the
cluster without a leader for 3 minutes ...
I would suggest some debugging to have a better idea of the internals, and
i would be really interesting in a better insight !
> I created a cluster of 2 nodes copying the server dir to node1 and node2
> and using those as solrhome for the nodes
> created the collection with
> bin/solr create -c test
> so it's using the builtin schemaless configuration
>
> there's nothing custom, should be all pretty standard
>
Adding some login should help, maybe you are hitting some additional
waiting time, added after 4.10 .
We should start the research after we have more insights from the logs.
Cheers
>
> 2015-10-15 17:42 GMT+02:00 Alessandro Benedetti <
> benedetti.ale...@gmail.com>
> :
>
> > Hi Matteo,
> >
> > On 15 October 2015 at 16:16, Matteo Grolla <matteo.gro...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I'm doing this test
> > > collection test is replicated on two solr nodes running on 8983, 8984
> > > using external zk
> > >
> > > 1)turn OFF solr 8984
> > > 2)add,commit a doc x con solr 8983
> > > 3)turn OFF solr 8983
> > > 4)turn ON solr 8984
> > >
> > At this point 8984 will be elected leader, because only element in the
> > cluster, it can not do anything to recover, so it will not replicate Doc
> x
> >
> > > 5)shortly after (leader still not elected) turn ON solr 8983
> > >
> > I assume that even if you are not able to see it, actually the leader
> > election was already starting, not taking in consideration 8983
> >
> > > 6)8984 is elected as leader
> > >
> > As expected
> >
> > > 7)doc x is present on 8983 but not on 8984 (check issuing a query)
> > >
> > This is expected as well.
> >
> > It is a very edge case, but i expect at the current status, the behaviour
> > you are obtaining is the expected one.
> > Probably the leader election should become smarter, for example, any
> time a
> > node came back to the cluster it should be checked, and in the case it
> > should be the leader a new election triggered.
> > Just thinking loud :)
> >
> >
> >
> > >
> > > attached are the logs of both solr
> > >
> > > BTW I'm using java 1.8.045 on osx yosemite and solr 5.2.1 seems much
> > > slower to startup than solr 4.10.3. it seems waiting on something
> > >
> >
> > I can not see any attached file, do you have any suggester in place ?
> > Anyway is weird as I assume you kept the solrconfig.xml the same.
> > Can you list the components you are currently using ?
> >
> > Cheers
> >
> >
> >
> > --
> > --------------------------
> >
> > Benedetti Alessandro
> > Visiting card - http://about.me/alessandro_benedetti
> > Blog - http://alexbenedetti.blogspot.co.uk
> >
> > "Tyger, tyger burning bright
> > In the forests of the night,
> > What immortal hand or eye
> > Could frame thy fearful symmetry?"
> >
> > William Blake - Songs of Experience -1794 England
> >
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
