RE: SPLITSHARD not working as expected

[Oakley, Craig (NIH/NLM/NCBI) [C]]

"Sometimes for one of the sub-shards, the new leader and one of the new 
followers end up on the same instance"

Actually, it seems to be the case that every single time in the entire history 
of SPLITSHARD for one of the sub-shards, both the new leader and one of the new 
followers end up on the exact same instance.

I asked several months ago (see below under "ATTACHED MESSAGE") whether anyone 
anywhere had ever seen a case where this bug did not occur, and it seems that 
no one has been able to provide a counterexample: I think we have to concluded 
that this bug is universal

-Original Message-
From: Chris Ulicny  
Sent: Wednesday, January 30, 2019 1:46 PM
To: solr-user@lucene.apache.org
Subject: Re: SPLITSHARD not working as expected

I'm not sure what the expected behavior is. However, as of 7.4.0, it
doesn't seem like there is any attempt to prevent both the new leader and
follower replicas from being created on the same instance.

Sometimes for one of the sub-shards, the new leader and one of the new
followers end up on the same instance. We just manually end up moving them
since we don't split shards very often.

Best,
Chris

On Wed, Jan 30, 2019 at 12:46 PM Rahul Goswami 
wrote:

> Hello,
> I have a followup question on SPLITSHARD behavior. I understand that after
> a split, the leader replicas of the sub shards would reside on the same
> node as the leader of the parent. However, is there an expected behavior
> for the follower replicas of the sub shards as to where they will be
> created post split?
>
> Regards,
> Rahul
>
>
>
> On Wed, Jan 30, 2019 at 1:18 AM Rahul Goswami 
> wrote:
>
> > Thanks for the reply Jan. I have been referring to documentation for
> > SPLISHARD on 7.2.1
> > <
> https://lucene.apache.org/solr/guide/7_2/collections-api.html#splitshard>
> which
> > seems to be missing some important information present in 7.6
> > <
> https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard>.
> > Especially these two pieces of information.:
> > "When using splitMethod=rewrite (default) you must ensure that the node
> > running the leader of the parent shard has enough free disk space i.e.,
> > more than twice the index size, for the split to succeed "
> >
> > "The first replicas of resulting sub-shards will always be placed on the
> > shard leader node"
> >
> > The idea of having an entire shard (both the replicas of it) present on
> > the same node did come across as an unexpected behavior at the beginning.
> > Anyway, I guess I am going to have to take care of the rebalancing with
> > MOVEREPLICA following a SPLITSHARD.
> >
> > Thanks for the clarification.
> >
> >
> > On Mon, Jan 28, 2019 at 3:40 AM Jan Høydahl 
> wrote:
> >
> >> This is normal. Please read
> >>
> https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard
> >> PS: Images won't make it to the list, but don't think you need a
> >> screenshot here, what you describe is the default behaviour.
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >> > 28. jan. 2019 kl. 09:05 skrev Rahul Goswami :
> >> >
> >> > Hello,
> >> > I am using Solr 7.2.1. I created a two node example collection on the
> >> same machine. Two shards with two replicas each. I then called
> SPLITSHARD
> >> on shard2 and expected the split shards to have one replica on each
> node.
> >> However I see that for shard2_1, both replicas reside on the same node.
> Is
> >> this a valid behavior?  Unless I am missing something, this could be
> >> potentially fatal.
> >> >
> >> > Here's the query and the cluster state post split:
> >> >
> >>
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
> >> <
> >>
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
> >
> >>
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > Rahul
> >>
> >>
>








 ATTACHED MESSAGE 
-Original Message-
From: Oakley, Craig (NIH/NLM/NCBI) [C]  
Sent: Wednesday, September 19, 2018 4:52 PM
To: solr-user@lucene.apache.org
Subject: RE: sharding and placement of replicas

I am still wondering whether anyone has ever seen any examples of this actually 
working (has anyone ever seen any example of SPLITSHARD on a two-node SolrCloud 
placing replicas of the each shard on different hosts than other replicas of 
the same shards

Re: SPLITSHARD not working as expected

[Chris Ulicny]

I'm not sure what the expected behavior is. However, as of 7.4.0, it
doesn't seem like there is any attempt to prevent both the new leader and
follower replicas from being created on the same instance.

Sometimes for one of the sub-shards, the new leader and one of the new
followers end up on the same instance. We just manually end up moving them
since we don't split shards very often.

Best,
Chris

On Wed, Jan 30, 2019 at 12:46 PM Rahul Goswami 
wrote:

> Hello,
> I have a followup question on SPLITSHARD behavior. I understand that after
> a split, the leader replicas of the sub shards would reside on the same
> node as the leader of the parent. However, is there an expected behavior
> for the follower replicas of the sub shards as to where they will be
> created post split?
>
> Regards,
> Rahul
>
>
>
> On Wed, Jan 30, 2019 at 1:18 AM Rahul Goswami 
> wrote:
>
> > Thanks for the reply Jan. I have been referring to documentation for
> > SPLISHARD on 7.2.1
> > <
> https://lucene.apache.org/solr/guide/7_2/collections-api.html#splitshard>
> which
> > seems to be missing some important information present in 7.6
> > <
> https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard>.
> > Especially these two pieces of information.:
> > "When using splitMethod=rewrite (default) you must ensure that the node
> > running the leader of the parent shard has enough free disk space i.e.,
> > more than twice the index size, for the split to succeed "
> >
> > "The first replicas of resulting sub-shards will always be placed on the
> > shard leader node"
> >
> > The idea of having an entire shard (both the replicas of it) present on
> > the same node did come across as an unexpected behavior at the beginning.
> > Anyway, I guess I am going to have to take care of the rebalancing with
> > MOVEREPLICA following a SPLITSHARD.
> >
> > Thanks for the clarification.
> >
> >
> > On Mon, Jan 28, 2019 at 3:40 AM Jan Høydahl 
> wrote:
> >
> >> This is normal. Please read
> >>
> https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard
> >> PS: Images won't make it to the list, but don't think you need a
> >> screenshot here, what you describe is the default behaviour.
> >>
> >> --
> >> Jan Høydahl, search solution architect
> >> Cominvent AS - www.cominvent.com
> >>
> >> > 28. jan. 2019 kl. 09:05 skrev Rahul Goswami :
> >> >
> >> > Hello,
> >> > I am using Solr 7.2.1. I created a two node example collection on the
> >> same machine. Two shards with two replicas each. I then called
> SPLITSHARD
> >> on shard2 and expected the split shards to have one replica on each
> node.
> >> However I see that for shard2_1, both replicas reside on the same node.
> Is
> >> this a valid behavior?  Unless I am missing something, this could be
> >> potentially fatal.
> >> >
> >> > Here's the query and the cluster state post split:
> >> >
> >>
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
> >> <
> >>
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
> >
> >>
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > Rahul
> >>
> >>
>

Re: SPLITSHARD not working as expected

[Rahul Goswami]

Hello,
I have a followup question on SPLITSHARD behavior. I understand that after
a split, the leader replicas of the sub shards would reside on the same
node as the leader of the parent. However, is there an expected behavior
for the follower replicas of the sub shards as to where they will be
created post split?

Regards,
Rahul



On Wed, Jan 30, 2019 at 1:18 AM Rahul Goswami  wrote:

> Thanks for the reply Jan. I have been referring to documentation for
> SPLISHARD on 7.2.1
>  
> which
> seems to be missing some important information present in 7.6
> .
> Especially these two pieces of information.:
> "When using splitMethod=rewrite (default) you must ensure that the node
> running the leader of the parent shard has enough free disk space i.e.,
> more than twice the index size, for the split to succeed "
>
> "The first replicas of resulting sub-shards will always be placed on the
> shard leader node"
>
> The idea of having an entire shard (both the replicas of it) present on
> the same node did come across as an unexpected behavior at the beginning.
> Anyway, I guess I am going to have to take care of the rebalancing with
> MOVEREPLICA following a SPLITSHARD.
>
> Thanks for the clarification.
>
>
> On Mon, Jan 28, 2019 at 3:40 AM Jan Høydahl  wrote:
>
>> This is normal. Please read
>> https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard
>> PS: Images won't make it to the list, but don't think you need a
>> screenshot here, what you describe is the default behaviour.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> > 28. jan. 2019 kl. 09:05 skrev Rahul Goswami :
>> >
>> > Hello,
>> > I am using Solr 7.2.1. I created a two node example collection on the
>> same machine. Two shards with two replicas each. I then called SPLITSHARD
>> on shard2 and expected the split shards to have one replica on each node.
>> However I see that for shard2_1, both replicas reside on the same node. Is
>> this a valid behavior?  Unless I am missing something, this could be
>> potentially fatal.
>> >
>> > Here's the query and the cluster state post split:
>> >
>> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
>> <
>> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true>
>>
>> >
>> >
>> >
>> > Thanks,
>> > Rahul
>>
>>

Re: SPLITSHARD not working as expected

[Rahul Goswami]

Thanks for the reply Jan. I have been referring to documentation for
SPLISHARD on 7.2.1

which
seems to be missing some important information present in 7.6
.
Especially these two pieces of information.:
"When using splitMethod=rewrite (default) you must ensure that the node
running the leader of the parent shard has enough free disk space i.e.,
more than twice the index size, for the split to succeed "

"The first replicas of resulting sub-shards will always be placed on the
shard leader node"

The idea of having an entire shard (both the replicas of it) present on the
same node did come across as an unexpected behavior at the beginning.
Anyway, I guess I am going to have to take care of the rebalancing with
MOVEREPLICA following a SPLITSHARD.

Thanks for the clarification.


On Mon, Jan 28, 2019 at 3:40 AM Jan Høydahl  wrote:

> This is normal. Please read
> https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard
> PS: Images won't make it to the list, but don't think you need a
> screenshot here, what you describe is the default behaviour.
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 28. jan. 2019 kl. 09:05 skrev Rahul Goswami :
> >
> > Hello,
> > I am using Solr 7.2.1. I created a two node example collection on the
> same machine. Two shards with two replicas each. I then called SPLITSHARD
> on shard2 and expected the split shards to have one replica on each node.
> However I see that for shard2_1, both replicas reside on the same node. Is
> this a valid behavior?  Unless I am missing something, this could be
> potentially fatal.
> >
> > Here's the query and the cluster state post split:
> >
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
> <
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true>
>
> >
> >
> >
> > Thanks,
> > Rahul
>
>

Re: SPLITSHARD not working as expected

[Jan Høydahl]

This is normal. Please read 
https://lucene.apache.org/solr/guide/7_6/collections-api.html#splitshard
PS: Images won't make it to the list, but don't think you need a screenshot 
here, what you describe is the default behaviour.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 28. jan. 2019 kl. 09:05 skrev Rahul Goswami :
> 
> Hello,
> I am using Solr 7.2.1. I created a two node example collection on the same 
> machine. Two shards with two replicas each. I then called SPLITSHARD on 
> shard2 and expected the split shards to have one replica on each node. 
> However I see that for shard2_1, both replicas reside on the same node. Is 
> this a valid behavior?  Unless I am missing something, this could be 
> potentially fatal.
> 
> Here's the query and the cluster state post split:
> http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true
>  
> 
>  
> 
> 
>  
> Thanks,
> Rahul

SPLITSHARD not working as expected

[Rahul Goswami]

Hello,
I am using Solr 7.2.1. I created a two node example collection on the same
machine. Two shards with two replicas each. I then called SPLITSHARD on
shard2 and expected the split shards to have one replica on each node.
However I see that for shard2_1, both replicas reside on the same node. Is
this a valid behavior?  Unless I am missing something, this could be
potentially fatal.

Here's the query and the cluster state post split:
http://localhost:8983/solr/admin/collections?action=SPLITSHARD=gettingstarted=shard2=true


[image: image.png]

Thanks,
Rahul